Connector development on Windows

[LarsHanegraaf]

Update the documentation to make development of connectors on Windows easier

Hi team,

First of all, credits to where the Airbyte product is so far! 🎉

This week I went through the process of creating an Airbyte Source connector on a Windows PC (It's what our company works with 😉). I would like to report the things I have found and how the process could be improved for other people who go through a similar workflow. I'm willing to submit PRs for the documentation and Airbyte itself, but as it is the first time I contribute to an open-source project, I need some guidance on whether it will be helpful for Airbyte or not. Thanks!

Things I've found:

1. Documentation about developing a Python Source is complete, but scattered

The documentation to develop a Python Source is complete, but as there are multiple pages with different approaches, it can be confusing for a newcomer (at least within my team). I know that the process around building a connector is evolving, but I think more consistency would help to improve the Developer Experience.

Examples that could use more consistency:

• Steps to implement a connector:
  • You have the CDK (I know, this is generic also for other languages)
  • You have a speedrun
  • A generic Python source
  • The README that is generated from the CDK.
  • Idea: Create one page that contains all the background knowledge and add code blocks in between, where you can toggle between programming languages (like the Apache Spark documentation does).
• The documentation on how to generate template files with the CDK:
  • Instructions within the Airbyte Repo.
  • Instructions in the docs.
• How tests work:
  • Source Acceptance tests
  • Integration tests (I think the title is confusing for this page)
  • Tests within the tutorial
  • The README that is generated from the CDK.

2. Code generation scripts are tailored towards Unix

All scripts for generating Python boilerplate code are heavily geared towards Unix. I can reverse engineer the proposed working of a Bash script myself, so I can run it in Powershell or CMD, but maybe the process could be changed so it works for all platforms (or add steps to the documentation, I'm not sure what has priority in your opnion)

The steps I have taken to make it work:

0. Have Python 3.9 (important, but not documented in the README of the CDK), and Node installed.
1. Clone the Airbyte repo
2. Go to the generator folder.
3. Run npm install and npm run generate (Docker will also be possible, using the command in generate.sh, but this one is easier for me)
4. Follow the instructions of the generator
5. Go to the folder that is generated for you
6. In that folder run python -m venv .venv
7. When you're on Powershell, run .venv/Scripts/activate to activate the virtual environment.
8. Install dependencies using python -m pip install -r requirements.txt
9. Install the source-acceptance-test plugin by navigating to this folder with the virtual environment still activated and run python setup.py install
10. Navigate back to the connector folder
11. Run pip install '.[tests]'

3. Bugs in source-acceptance-test Pytest plugin when running on Windows

When you want to run the source acceptance tests on Windows, I was only able to get it working within the virtual environment (not in Docker, but unsure why).

In order to make the tests work in the virtual environment, I had to change to following things:

• Change this line to streams_without_records = streams_without_records - set(allowed_empty_streams)
• Change this line to run(["mkdir", "-p", LOG_DIR], shell=True).
• Remove pytest-sugar as dependency, due to this bug on Windows. (using pip uninstall pytest-sugar)

It has become a bit longer than expected 😛 . So if I need to split it into multiple issues, just let me know!

Thanks!

[djongeb]

@LarsHanegraaf Thanks for this little writeup. I've been struggling to find information on how to develop your own connector whilst using windows. This has been very helpful and I would agree with you, that there could be more information on how to use windows! :D

[marcosmarxm]

@Amruta-Ranade + @arimbr maybe something we can bring to content team to work on?

[marcosmarxm]

@LarsHanegraaf thanks for the suggestion probably this will help a large number of users of Airbyte.

[francesco-av]

@LarsHanegraaf Thanks for this little writeup. I've been struggling to find information on how to develop your own connector whilst using windows. This has been very helpful and I would agree with you, that there could be more information on how to use windows! :D

@djongeb I was wondering if you ever found clearer documentation / guides for Windows? I am struggling to execute several of the Linux commands and had to add the source manually for example.

[natikgadzhi]

@LarsHanegraaf belated thank you for the writeup, it's excellent. I agree the docs are very scattered — working on them more now.

I'll keep the issue open to track any Windows-related questions. Windows workflow is not necessarily a priority right away.

If anyone recently made basic workflows work on Windows, and you had to tweak any tests to do that (they changed since 2022 very significantly) — contributions are very welcome.