ci_connector_ops: POC of CI connector pipelines in python

[alafanechere]

What

Closes #23504
This is an attempt at rewriting airbyte-python.gradle and airbyte-connector-acceptance-test.gradle in Python (with the use of Dagger).
The motivation behind this work can be found here

I apologize for the size of the PR but I'd like to use it as a common playground for Dagger team and us to iterate on best practices and optimization they could suggest.

How

• Declare a new pipelines subpackage in ci_connector_ops
• Expose a connectors-ci command that can run CI pipelines on one our multiple connectors and according to code change:

connectors-ci test-connectors --name=source-pokeapi --name=source-openweather
connectors-ci test-connectors --modified # Test only connectors modified in the current branch

Recommended reading order

1. The README !!!
2. The entrypoints with DAG logic
3. A context object to customize execution and hold pipeline configuration
4. The actual tests we run
5. The reusable environments

All the other change (commits before 0431cf6) are made to improve/patch our existing tooling to work in a CI context with Dagger.

The target DAG

flowchart LR;
    AB_GIT_REPO[Airbyte Git Repo] --> MOUNT_AB[Mount Airbyte repo to container];
    AB_GIT_REPO[Airbyte Git Repo] --> MOUNT_CONN[Mount connector source code to container];
    DOWN_SECRETS[Download secrets from GSM]-->CAT
    MOUNT_AB-->QA[Run QA checks];
    MOUNT_CONN-->FORMAT[Code format checks];
    MOUNT_CONN-->INSTALL_CONN[Install connector package in container];
    INSTALL_CONN-->UNIT_TESTS[Run unit tests];
    UNIT_TESTS-->INTEGRATION_TESTS[Run integration tests];
    UNIT_TESTS-->DOCKER_BUILD[Docker build connector dev image];
    DOCKER_BUILD-->CAT[Run acceptance tests];
    CAT-->UPLOAD_SECRETS[Upload updated secrets to GSM];
    CAT-->REPORT[Build test report];
    UNIT_TESTS-->REPORT;
    INTEGRATION_TESTS-->REPORT;
    QA-->REPORT;
    FORMAT-->REPORT;
    REPORT--if in CI-->UPLOAD[Upload to S3];

Loading

🚨 User Impact 🚨

Before assessing a user impact I'd like to first get my question to Dagger teams answered:
https://github.com/airbytehq/airbyte/blob/augustin/poc/ci-in-python/tools/ci_connector_ops/ci_connector_ops/pipelines/README.md

[github-actions]

Affected Connector Report

The latest commit has removed all connector-related changes. There are no more dependent connectors for this PR.

[alafanechere]

This is a patch whose reason is detailed here and that I'd prefer to avoid.

[alafanechere]

This is to allow black to run in a non-git context.

[evantahler]

Very cool POC!

[evantahler]

How do we enforce or change our python version?

[alafanechere]

The package declares that it requires Python 3.10 here:

airbyte/tools/ci_connector_ops/setup.py

Line 51 in 1c49dc7

python_requires=">=3.10",

I'm using pyenv to switch my python version to 3.10 when I'm working on this project. Our developers should have pyenv installed as I think it's part of our core python development tooling.

This file says to pyenv to switch to 3.10 when you're working on this code directory:

airbyte/tools/ci_connector_ops/.python-version

Line 1 in 1c49dc7

3.10.0

[helderco]

Nice work 👍 🙂

I didn't review everything because I need to stop for now and don't want to leave it pending.

[alafanechere]

Thank you so much @helderco for your review and suggestions!!!

[alafanechere]

I'll merge now. There are still great improvements to make but I feel like they should be done incrementally. IMO the current state of this PR accomplished the goal of reimplementing airbyte-python.gradle and airbyte-connector-acceptance.gradle in Python with Dagger.
We'll observe how it behaves on connector PRs after the merge and probably find interesting learnings.
Here are the main improvements to make according to the review from @helderco and my personal opinion:

• Rework the concurrency implementation. We shall rely only on aniyo and not call asyncio native code or asyncer.asyncify.
• The DAG logic is not perfect: the integration tests should get secrets too.
• The ci_connector_ops python package and dependencies are becoming difficult to manage because we stuffed it with different projects that have very diverse needs in terms of python version and libraries.
• We need to focus a bit more on making sure each step re-execution is legit and leverage caching as much as possible. (e.g. writing secrets on every run leads to re-trigger of the acceptance tests on every run because the secrets file timestamp changed even if their content is different).
• We might consider relying more on OOP to benefit from the abstraction that can be useful to run pipelines for Python and Java connectors with common logic and reusable code.

cc @evantahler @cpdeethree @bnchrch @jpadams @kpenfound

[bazarnov]

@alafanechere the https://github.com/airbytehq/airbyte/actions/runs/4391568285 test is failing but says success, not sure what's going on here, need more context around the work you've done, but I think I can safely merge in the related PR:
#23822

Am I right? or should I wait until it's fixed on your side?