internal poetry packages: declare poe tasks and airbyte-ci sections in pyproject.toml

[alafanechere]

What

Relates to #33880
The motivation behind this change is:

• I don't want the logic about what should run in CI to be maintained at the GHA workflow or airbyte-ci level.
• I want poetry packages to self declare what should run in CI.

airbyte-ci and GHA workflow remain orchestrators.
If we want to enforce consistency between internal packages I think we should add static code analysis of pyproject.toml files (in a dedicated new package or in airbyte-ci).

I believe this approach would also be highly beneficial for connectors, but first applying this to our internal packages to experience this pattern is a good enough first step.

How

• Use poe task runner on our internal poetry packages.
• In pyproject.toml: Declare test, lint etc. tasks to wrap pytest / mypy etc.
• In pyproject.toml: Declare a new [tool.airbyte-ci] section which airbyte-ci will consume to run the listed poe task in CI.
• Declare a pydantic Model in airbyte-ci (pipelines) to deserialize and validate the [tool.airbyte-ci] section in pyproject.toml.

🚨 User Impact 🚨

This change has no effect. It's actually used in the downstream PR on this stack

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Feb 7, 2024 2:30pm

[alafanechere]

• master
  • internal poetry packages: declare poe tasks and airbyte-ci sections in pyproject.toml #34735 👈
    • airbyte-ci: run poe tasks declared in pyproject.toml file of internal poetry packages #34736
      • rename airbyte-ci-test.yml workflow to internal-poetry-packages-ci.yml #34737

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @alafanechere and the rest of your teammates on Graphite

[natikgadzhi]

Once we do this, and connectors use poetry, do we expect them to also declare tasks in poe, or they will have a special case where airbyte-ci has opinions on how to test them? That part does not change, right? We'll run the full battery of tests without connector pyproject.toml defining them?

[girarda]

this is great!

[girarda]

This isn't used as part of this PR, but it is used in this follow up PR by get_airbyte_ci_package_config, which is itself called by the test command to identify the poe_tasks to run and how to configure the container 👍

[alafanechere]

Once we do this, and connectors use poetry, do we expect them to also declare tasks in poe, or they will have a special case where airbyte-ci has opinions on how to test them? That part does not change, right? We'll run the full battery of tests without connector pyproject.toml defining them?

@natikgadzhi in an ideal world I'd love airbyte-ci to have 0 opinion on which test should run.
I'd love to:

• make connectors declare what should run in the CI from the airbyte-ci config section in pyproject.toml
  • unit test
  • integration test
  • type checks
  • metadata validation
  • CAT
  • etc.
• have a separate tool (called by airbyte-ci) that will check that the airbyte-ci config section in pyproject.toml matches our requirements for a specific set of connectors (e.g. certified)

This will give more flexibility to connector developers in term of CI configuration and also more visibility about what's running in CI. airbyte-ci is currently too much of a blackbox.

I suggest to start the adoption this pattern for connectors with the introduction of type checks / mypy.
More to come in an upcoming tech spec.