airbyte-ci: CLI exposes CI requirements #34218
Conversation
|
In this stack:
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @alafanechere and the rest of your teammates on |
alafanechere
force-pushed
the
augustin/01-11-airbyte-ci_CLI_exposes_CI_requirements
branch
2 times, most recently
from
fb831ed
to
91b9c91
Compare
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
alafanechere
force-pushed
the
augustin/01-11-airbyte-ci_CLI_exposes_CI_requirements
branch
7 times, most recently
from
5163fd2
to
18552bb
Compare
alafanechere
force-pushed
the
augustin/01-11-airbyte-ci_CLI_exposes_CI_requirements
branch
from
18552bb
to
9087f08
Compare
| @@ -121,3 +126,27 @@ def decorated_function(*args: Any, **kwargs: Any) -> Any: # noqa: ANN401 | |||
| return f(*args, **kwargs) | |||
|
|
|||
| return decorated_function | |||
|
|
|||
|
|
|||
| def click_ci_requirements_option() -> Callable[[FC], FC]: | |||
There was a problem hiding this comment.
We can eventually add arguments to this decorator to customize requirements according to commands.
|
|
||
|
|
||
| @dataclass | ||
| class CIRequirements: |
There was a problem hiding this comment.
I used a dataclass here in case we want to make the CI requirements payload a bit more complex. It does not have custom attributes at the moment but we can easily add some if needed.
alafanechere
force-pushed
the
augustin/01-11-airbyte-ci_CLI_exposes_CI_requirements
branch
from
9087f08
to
211409e
Compare
alafanechere
force-pushed
the
augustin/01-11-airbyte-ci_CLI_exposes_CI_requirements
branch
from
211409e
to
88c7b9e
Compare
| @@ -83,6 +88,9 @@ def check_local_docker_configuration() -> None: | |||
|
|
|||
|
|
|||
| def is_dagger_run_enabled_by_default() -> bool: | |||
| if CI_REQUIREMENTS_OPTION_NAME in sys.argv: | |||
There was a problem hiding this comment.
💅 Note for future: We do need to invert this at some point where commands specifiy if they need dagger run or not.
|
/approve-and-merge reason="format is known to be broken on airbyte-ci changes, #34220 will solve it" |
|
After a careful ML study, |
octavia-approvington
deleted the
augustin/01-11-airbyte-ci_CLI_exposes_CI_requirements
branch
What
Relates to #34076
We currently have to define the dagger version to use at two places:
dagger-iopython package versionOn the infra side we have pre-populated runners using different dagger version: 0.6.4 and 0.9.5 .
The targeted runner for a CI job is currently in the
runs-onfield of our GHA workflows.We can face CI failures when we have a mismatch between the client and infra side dagger version.
Mismatch can happen when:
Fixing these mismatch scenario will make the Dagger upgrade / airbyte-ci version pinning a lot easier.
How
airbyte-cidefine its CI requirements, (only the dagger version ATM): we expose a new--ci-requirementsoption to the commands in the CI. It outputs a JSON payload, currently{"dagger_version": "0.9.5"}runs-onfrom--ci-requirements#34220 34220: In our GHA worfklows using airbyte-ci: Run an initialget-ci-runnerjob on a GHA runner which will call--ci-requirementsfor the specific command that will be run in a next job. The output ofget-ci-runneris used as theruns-onfield value of the main job.We'll be able to introduce more complex scenario, like picking a runner size dynamically, by changing the output of
--ci-requirements. We'll have to adapt workflows accordingly.