Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

airbyte-ci: compute GHA runs-on from --ci-requirements #34220

Conversation

alafanechere
Copy link
Contributor

@alafanechere alafanechere commented Jan 12, 2024

What

Closes #34076

We want to dynamically determine the self hosted CI runner to use according to the output of --ci-requirements

How

  • Introduce a new airbyte-ci-requirements action: it will run --ci-requirements on inputted commands, on a github hosted ubuntu runner.
  • Use this action in an upstream job in all our existing GHA workflow using airbyte-ci. We run this preliminary job to compute the name of the self-hosted runner the main job will use (runs-on field).

Copy link

vercel bot commented Jan 12, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Jan 16, 2024 2:09pm

Copy link
Contributor Author

alafanechere commented Jan 12, 2024

In this stack: 🔧 Add CI requirements to Airbyte CLI(?)

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @alafanechere and the rest of your teammates on Graphite Graphite

@alafanechere alafanechere force-pushed the augustin/01-11-airbyte-ci_CLI_exposes_CI_requirements branch from 18552bb to 9087f08 Compare January 12, 2024 16:20
@alafanechere alafanechere force-pushed the augustin/01-12-airbyte-ci_compute_GHA_runs-on_from_--ci-requirements_ branch from f73a63c to 08e88e7 Compare January 12, 2024 16:20
@alafanechere alafanechere marked this pull request as ready for review January 12, 2024 16:27
@alafanechere alafanechere requested a review from a team January 12, 2024 16:27
Copy link
Contributor

@evantahler evantahler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!
Testing GHA is always tricky, but this looks right to me.

I think maybe a note should be added in the docs that all N of the actions that run stuff with airbyte-ci will dynamically choose which runner to work on based on this (link to python code)

@octavia-squidington-iv octavia-squidington-iv requested a review from a team January 12, 2024 16:35
@alafanechere alafanechere force-pushed the augustin/01-12-airbyte-ci_compute_GHA_runs-on_from_--ci-requirements_ branch 2 times, most recently from f897de6 to 2f81a55 Compare January 12, 2024 18:03
@alafanechere alafanechere force-pushed the augustin/01-11-airbyte-ci_CLI_exposes_CI_requirements branch 2 times, most recently from 211409e to 88c7b9e Compare January 15, 2024 11:08
@alafanechere
Copy link
Contributor Author

Nice work! Testing GHA is always tricky, but this looks right to me.

I think maybe a note should be added in the docs that all N of the actions that run stuff with airbyte-ci will dynamically choose which runner to work on based on this (link to python code)

I updated the command documentation in #34218

@alafanechere alafanechere force-pushed the augustin/01-12-airbyte-ci_compute_GHA_runs-on_from_--ci-requirements_ branch 2 times, most recently from 5cadcb2 to 0e0f356 Compare January 15, 2024 11:15
Base automatically changed from augustin/01-11-airbyte-ci_CLI_exposes_CI_requirements to master January 15, 2024 18:09
@alafanechere alafanechere force-pushed the augustin/01-12-airbyte-ci_compute_GHA_runs-on_from_--ci-requirements_ branch 2 times, most recently from e2a980a to 2a14f08 Compare January 15, 2024 18:14
@alafanechere alafanechere force-pushed the augustin/01-12-airbyte-ci_compute_GHA_runs-on_from_--ci-requirements_ branch from 0876dde to 9a404b6 Compare January 16, 2024 14:09
@alafanechere alafanechere enabled auto-merge (squash) January 16, 2024 14:17
@alafanechere alafanechere merged commit 9ddb432 into master Jan 16, 2024
20 checks passed
@alafanechere alafanechere deleted the augustin/01-12-airbyte-ci_compute_GHA_runs-on_from_--ci-requirements_ branch January 16, 2024 14:23
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants