airbyte-ci: make `airbyte-ci test` able to run any poetry run command #33784

alafanechere · 2023-12-26T18:30:45Z

What

Relates to #33719

The current airbyte-ci test command is made to run tests on a poetry project.
It's currently only running a pytest command.
We have to change this command if we want to run other kind of commands to assess the health of the package (e.g. mypy) .

How

Add a --poetry-run-command options (mulitple) that will be run on the container where the poetry package is installed
Make required changes to our workflow according to this interface change.

vercel · 2023-12-26T18:30:51Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jan 3, 2024 4:24pm

alafanechere · 2023-12-26T18:30:56Z

Current dependencies on/for this PR:

master
- PR airbyte-ci: make airbyte-ci test able to run any poetry run command #33784 👈
  - PR airbyte-ci: fix types (mypy) #33785
    - PR airbyte-ci: fix linting and missing annotations #33797
    - PR airbyte-ci: enable mypy + ruff for airbyte-ci code #33712

This stack of pull requests is managed by Graphite.

github-actions · 2023-12-26T18:30:58Z

Warning

Soft code freeze is in effect until 2024-01-02. Please avoid merging to master. #freedom-and-responsibility

erohmensing

Functionally this seems fine - but I am wondering, why did we decide to do this under airbyte-ci test instead of introducing e.g. a airbyte-ci lint, similar to formatting, or trying to put format and lint in similar places?

To me, test implies running test, while running e.g mypy involves static checking that could for example have a certain list of files to ignore (as format does) or other linting-specific configuration.

alafanechere · 2024-01-03T12:00:14Z

Functionally this seems fine - but I am wondering, why did we decide to do this under airbyte-ci test instead of introducing e.g. a airbyte-ci lint, similar to formatting, or trying to put format and lint in similar places?

To me, test implies running test, while running e.g mypy involves static checking that could for example have a certain list of files to ignore (as format does) or other linting-specific configuration.

I agree it would be more elegant to place this under a different command.
But I'm not sure lint can be similar to format in this scenario as the linting rules are poetry package specific.
I introduced this in test as airbyte-ci test is poetry package specific.

Maybe it's time for a poetry command group or a different approach for CI for poetry packages.
@erohmensing I filled up this issue with a couple of options I came up with. Do you agree this can be done in a follow-up change?

erohmensing · 2024-01-03T15:49:21Z

@alafanechere thanks! this actually clears things up for me - we're only linting our code, not the whole codebase. So if we wanted to e.g. lint connector code, we would probably be doing that in connectors test, and if we wanted to do it for the cdk... i'm not sure where we'd do it. But I agree that in the way we're doing it now, it is closer to our testing than our formatting.

Yes, a follow up PR sounds fine to me. Will do a proper review!

erohmensing

One question about erroring on failure, otherwise looks 🚢 able to me! I agree that the behavior is starting to look more like "airbyte ci do whatever poetry wants to perform on this package", thanks for writing up the other issue :)

airbyte-ci/connectors/pipelines/README.md

erohmensing · 2024-01-03T16:08:42Z

airbyte-ci/connectors/pipelines/README.md

-`airbyte-ci test airbyte-ci/connectors/pipelines --test-directory=tests`
-`airbyte-ci tests airbyte-integrations/bases/connector-acceptance-test --test-directory=unit_tests`
+`airbyte-ci test airbyte-ci/connectors/pipelines --poetry-run-command='pytest tests'`
+`airbyte-ci tests airbyte-integrations/bases/connector-acceptance-test--poetry-run-command='pytest tests/unit_tests'`


Suggested change

`airbyte-ci tests airbyte-integrations/bases/connector-acceptance-test--poetry-run-command='pytest tests/unit_tests'`

`airbyte-ci tests airbyte-integrations/bases/connector-acceptance-test--poetry-run-command='pytest unit_tests'`

erohmensing · 2024-01-03T16:11:19Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/test/commands.py

+@click.option(
+    "-c",
+    "--poetry-run-command",
+    multiple=True,


This means you can pass the option multiple times, not that you can pass multiple arguements to the option, right? An example in the docs of how to pass multiple may be helpful, as well as noting somewhere (new column?) that it can be passed multiple times.

Yes you're right. I added an example to the README.

erohmensing · 2024-01-03T16:16:15Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/test/commands.py

+    soon_command_executions_results = []
+    async with asyncer.create_task_group() as poetry_commands_task_group:
+        for command in commands_to_run:
+            logger.info(f"Running command: {command}")
+            soon_command_execution_result = poetry_commands_task_group.soonify(run_poetry_command)(test_container, command)
+            soon_command_executions_results.append(soon_command_execution_result)
+
+    for result in soon_command_executions_results:
+        stdout, stderr = result.value
+        logger.info(stdout)
+        logger.error(stderr)


Do we still get the exit on failure behavior that we want? I know in format, when running multiple steps concurrently we had to add explicit error exiting

If an execution fails the async context will throw an ExceptionGroup which will lead to a click command failure and a failed CI pipeline.

Ah, great! I guess before it was different because we wanted to continue all the others and collect successes and failures? I suppose that could be helpful here too, but it's not blocking for me, as its all our code that we're checking

Yes you're right, we could a Dagger exec error handling to better handle success / failure.

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

postamar · 2024-01-03T16:46:36Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/test/commands.py

+    for result in soon_command_executions_results:
+        stdout, stderr = result.value
+        logger.info(stdout)
+        logger.error(stderr)


With multiple commands this will output a jumble of lines which will be very hard to relate to the command itself. Sequentializing the output will require more code. Is the concurrency here really worth it in terms of the added complexity? I'm asking because I don't actually know; however if it's not really worth it then let's execute the poetry commands sequentially. More code => bigger maintenance burden.

postamar · 2024-01-03T16:47:30Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/test/commands.py

+        Tuple[str, str]: The stdout and stderr of the command.
+    """
+    container = container.with_exec(["poetry", "run", *command.split(" ")])
+    return await container.stdout(), await container.stderr()


Wouldn't we be better off returning a dagger.Container here? I'm a bit surprised that we're not interested in the exit code, but I guess that dagger handles failures for us transparently.

…#33784) Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

sentry-io · 2024-01-03T22:36:29Z

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

‼️ ExecError: process "poetry run mypy pipelines --disallow-untyped-defs" did not complete successfully: exit c... pipelines.airbyte_ci.test.commands in run_poetr... View Issue
‼️ ExecError: process "poetry run ruff check pipelines" did not complete successfully: exit code: 1 pipelines.airbyte_ci.test.commands in run_poetr... View Issue
‼️ ExecError: process "poetry run pytest tests" did not complete successfully: exit code: 1 pipelines.airbyte_ci.test.commands in run_poetr... View Issue
‼️ ExecError: process "poetry run pytest tests" did not complete successfully: exit code: 1 pipelines.airbyte_ci.test.commands in run_poetr... View Issue
‼️ ExecError: process "poetry run ruff check pipelines" did not complete successfully: exit code: 1 pipelines.airbyte_ci.test.commands in run_poetr... View Issue

_{Did you find this useful? React with a 👍 or 👎}

…airbytehq#33784) Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

This was referenced Dec 26, 2023

airbyte-ci: enable mypy + ruff for airbyte-ci code #33712

Merged

airbyte-ci: fix types (mypy) #33785

Merged

alafanechere force-pushed the augustin/12-26-airbyte-ci_make_airbyte-ci_test_able_to_run_any_poetry_run_command branch from 9a6d133 to ab794c1 Compare December 26, 2023 19:16

alafanechere marked this pull request as ready for review December 27, 2023 07:52

alafanechere requested a review from a team December 27, 2023 07:52

alafanechere mentioned this pull request Dec 27, 2023

airbyte-ci: fix linting and missing annotations #33797

Merged

alafanechere force-pushed the augustin/12-26-airbyte-ci_make_airbyte-ci_test_able_to_run_any_poetry_run_command branch from 3a4bd1e to 6b1ff5e Compare December 27, 2023 12:02

erohmensing reviewed Jan 2, 2024

View reviewed changes

alafanechere force-pushed the augustin/12-26-airbyte-ci_make_airbyte-ci_test_able_to_run_any_poetry_run_command branch from 6b1ff5e to b8a63f7 Compare January 3, 2024 13:21

erohmensing approved these changes Jan 3, 2024

View reviewed changes

alafanechere and others added 4 commits January 3, 2024 17:23

airbyte-ci: make airbyte-ci test able to run any poetry run command

87665c1

improve logging and parallelization

a84b5a3

Update airbyte-ci/connectors/pipelines/README.md

004cdcc

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

update readme with better example

05f5378

alafanechere force-pushed the augustin/12-26-airbyte-ci_make_airbyte-ci_test_able_to_run_any_poetry_run_command branch from 9ed1ae5 to 05f5378 Compare January 3, 2024 16:24

alafanechere enabled auto-merge (squash) January 3, 2024 16:28

alafanechere merged commit 823d122 into master Jan 3, 2024
20 checks passed

alafanechere deleted the augustin/12-26-airbyte-ci_make_airbyte-ci_test_able_to_run_any_poetry_run_command branch January 3, 2024 16:45

postamar reviewed Jan 3, 2024

View reviewed changes

gisripa pushed a commit that referenced this pull request Jan 3, 2024

airbyte-ci: make airbyte-ci test able to run any poetry run command (…

14435db

…#33784) Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

airbyte-ci: make airbyte-ci test able to run any poetry run command (…

2f0a913

…airbytehq#33784) Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

airbyte-ci: make airbyte-ci test able to run any poetry run command (…

c37f25f

…airbytehq#33784) Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

airbyte-ci: make airbyte-ci test able to run any poetry run command (…

23b09c7

…airbytehq#33784) Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

airbyte-ci: make `airbyte-ci test` able to run any poetry run command #33784

airbyte-ci: make `airbyte-ci test` able to run any poetry run command #33784

alafanechere commented Dec 26, 2023 •

edited

Loading

vercel bot commented Dec 26, 2023 •

edited

Loading

alafanechere commented Dec 26, 2023 •

edited

Loading

github-actions bot commented Dec 26, 2023

erohmensing left a comment

alafanechere commented Jan 3, 2024

erohmensing commented Jan 3, 2024

erohmensing left a comment

erohmensing Jan 3, 2024

erohmensing Jan 3, 2024

alafanechere Jan 3, 2024

erohmensing Jan 3, 2024

alafanechere Jan 3, 2024

erohmensing Jan 3, 2024

alafanechere Jan 3, 2024

postamar Jan 3, 2024

postamar Jan 3, 2024

sentry-io bot commented Jan 3, 2024 •

edited

Loading

	`airbyte-ci tests airbyte-integrations/bases/connector-acceptance-test--poetry-run-command='pytest tests/unit_tests'`
	`airbyte-ci tests airbyte-integrations/bases/connector-acceptance-test--poetry-run-command='pytest unit_tests'`

airbyte-ci: make airbyte-ci test able to run any poetry run command #33784

airbyte-ci: make airbyte-ci test able to run any poetry run command #33784

Conversation

alafanechere commented Dec 26, 2023 • edited Loading

What

How

vercel bot commented Dec 26, 2023 • edited Loading

alafanechere commented Dec 26, 2023 • edited Loading

github-actions bot commented Dec 26, 2023

erohmensing left a comment

Choose a reason for hiding this comment

alafanechere commented Jan 3, 2024

erohmensing commented Jan 3, 2024

erohmensing left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sentry-io bot commented Jan 3, 2024 • edited Loading

Suspect Issues

airbyte-ci: make `airbyte-ci test` able to run any poetry run command #33784

airbyte-ci: make `airbyte-ci test` able to run any poetry run command #33784

alafanechere commented Dec 26, 2023 •

edited

Loading

vercel bot commented Dec 26, 2023 •

edited

Loading

alafanechere commented Dec 26, 2023 •

edited

Loading

sentry-io bot commented Jan 3, 2024 •

edited

Loading