Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

airbyte-ci: group shell commands for better layering #30279

Merged
merged 4 commits into from
Sep 11, 2023

Conversation

postamar
Copy link
Contributor

@postamar postamar commented Sep 8, 2023

This PR introduces a sh_dash_c utility function to pass to .with_exec to avoid chaining these in cases where we actually want the commands to pass or fail atomically.

The commands are grouped together by &&-ing them and the group is prefixed with a set -o xtrace which makes the logs easier to understand.

Fixes #29735

@postamar postamar force-pushed the postamar/fix-airbyte-ci-caching-errors branch from ce90c8d to 78e029a Compare September 8, 2023 13:45
Copy link
Contributor

@alafanechere alafanechere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very neat change, thank you!
Approving
Could you please:

  • bump the version in pyproject.toml
  • add a changelog entry in the README
  • add a very small unit test in airbyte-ci/connectors/pipelines/tests/test_utils.py to test sh_dash_c

]
)
)
.with_env_variable("VERSION", "24.0.2")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use the constant that determine the docker version we use here (DOCKER_VERSION)?
It will make more obvious that VERSION means docker version :)
https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/consts.py#L29

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes please

@postamar postamar merged commit ea01af7 into master Sep 11, 2023
19 checks passed
@postamar postamar deleted the postamar/fix-airbyte-ci-caching-errors branch September 11, 2023 15:59
@postamar
Copy link
Contributor Author

Thanks for the review! Addressed all comments.

@sentry-io
Copy link

sentry-io bot commented Sep 13, 2023

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ ImportError: cannot import name 'PYTHON_DOCKER_VERSION' from 'pipelines.consts' (/home/runner/.local/pipx/venv... pipelines.commands.groups.tests in <module> View Issue

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

airbyte-ci: apt-get update should exec in same layer as apt-get install
2 participants