New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

connector base image: declare the base image package and implement #30303

Merged

alafanechere merged 8 commits into master from augustin/declare-first-base-image

Sep 22, 2023

Contributor

alafanechere commented Sep 11, 2023 •

edited

Loading

Closes #30235
Closes #30240

What

We want to build a "framework" to declare versioned base images for our connectors.

Context:

Approach

Declare a new poetry package under airbyte-ci/connectors/base_images
Design and implement a framework to declare base image version in code (see README.md)
Expose a poetry run cut-new-version command that will run sanity checks on base images and publish it.

Please note that no publish on merge automation has been implemented in this PR. I don't think its high priority given that we won't be cutting new images very often.

Next steps

TODO in this PR

Write more tests
Automatically run the test suite in the CI

🚨 User Impact 🚨

None, this package is not yet used.

alafanechere force-pushed the augustin/declare-first-base-image branch from 5a6f52f to ee6791f Compare

September 11, 2023 17:13

alafanechere changed the title ~~connector base image: declare the base image package and implement build~~ connector base image: declare the base image package and implement

alafanechere marked this pull request as ready for review

September 11, 2023 17:28

alafanechere requested a review from a team

September 11, 2023 17:28

alafanechere mentioned this pull request

Source DuckDB: ✨ Add MotherDuck support 🦆🦆 #29428

Merged

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/__init__.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/__init__.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/__init__.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/common.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/python_bases.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/python_bases.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/python_bases.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/python_bases.py Outdated

		raise common.SanityCheckError(f"missing environment variables: {missing_env_vars}")


		class _0_1_0(AirbytePythonConnectorBaseImage):

Contributor Author

alafanechere Sep 11, 2023

TODO: add a check that patch version inherit from previous version

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/python_bases.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/python_bases.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/python_bases.py Outdated

Comment on lines 93 to 98

+                  async def run_sanity_checks(self):
+                      await super().run_sanity_checks()
+                      await self.check_python_version()
+                      await self.check_pip_version()
+                      await self.check_time_zone()
+                      await self.check_bash_is_installed()

Contributor Author

alafanechere Sep 11, 2023

TODO: add docstring explaining this checks are from the results of our analysis

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/utils.py Outdated Show resolved Hide resolved

alafanechere commented

View reviewed changes

airbyte-ci/connectors/base_images/base_images/python_bases.py Outdated Show resolved Hide resolved

octavia-squidington-iii removed area/connectors CDK labels

vercel bot deployed to Preview

September 22, 2023 09:25

View deployment

alafanechere enabled auto-merge (squash)

September 22, 2023 09:27

alafanechere disabled auto-merge

September 22, 2023 09:42

alafanechere removed connectors/source/sentry connectors/source/stripe connectors/source/trello connectors/source/facebook-marketing connectors/destination/bigquery connectors/source/apify-dataset connectors/destination/bigquery-denormalized connectors/source/bigcommerce connectors/destination/snowflake connectors/source/google-ads connectors/source/shopify connectors/source/jira connectors/source/slack connectors/source/mssql connectors/source/mysql connectors/source/paypal-transaction connectors/source/pipedrive connectors/source/postgres connectors/source/zendesk-support connectors/source/google-analytics-data-api connectors/source/mysql-strict-encrypt connectors/destination/duckdb labels

alafanechere merged commit a104721 into master

49 checks passed

alafanechere deleted the augustin/declare-first-base-image branch

September 22, 2023 10:26

jbfbell pushed a commit that referenced this pull request


          connector base image: declare the base image package and implement (#…

660bfa2

…30303)

Co-authored-by: alafanechere <alafanechere@users.noreply.github.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet