Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

connectors-ci: validate the correctness of the images built with the new publish pipeline #25506

Closed
alafanechere opened this issue Apr 25, 2023 · 11 comments
Assignees
Labels

Comments

@alafanechere
Copy link
Contributor

Tell us about the problem you're trying to solve

If we want to rely on the new publish pipeline (publish on merge to master) we need to triple check the correctness of the images it build and publishes.

I would especially love to validate that:

  • The packaging of normalization inside Java destination is correct
  • Java sources behave correctly
  • Python and low code connector behave correctly
  • The image that are push are not significantly bigger than the one pushed with /publish

Can we assume that passing nightly builds on GA connectors on the new test pipelines is enough to guarantee image correctness?

@evantahler
Copy link
Contributor

Grooming:

Extra tests we do:

  • Run syncs in multiple architectures
  • Add a multi-arch CAT test
  • E2E Tests

Worries:

  • normalization + destinations combined

Path Forward:

  • Build pre-release versions of GA Sources (1 java, 1 python, 1 low-code) and GA destinations (1 java, 1 python)
  • Run syncs locally with existing GA publish versions and time them
  • Run syncs locally with pre-release + dagger versions and time them
  • Be sure to test both a strict encrypt and normal version of a destination
  • Be sure to test the destinations with and without normalization

If times match and no bugs, we are good!

@alafanechere
Copy link
Contributor Author

I sucessfully tested source-facebook-marketing (GA Python connector) with pre-release 0.3.6-dev.e20fb9d994
:
Screen Shot 2023-05-06 at 11.47.53.png

@alafanechere
Copy link
Contributor Author

Successfully tested destination-bigquery (GA Java destination) with a locally built image:
Screen Shot 2023-05-09 at 16.15.04.png

@alafanechere
Copy link
Contributor Author

Successfully test source-postgres (GA Java source) with docker pull 2.0.28-dev.b0de4a8dfa
Screen Shot 2023-05-12 at 17 48 02

@alafanechere
Copy link
Contributor Author

Successfully tested source postgres strict encrypt with 2.0.28-dev.26a60c5a08
Screen Shot 2023-05-15 at 15 16 18

@alafanechere
Copy link
Contributor Author

Successfully test destination-postgres-strict-encrypt with airbyte/destination-postgres-strict-encrypt:0.3.27-dev.be6dbae6df
Screen Shot 2023-05-15 at 22 50 39

@bnchrch
Copy link
Contributor

bnchrch commented May 15, 2023

Woohoo!

@bnchrch
Copy link
Contributor

bnchrch commented May 15, 2023

Successfully tested a python destination (airbyte/destination-sqlite:0.1.0-dev.780903ff80)

image

@alafanechere
Copy link
Contributor Author

Successfully tested a low code CDK source (airbyte/source-greenhouse:0.4.0-dev.2d2493522b)

Screen Shot 2023-05-16 at 10 16 08

@alafanechere
Copy link
Contributor Author

Status

  • ✅ successful sync with a Python source: source-facebook-marketing behave correctly
  • ✅ successful sync with a Low code CDK source: source-greenhouse behave correctly
  • ✅ successful sync with a Python destination: destination-sqlite behave correctly
  • ✅ successful sync with a Java sources: source-postgres behave correctly
  • ✅ successful sync with a strict encrypt variant: source-postgres-strict-encrypt behave correctly
  • ✅ successful sync with a Java destination + the packaging of normalization inside Java destination is correct: destination-bigquery was tested with normalization enabled and behaved correctly.
  • ✅ successful sync with a Java destination strict encrypt variant: destination-postgres-strict-encrypt behave correctly
  • ✅ After this fix Java connectors image are not significantly bigger compared to the original ones
  • 🔴 Because we use a recent Docker version (> 21) on the CI cluster some images are occasionally built with layers compressed with an incompatible format (zstd) for Docker engines version < 20. More details can be found in dagger: corrupted image being published? #26085 . The Dagger team will provide followup to maximize image compatibility, or we may make in infra change on the cluster to downgrade the docker engine.

@alafanechere
Copy link
Contributor Author

Closing now the publish on merge is live.

Will implement some connector specific fixes (see #26426), but the overall health of the images look correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants