connectors-ci: validate the correctness of the images built with the new publish pipeline

[alafanechere]

Tell us about the problem you're trying to solve

If we want to rely on the new publish pipeline (publish on merge to master) we need to triple check the correctness of the images it build and publishes.

I would especially love to validate that:

• The packaging of normalization inside Java destination is correct
• Java sources behave correctly
• Python and low code connector behave correctly
• The image that are push are not significantly bigger than the one pushed with /publish

Can we assume that passing nightly builds on GA connectors on the new test pipelines is enough to guarantee image correctness?

[evantahler]

Grooming:

Extra tests we do:

• Run syncs in multiple architectures
• Add a multi-arch CAT test
• E2E Tests

Worries:

• normalization + destinations combined

Path Forward:

• Build pre-release versions of GA Sources (1 java, 1 python, 1 low-code) and GA destinations (1 java, 1 python)
• Run syncs locally with existing GA publish versions and time them
• Run syncs locally with pre-release + dagger versions and time them
• Be sure to test both a strict encrypt and normal version of a destination
• Be sure to test the destinations with and without normalization

If times match and no bugs, we are good!

[alafanechere]

I sucessfully tested source-facebook-marketing (GA Python connector) with pre-release 0.3.6-dev.e20fb9d994
:

[alafanechere]

Successfully tested destination-bigquery (GA Java destination) with a locally built image:

[alafanechere]

Successfully test source-postgres (GA Java source) with docker pull 2.0.28-dev.b0de4a8dfa

[alafanechere]

Successfully tested source postgres strict encrypt with 2.0.28-dev.26a60c5a08

[alafanechere]

Successfully test destination-postgres-strict-encrypt with airbyte/destination-postgres-strict-encrypt:0.3.27-dev.be6dbae6df

[bnchrch]

Woohoo!

[bnchrch]

Successfully tested a python destination (airbyte/destination-sqlite:0.1.0-dev.780903ff80)

[alafanechere]

Successfully tested a low code CDK source (airbyte/source-greenhouse:0.4.0-dev.2d2493522b)

[alafanechere]

Status

• ✅ successful sync with a Python source: source-facebook-marketing behave correctly
• ✅ successful sync with a Low code CDK source: source-greenhouse behave correctly
• ✅ successful sync with a Python destination: destination-sqlite behave correctly
• ✅ successful sync with a Java sources: source-postgres behave correctly
• ✅ successful sync with a strict encrypt variant: source-postgres-strict-encrypt behave correctly
• ✅ successful sync with a Java destination + the packaging of normalization inside Java destination is correct: destination-bigquery was tested with normalization enabled and behaved correctly.
• ✅ successful sync with a Java destination strict encrypt variant: destination-postgres-strict-encrypt behave correctly
• ✅ After this fix Java connectors image are not significantly bigger compared to the original ones
• 🔴 Because we use a recent Docker version (> 21) on the CI cluster some images are occasionally built with layers compressed with an incompatible format (zstd) for Docker engines version < 20. More details can be found in dagger: corrupted image being published? #26085 . The Dagger team will provide followup to maximize image compatibility, or we may make in infra change on the cluster to downgrade the docker engine.

[alafanechere]

Closing now the publish on merge is live.

Will implement some connector specific fixes (see #26426), but the overall health of the images look correct.