Multi-architecture normalization build (local)

[evantahler]

When building and testing normalization locally, we need to force the base images to match the local host OS.

This is not a problem when publishing the connectors as airbyte-ci/dagger handles this for us

[github-actions]

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

• Run integration tests
• Bump connector or module version
• Add changelog
• Publish the new version

✅ Sources (0)

Connector	Version	Changelog	Publish

• See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (15)

Connector	Version	Changelog	Publish
destination-bigquery	1.4.4	✅	✅
destination-bigquery-denormalized	1.4.1	✅	✅
destination-clickhouse	0.2.3	✅	✅
destination-clickhouse-strict-encrypt	0.2.3	🔵 (ignored)	🔵 (ignored)
destination-mssql	0.1.23	✅	✅
destination-mssql-strict-encrypt	0.1.23	🔵 (ignored)	🔵 (ignored)
destination-mysql	0.1.20	✅	✅
destination-mysql-strict-encrypt	❌ 0.1.21 (mismatch: 0.1.20)	🔵 (ignored)	🔵 (ignored)
destination-oracle	0.1.19	✅	✅
destination-oracle-strict-encrypt	0.1.19	🔵 (ignored)	🔵 (ignored)
destination-postgres	0.3.27	✅	✅
destination-postgres-strict-encrypt	0.3.27	🔵 (ignored)	🔵 (ignored)
destination-redshift	0.4.8	✅	✅
destination-snowflake	1.0.4	✅	✅
destination-tidb	0.1.1	✅	❌ (diff seed version)

• See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

• base-normalization

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled.

[evantahler]

There may also be a slight speed improvement on building these images now that they aren't emulating some other arch too...

[edgao]

nice. this has no impact on prod, right? because /publish doesn't go through this path anymore

[evantahler]

nice. this has no impact on prod, right? because /publish doesn't go through this path anymore

I think this should be OK for 2 reasons:

• we build and test on AMD machines, so there shouldn't be a change to the images we produce
• yep, manage.sh is what we use for publishing (not the new pipelines yet), and that rebuilds the normalization containers in different architectures anyway https://github.com/airbytehq/airbyte/blob/master/tools/integrations/manage.sh#L237-L243

[davinchia]

@evantahler was this not detecting the correct local arch before this change?

I ran into this because I was attempting to build an amd image on my arm machine. I was doing so by setting DOCKER_BUILD_PLATFORM from here. However this was failing as the normalization builds only build arm images from this change onwards.

[evantahler]

no... because the base image we use is so old, they only ever published AMD images... so any image built on it remains AMD unless you do something drastic... like this! Anyway, all of this is going away soon so 🤷

[evantahler]

You can update the changes if you want to look at the ENV DOCKER_BUILD_PLATFORM first, and if that's missing, use the local arch, that would be fine