airbyte-ci,gradle: replace airbyte-docker with airbyte-ci

[postamar]

Note that this PR is based on #30607 ; however for what's of interest in this commit those changes are for the most part unrelated.

This PR replaces the airbyte-docker plugin on connectors with a task that execs airbyte-ci to build and load the docker images for each connector. For this purpose I had to mess with airbyte-ci a bit and add a couple of flags which make this tolerably fast.

This is very largely inspired by #28817

Fixes #30403

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 4, 2023 3:18pm

[github-actions]

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

• PR name follows PR naming conventions
• Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
• Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
• You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
• Secrets in the connector's spec are annotated with airbyte_secret
• All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
• Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
• Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
• If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

1. Check for hidden checklists in your PR description

2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

[airbyte-oss-build-runner]

destination-elasticsearch-strict-encrypt test report (commit 87f6a85d38) - ❌

⏲️ Total pipeline duration: 07mn49s

Step	Result
Build connector tar	✅
Build destination-elasticsearch-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	❌
Validate metadata for destination-elasticsearch-strict-encrypt	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-elasticsearch-strict-encrypt test

[airbyte-oss-build-runner]

source-kafka test report (commit 87f6a85d38) - ❌

⏲️ Total pipeline duration: 05mn36s

Step	Result
Build connector tar	✅
Build source-kafka docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate metadata for source-kafka	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-kafka test

[postamar]

I really ran this branch through its paces today on my laptop and found a couple more bugs. This led me to rewrite the BuildConnectorImage* classes to allow more granular platform targeting. I kind of like the end result, in principle at least.

[bnchrch]

The code looks good to me!

Just a few minor comments.

Also im running tests for

-name=destination-azure-blob-storage --name=source-postgres --name=destination-clickhouse

https://github.com/airbytehq/airbyte/actions/runs/6399510931

Lets wait to see these pass (or fail with a valid excuse) before we merge

[airbyte-oss-build-runner]

destination-bigquery-denormalized test report (commit 13479b1947) - ❌

⏲️ Total pipeline duration: 01mn09s

Step	Result
Validate metadata for destination-bigquery-denormalized	✅
Connector version semver check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

[airbyte-oss-build-runner]

destination-tidb test report (commit 13479b1947) - ❌

⏲️ Total pipeline duration: 03mn00s

Step	Result
Build connector tar	✅
Build destination-tidb docker image for platform(s) linux/x86_64	❌
Validate metadata for destination-tidb	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-tidb test

[airbyte-oss-build-runner]

destination-duckdb test report (commit 13479b1947) - ✅

⏲️ Total pipeline duration: 03mn06s

Step	Result
Connector package install	✅
Build destination-duckdb docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

source-file-secure test report (commit 13479b1947) - ❌

⏲️ Total pipeline duration: 04mn14s

Step	Result
Connector package install	✅
Build source-file-secure docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-file-secure	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file-secure test

[postamar]

Thanks again for taking another look. Everything seems as green as can be including that workflow you ran. In addition to what you reviewed I pushed some very trivial changes like requiring python 3.10 across the board instead of 3.9, and setting an env var when calling airbyte-ci to make the logging output prettier. I'll wait for the CI to finish before merging.

[airbyte-oss-build-runner]

destination-bigquery-denormalized test report (commit a42fe26401) - ❌

⏲️ Total pipeline duration: 01mn08s

Step	Result
Validate metadata for destination-bigquery-denormalized	✅
Connector version semver check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

[airbyte-oss-build-runner]

destination-duckdb test report (commit a42fe26401) - ✅

⏲️ Total pipeline duration: 03mn48s

Step	Result
Connector package install	✅
Build destination-duckdb docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

source-file-secure test report (commit a42fe26401) - ❌

⏲️ Total pipeline duration: 04mn22s

Step	Result
Connector package install	✅
Build source-file-secure docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-file-secure	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file-secure test

[airbyte-oss-build-runner]

destination-bigquery-denormalized test report (commit b16da1b134) - ❌

⏲️ Total pipeline duration: 41.75s

Step	Result
Validate metadata for destination-bigquery-denormalized	✅
Connector version semver check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

[airbyte-oss-build-runner]

source-file-secure test report (commit b16da1b134) - ❌

⏲️ Total pipeline duration: 02mn06s

Step	Result
Connector package install	✅
Build source-file-secure docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-file-secure	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file-secure test

[airbyte-oss-build-runner]

destination-duckdb test report (commit b16da1b134) - ✅

⏲️ Total pipeline duration: 02mn32s

Step	Result
Connector package install	✅
Build destination-duckdb docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

destination-exasol test report (commit b16da1b134) - ❌

⏲️ Total pipeline duration: 07mn13s

Step	Result
Build connector tar	✅
Build destination-exasol docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	❌
Validate metadata for destination-exasol	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-exasol test

[airbyte-oss-build-runner]

source-alloydb test report (commit b16da1b134) - ❌

⏲️ Total pipeline duration: 03mn53s

Step	Result
Build connector tar	✅
Build source-alloydb docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	❌
Validate metadata for source-alloydb	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

[postamar]

/approve-and-merge reason="cannot run all connector tests for this PR"