Fix `destination-s3` build #27786

evantahler · 2023-06-27T22:11:01Z

Reference for failing build https://github.com/airbytehq/airbyte/actions/runs/5387253975

Locally, airbyte-ci connectors --name=destination-s3 build does work fine...

╭─────────────────────────────────────────────────────────────────────────────────────────────────────── DESTINATION-S3 - BUILD RESULTS ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                        Steps results                                                                                                                                                                                         │
│ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓                                                                                                                                                 │
│ ┃ Step                                                       ┃ Result     ┃ Finished after ┃                                                                                                                                                 │
│ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩                                                                                                                                                 │
│ │ Build destination-s3 docker image for platform linux/amd64 │ Successful │ 79s            │                                                                                                                                                 │
│ │ Build destination-s3 docker image for platform linux/arm64 │ Successful │ 23s            │                                                                                                                                                 │
│ │ Load airbyte/destination-s3:dev to the local docker host.  │ Successful │ 0s             │                                                                                                                                                 │
│ └────────────────────────────────────────────────────────────┴────────────┴────────────────┘                                                                                                                                                 │
╰───────────────────────────────────────────────────────────────────────────────────────── ⏲️  Total pipeline duration for destination-s3: 412 seconds ─────────────────────────────────────────────────────────────────────────────────────────╯
[14:52:03] INFO     Build: {"connector_technical_name": "destination-s3", "connector_version": "0.5.0", "run_timestamp": "2023-06-27T21:52:03.101838", "run_duration": 412.072144, "success": true, "failed_steps": [],          contexts.py:401
                    "successful_steps": ["BuildConnectorImage", "BuildConnectorImage", "LoadContainerToLocalDockerHost"], "skipped_steps": [], "gha_workflow_run_url": null, "pipeline_start_timestamp": 1687927511,
                    "pipeline_end_timestamp": 1687927923, "pipeline_duration": 412, "git_branch": "evan/destination-s3-build", "git_revision": "fd34fe3e2b3c037bd686e9a270dc91e881aeb826", "ci_context": "manual"}

github-actions · 2023-06-27T22:11:22Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
The connector tests are passing in CI
You've updated the connector's metadata.yaml file (new!)
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

edgao

lgtm pending pr checks I guess? augustin's comment said the sdk install java 8.0.342-librca failed, so if that's not happening then we should be good to go 🤷

octavia-squidington-iii · 2023-06-27T22:32:53Z

destination-s3 test report (commit `8e0f8874ec`) - ✅

⏲️ Total pipeline duration: 908 seconds

Step	Result
Validate airbyte-integrations/connectors/destination-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-s3 test

octavia-squidington-iii · 2023-06-27T23:16:52Z

destination-s3 test report (commit `ef6b83828e`) - ✅

⏲️ Total pipeline duration: 826 seconds

Step	Result
Validate airbyte-integrations/connectors/destination-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-s3 test

evantahler · 2023-06-27T23:42:05Z

airbyte-integrations/bases/base-normalization/build.gradle

@@ -59,7 +59,7 @@ def buildAirbyteDocker(String customConnector) {
    }

    def baseCommand = ['docker', 'buildx', 'build', '--load', '--platform', arch, '-f', getDockerfile(customConnector), '-t', getImageNameWithTag(customConnector), '.']
-    println("Building normalization container: " + baseCommand.join(" "))
+    // println("Building normalization container: " + baseCommand.join(" "))


unrelated to this PR, but I was annoyed. I added this log statement in #26677, but I didn't realize it would log on every gradle run, even those that have nothing to do with normalization 😓

octavia-squidington-iii · 2023-06-28T00:08:36Z

destination-s3 test report (commit `5ea95c0ae0`) - ✅

⏲️ Total pipeline duration: 866 seconds

Step	Result
Validate airbyte-integrations/connectors/destination-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-s3 test

evantahler · 2023-06-28T00:15:34Z

cc @alafanechere
I'm going to merge this PR, as we are getting green test reports for the connector (yay!). However, there's something odd with the test reporting:

The green test reports link to the latest commmits
Connectors tests / Connectors CI (pull_request) seem to be waiting on a runner for the test
The event timing is strange in the workflow report. It says it ran for 1h 34min, but the action within only took 15min.

Screenshots from right before my merge:

octavia-squidington-iii · 2023-06-28T00:46:24Z

destination-s3 test report (commit `dab48992ef`) - ❌

⏲️ Total pipeline duration: 912 seconds

Step	Result
Validate airbyte-integrations/connectors/destination-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Build connector tar	✅
Build destination-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-s3 test

bump version

0d9f729

evantahler marked this pull request as ready for review June 27, 2023 22:11

evantahler requested a review from a team as a code owner June 27, 2023 22:11

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Jun 27, 2023

octavia-squidington-iii added the connectors/destination/s3 label Jun 27, 2023

PR id

8e0f887

evantahler changed the title ~~Fix destination-s3 build.~~ Fix destination-s3 build Jun 27, 2023

shh normalization, shh

9fab086

edgao approved these changes Jun 27, 2023

View reviewed changes

remove a bunch of arm64 deps?

ef6b838

octavia-squidington-iii added the normalization label Jun 27, 2023

evantahler requested a review from alafanechere June 27, 2023 23:39

might as well match the dockerfile

5ea95c0

evantahler commented Jun 27, 2023

View reviewed changes

evantahler merged commit 4fb1f98 into master Jun 28, 2023
17 checks passed

evantahler deleted the evan/destination-s3-build branch June 28, 2023 00:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `destination-s3` build #27786

Fix `destination-s3` build #27786

evantahler commented Jun 27, 2023 •

edited

Loading

github-actions bot commented Jun 27, 2023 •

edited by evantahler

Loading

edgao left a comment

octavia-squidington-iii commented Jun 27, 2023

octavia-squidington-iii commented Jun 27, 2023

evantahler Jun 27, 2023

octavia-squidington-iii commented Jun 28, 2023

evantahler commented Jun 28, 2023

octavia-squidington-iii commented Jun 28, 2023

Fix destination-s3 build #27786

Fix destination-s3 build #27786

Conversation

evantahler commented Jun 27, 2023 • edited Loading

github-actions bot commented Jun 27, 2023 • edited by evantahler Loading

Before Merging a Connector Pull Request

edgao left a comment

Choose a reason for hiding this comment

octavia-squidington-iii commented Jun 27, 2023

destination-s3 test report (commit 8e0f8874ec) - ✅

octavia-squidington-iii commented Jun 27, 2023

destination-s3 test report (commit ef6b83828e) - ✅

evantahler Jun 27, 2023

Choose a reason for hiding this comment

octavia-squidington-iii commented Jun 28, 2023

destination-s3 test report (commit 5ea95c0ae0) - ✅

evantahler commented Jun 28, 2023

octavia-squidington-iii commented Jun 28, 2023

destination-s3 test report (commit dab48992ef) - ❌

Fix `destination-s3` build #27786

Fix `destination-s3` build #27786

evantahler commented Jun 27, 2023 •

edited

Loading

github-actions bot commented Jun 27, 2023 •

edited by evantahler

Loading

destination-s3 test report (commit `8e0f8874ec`) - ✅

destination-s3 test report (commit `ef6b83828e`) - ✅

destination-s3 test report (commit `5ea95c0ae0`) - ✅

destination-s3 test report (commit `dab48992ef`) - ❌