✨ Source S3: v4 rollout by maxi297 · Pull Request #29757 · airbytehq/airbyte

Maxime Carbonneau-Leclerc (maxi297) · 2023-08-23T15:54:18Z

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user?

For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.

If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.0.1
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

github-actions · 2023-08-23T15:54:42Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

github-actions · 2023-08-23T16:07:08Z

source-s3 test report (commit `25136241ee`) - ❌

⏲️ Total pipeline duration: 11mn07s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌
Code format checks	❌
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

github-actions · 2023-08-23T16:48:58Z

source-s3 test report (commit `6af59b683c`) - ❌

⏲️ Total pipeline duration: 11mn01s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

github-actions · 2023-08-23T17:00:26Z

source-s3 test report (commit `6af59b683c`) - ❌

⏲️ Total pipeline duration: 09mn16s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

github-actions · 2023-08-23T21:40:33Z

source-s3 test report (commit `2bf73a928a`) - ❌

⏲️ Total pipeline duration: 11mn11s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

github-actions · 2023-08-25T14:47:35Z

source-s3 test report (commit `ca7f891645`) - ❌

⏲️ Total pipeline duration: 11mn23s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

github-actions · 2023-08-25T16:21:20Z

source-s3 test report (commit `dceec7e847`) - ❌

⏲️ Total pipeline duration: 11mn37s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

github-actions · 2023-08-28T22:54:50Z

source-s3 test report (commit `f6062a18c6`) - ❌

⏲️ Total pipeline duration: 11mn08s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

github-actions · 2023-08-31T20:26:09Z

source-s3 test report (commit `6bc1edf4b4`) - ✅

⏲️ Total pipeline duration: 11mn30s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

This reverts commit 676a3e2.

github-actions · 2023-09-02T14:28:55Z

source-s3 test report (commit `39c5be0ec6`) - ✅

⏲️ Total pipeline duration: 09mn20s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

This reverts commit 7f85689.

This reverts commit 39c5be0.

This reverts commit d0a42e6.

This reverts commit 098b7a0.

github-actions · 2023-09-05T02:45:07Z

source-hubplanner test report (commit `0dc5cbdb07`) - ❌

⏲️ Total pipeline duration: 01mn45s

Step	Result
Connector package install	✅
Build source-hubplanner docker image for platform linux/x86_64	✅
Acceptance tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/source-hubplanner/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubplanner test

Maxime Carbonneau-Leclerc (maxi297) · 2023-09-05T16:15:54Z

Superseded by #30147 to see if we could avoid autoformat

maxi297 added 2 commits August 23, 2023 11:30

Support endpoint and config at the same time

904904d

Set up source s3 for v4 rollout

2513624

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/s3 labels Aug 23, 2023

Maxime Carbonneau-Leclerc (maxi297) mentioned this pull request Aug 23, 2023

✨ Source S3: v4 rollout (old) #29471

Closed

Maxime Carbonneau-Leclerc (maxi297) changed the title ~~Source s3 v4 rollout/v4 rollout~~ ✨ Source S3: v4 rollout Aug 23, 2023

maxi297 added 2 commits August 23, 2023 12:29

update expected records

0d73a49

Formatting

6af59b6

Up CDK version

2bf73a9

Ignore block_size in advanced_options as well

ca7f891

Update airbyte-cdk version

dceec7e

Support CSV user-provided column_names

f6062a1

maxi297 added 3 commits August 30, 2023 15:33

Migration guide

9d81704

Empty commit to trigger pipeline

fbb6ccf

Merge branch 'master' into source-s3-v4-rollout/v4-rollout

6bc1edf

Maxime Carbonneau-Leclerc (maxi297) and others added 2 commits September 2, 2023 10:03

Support config migration for V4 (#30005)

676a3e2

Revert "Support config migration for V4 (#30005)"

39c5be0

This reverts commit 676a3e2.

Source s3 v4 rollout/cats remove v3 (#30146)

7f85689

Maxime Carbonneau-Leclerc (maxi297) requested a review from a team as a code owner September 5, 2023 01:12

Octavia Squidington III (octavia-squidington-iii) added connectors/destination/snowflake connectors/source/harvest connectors/source/lemlist connectors/source/mysql connectors/source/postgres connectors/source/zendesk-support labels Sep 5, 2023

maxi297 added 3 commits September 4, 2023 21:28

Revert "Source s3 v4 rollout/cats remove v3 (#30146)"

bad7cca

This reverts commit 7f85689.

Revert "Revert "Support config migration for V4 (#30005)""

bfcbfa8

This reverts commit 39c5be0.

Update CATs to v4

43a46d4

Octavia Squidington III (octavia-squidington-iii) removed the CDK Connector Development Kit label Sep 5, 2023

Maxime Carbonneau-Leclerc (maxi297) removed connectors/source/harvest connectors/source/lemlist connectors/destination/bigquery connectors/destination/snowflake connectors/source/mysql connectors/source/postgres connectors/source/zendesk-support labels Sep 5, 2023

maxi297 and others added 7 commits September 4, 2023 21:40

Merge branch 'master' into source-s3-v4-rollout/v4-rollout

8dba384

Automated Commit - Formatting Changes

d0a42e6

Retrigger build by changing version

cf155af

Revert "Automated Commit - Formatting Changes"

9ee45b6

This reverts commit d0a42e6.

Automated Commit - Formatting Changes

098b7a0

Revert "Automated Commit - Formatting Changes"

b642316

This reverts commit 098b7a0.

Automated Commit - Formatting Changes

0dc5cbd

Empty commit to trigger pipeline

5233009

Octavia Squidington III (octavia-squidington-iii) added the connectors/source/hubplanner label Sep 5, 2023

Maxime Carbonneau-Leclerc (maxi297) mentioned this pull request Sep 5, 2023

✨ Source S3: v4 rollout - take 2 #30147

Closed

Maxime Carbonneau-Leclerc (maxi297) closed this Sep 5, 2023

Navigation Menu

Repository navigation

✨ Source S3: v4 rollout - #29757

✨ Source S3: v4 rollout#29757
Maxime Carbonneau-Leclerc (maxi297) wants to merge 25 commits into
masterfrom
source-s3-v4-rollout/v4-rollout

Maxime Carbonneau-Leclerc (maxi297) commented Aug 23, 2023

Uh oh!

github-actions Bot commented Aug 23, 2023 •

edited by maxi297

Loading

Uh oh!

github-actions Bot commented Aug 23, 2023

Uh oh!

github-actions Bot commented Aug 23, 2023

Uh oh!

github-actions Bot commented Aug 23, 2023

Uh oh!

github-actions Bot commented Aug 23, 2023

Uh oh!

github-actions Bot commented Aug 25, 2023

Uh oh!

github-actions Bot commented Aug 25, 2023

Uh oh!

github-actions Bot commented Aug 28, 2023

Uh oh!

github-actions Bot commented Aug 31, 2023

Uh oh!

github-actions Bot commented Sep 2, 2023

Uh oh!

github-actions Bot commented Sep 5, 2023

Uh oh!

Maxime Carbonneau-Leclerc (maxi297) commented Sep 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Maxime Carbonneau-Leclerc (maxi297) commented Aug 23, 2023

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Uh oh!

github-actions Bot commented Aug 23, 2023 • edited by maxi297 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

Uh oh!

github-actions Bot commented Aug 23, 2023

source-s3 test report (commit 25136241ee) - ❌

Uh oh!

github-actions Bot commented Aug 23, 2023

source-s3 test report (commit 6af59b683c) - ❌

Uh oh!

github-actions Bot commented Aug 23, 2023

source-s3 test report (commit 6af59b683c) - ❌

Uh oh!

github-actions Bot commented Aug 23, 2023

source-s3 test report (commit 2bf73a928a) - ❌

Uh oh!

github-actions Bot commented Aug 25, 2023

source-s3 test report (commit ca7f891645) - ❌

Uh oh!

github-actions Bot commented Aug 25, 2023

source-s3 test report (commit dceec7e847) - ❌

Uh oh!

github-actions Bot commented Aug 28, 2023

source-s3 test report (commit f6062a18c6) - ❌

Uh oh!

github-actions Bot commented Aug 31, 2023

source-s3 test report (commit 6bc1edf4b4) - ✅

Uh oh!

github-actions Bot commented Sep 2, 2023

source-s3 test report (commit 39c5be0ec6) - ✅

Uh oh!

github-actions Bot commented Sep 5, 2023

source-hubplanner test report (commit 0dc5cbdb07) - ❌

Uh oh!

Maxime Carbonneau-Leclerc (maxi297) commented Sep 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Aug 23, 2023 •

edited by maxi297

Loading

source-s3 test report (commit `25136241ee`) - ❌

source-s3 test report (commit `6af59b683c`) - ❌

source-s3 test report (commit `6af59b683c`) - ❌

source-s3 test report (commit `2bf73a928a`) - ❌

source-s3 test report (commit `ca7f891645`) - ❌

source-s3 test report (commit `dceec7e847`) - ❌

source-s3 test report (commit `f6062a18c6`) - ❌

source-s3 test report (commit `6bc1edf4b4`) - ✅

source-s3 test report (commit `39c5be0ec6`) - ✅

source-hubplanner test report (commit `0dc5cbdb07`) - ❌