✨ Source S3: v4 rollout/feature parity by maxi297 · Pull Request #29753 · airbytehq/airbyte

Maxime Carbonneau-Leclerc (maxi297) · 2023-08-23T13:57:04Z

What

Missing features for releasing source-s3 v4 (either on the CDK or source-s3 specifically)

Each commit represent a feature:

(CATs) have schema within catalog like we would in the platform
CSV to drop fields that are not in schema
CSV null_values, true_values and false_values default does not align with pyarrow
ContinuationToken is not passed when fetch S3 objects hence pagination does not work
Support both endpoint and aws access key

Last commit is to update the version

…align with pyarrow

…schema

github-actions · 2023-08-23T13:57:29Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

Alexandre Girard (girarda)

🪨 🎸

Alexandre Girard (girarda) · 2023-08-23T13:57:39Z

                    except ValueError:
                        warnings.append(_format_warning(key, value, prop_type))

+                result[key] = cast_value


Alexandre Girard (girarda) · 2023-08-23T13:58:05Z

-                raise ValidationError(
-                    "Either `aws_access_key_id` and `aws_secret_access_key` or `endpoint` must be set, but not both.", model=Config
-                )
+        if (aws_access_key_id or aws_secret_access_key) and not (aws_access_key_id and aws_secret_access_key):


Alexandre Girard (girarda) · 2023-08-23T13:58:19Z

-                "true_values": ["y", "yes", "t", "true", "on", "1"],
-                "false_values": ["n", "no", "f", "false", "off", "0"],
+                # values taken from https://github.com/apache/arrow/blob/43c05c56b37daa93e76b94bc3e6952d56d1ea3f2/cpp/src/arrow/csv/options.cc#L41-L45
+                "null_values": additional_reader_options.pop("null_values", [


Alexandre Girard (girarda) · 2023-08-23T13:58:45Z

        kwargs = {"Bucket": bucket}
        while True:
-            response = s3.list_objects_v2(Bucket=bucket, Prefix=prefix) if prefix else s3.list_objects_v2(Bucket=bucket)
+            response = s3.list_objects_v2(Prefix=prefix, **kwargs) if prefix else s3.list_objects_v2(**kwargs)


github-actions · 2023-08-23T14:13:46Z

source-s3 test report (commit `5e2cf2415c`) - ❌

⏲️ Total pipeline duration: 14mn23s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	❌
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

github-actions · 2023-08-23T14:28:14Z

source-s3 test report (commit `4f2471e87c`) - ❌

⏲️ Total pipeline duration: 12mn05s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	❌
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

github-actions · 2023-08-23T14:43:02Z

source-s3 test report (commit `3f275d51c8`) - ✅

⏲️ Total pipeline duration: 13mn03s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

github-actions · 2023-08-23T14:57:31Z

source-s3 test report (commit `46c845f9d5`) - ❌

⏲️ Total pipeline duration: 12mn43s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	❌
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

github-actions · 2023-08-23T15:10:34Z

source-s3 test report (commit `37a7411245`) - ❌

⏲️ Total pipeline duration: 11mn17s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	❌
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

github-actions · 2023-08-23T15:28:45Z

source-s3 test report (commit `52081134f6`) - ✅

⏲️ Total pipeline duration: 05mn49s

Step	Result
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

maxi297 added 5 commits August 23, 2023 09:39

Support endpoint and config at the same time

a887fae

File-based CSV: do not add fields if they are not part of the schema

e4978f0

File-based CSV: null_values, true_values and false_values default to …

5a75d13

…align with pyarrow

Fix pagination on S3 V4

5af9d13

Update configured catalogs to align with the platform as it provides …

5e2cf24

…schema

Maxime Carbonneau-Leclerc (maxi297) requested a review from a team August 23, 2023 13:57

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues CDK Connector Development Kit connectors/source/s3 labels Aug 23, 2023

Maxime Carbonneau-Leclerc (maxi297) requested review from Alexandre Girard (girarda) and removed request for a team August 23, 2023 13:57

Maxime Carbonneau-Leclerc (maxi297) changed the title ~~Source s3 v4 rollout/feature parity~~ ✨ Source S3: v4 rollout/feature parity Aug 23, 2023

source-s3 release documentation/version bump

4f2471e

Maxime Carbonneau-Leclerc (maxi297) added checklist-action-run and removed checklist-action-run labels Aug 23, 2023

Octavia Squidington III (octavia-squidington-iii) added the area/documentation Improvements or additions to documentation label Aug 23, 2023

Alexandre Girard (girarda) approved these changes Aug 23, 2023

View reviewed changes

formatting

3f275d5

formatting

46c845f

Fix CDK tests

37a7411

Formatting

5208113

Maxime Carbonneau-Leclerc (maxi297) merged commit 40b76a7 into master Aug 23, 2023

Maxime Carbonneau-Leclerc (maxi297) deleted the source-s3-v4-rollout/feature-parity branch August 23, 2023 15:30

Harry Touloupas (harrytou) pushed a commit to KYVENetwork/airbyte that referenced this pull request Sep 1, 2023

✨ Source S3: v4 rollout/feature parity (airbytehq#29753)

3c4bb97

Navigation Menu

Repository navigation

✨ Source S3: v4 rollout/feature parity - #29753

✨ Source S3: v4 rollout/feature parity#29753
Maxime Carbonneau-Leclerc (maxi297) merged 10 commits into
masterfrom
source-s3-v4-rollout/feature-parity

Maxime Carbonneau-Leclerc (maxi297) commented Aug 23, 2023 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 23, 2023 •

edited by maxi297

Loading

Uh oh!

Alexandre Girard (girarda) left a comment

Uh oh!

Alexandre Girard (girarda) Aug 23, 2023

Uh oh!

Alexandre Girard (girarda) Aug 23, 2023

Uh oh!

Alexandre Girard (girarda) Aug 23, 2023

Uh oh!

Alexandre Girard (girarda) Aug 23, 2023

Uh oh!

github-actions Bot commented Aug 23, 2023

Uh oh!

github-actions Bot commented Aug 23, 2023

Uh oh!

github-actions Bot commented Aug 23, 2023

Uh oh!

github-actions Bot commented Aug 23, 2023

Uh oh!

github-actions Bot commented Aug 23, 2023

Uh oh!

github-actions Bot commented Aug 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Maxime Carbonneau-Leclerc (maxi297) commented Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Uh oh!

github-actions Bot commented Aug 23, 2023 • edited by maxi297 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

Uh oh!

Alexandre Girard (girarda) left a comment

Choose a reason for hiding this comment

Uh oh!

Alexandre Girard (girarda) Aug 23, 2023

Choose a reason for hiding this comment

Uh oh!

Alexandre Girard (girarda) Aug 23, 2023

Choose a reason for hiding this comment

Uh oh!

Alexandre Girard (girarda) Aug 23, 2023

Choose a reason for hiding this comment

Uh oh!

Alexandre Girard (girarda) Aug 23, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 23, 2023

source-s3 test report (commit 5e2cf2415c) - ❌

Uh oh!

github-actions Bot commented Aug 23, 2023

source-s3 test report (commit 4f2471e87c) - ❌

Uh oh!

github-actions Bot commented Aug 23, 2023

source-s3 test report (commit 3f275d51c8) - ✅

Uh oh!

github-actions Bot commented Aug 23, 2023

source-s3 test report (commit 46c845f9d5) - ❌

Uh oh!

github-actions Bot commented Aug 23, 2023

source-s3 test report (commit 37a7411245) - ❌

Uh oh!

github-actions Bot commented Aug 23, 2023

source-s3 test report (commit 52081134f6) - ✅

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Maxime Carbonneau-Leclerc (maxi297) commented Aug 23, 2023 •

edited

Loading

github-actions Bot commented Aug 23, 2023 •

edited by maxi297

Loading

source-s3 test report (commit `5e2cf2415c`) - ❌

source-s3 test report (commit `4f2471e87c`) - ❌

source-s3 test report (commit `3f275d51c8`) - ✅

source-s3 test report (commit `46c845f9d5`) - ❌

source-s3 test report (commit `37a7411245`) - ❌

source-s3 test report (commit `52081134f6`) - ✅