Skip to content

Source S3: handle legacy path_prefix + path_patterns - #29382

Merged
Catherine Noll (clnoll) merged 4 commits into
masterfrom
source-s3-v3-v4-file-match-comparison
Aug 15, 2023
Merged

Source S3: handle legacy path_prefix + path_patterns#29382
Catherine Noll (clnoll) merged 4 commits into
masterfrom
source-s3-v3-v4-file-match-comparison

Conversation

@clnoll

@clnoll Catherine Noll (clnoll) commented Aug 12, 2023

Copy link
Copy Markdown
Contributor

This is the source-s3 update corresponding with #29389.

@github-actions

Copy link
Copy Markdown
Contributor

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-s3 test report (commit aab94d3004) - ❌

⏲️ Total pipeline duration: 04mn09s

Step Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml
Connector version semver check
Connector version increment check
QA checks
Code format checks
Connector package install
Build source-s3 docker image for platform linux/x86_64
Unit tests

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-gcs test report (commit 4d03beb76b) - ✅

⏲️ Total pipeline duration: 12mn09s

Step Result
Validate airbyte-integrations/connectors/destination-gcs/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-gcs docker image for platform linux/x86_64
./gradlew :airbyte-integrations:connectors:destination-gcs:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-gcs test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-mongodb-strict-encrypt test report (commit 4d03beb76b) - ✅

⏲️ Total pipeline duration: 05mn50s

Step Result
Validate airbyte-integrations/connectors/destination-mongodb-strict-encrypt/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-mongodb-strict-encrypt docker image for platform linux/x86_64
./gradlew :airbyte-integrations:connectors:destination-mongodb-strict-encrypt:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mongodb-strict-encrypt test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-bigquery test report (commit 4d03beb76b) - ✅

⏲️ Total pipeline duration: 34mn10s

Step Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-bigquery docker image for platform linux/x86_64
Build airbyte/normalization:dev
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-mariadb-columnstore test report (commit 4d03beb76b) - ❌

⏲️ Total pipeline duration: 90mn51s

Step Result
Validate airbyte-integrations/connectors/destination-mariadb-columnstore/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-mariadb-columnstore docker image for platform linux/x86_64
./gradlew :airbyte-integrations:connectors:destination-mariadb-columnstore:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mariadb-columnstore test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-redis test report (commit 4d03beb76b) - ❌

⏲️ Total pipeline duration: 06mn33s

Step Result
Validate airbyte-integrations/connectors/destination-redis/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-redis docker image for platform linux/x86_64
./gradlew :airbyte-integrations:connectors:destination-redis:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redis test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-selectdb test report (commit 4d03beb76b) - ❌

⏲️ Total pipeline duration: 20mn18s

Step Result
Validate airbyte-integrations/connectors/destination-selectdb/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-selectdb docker image for platform linux/x86_64
./gradlew :airbyte-integrations:connectors:destination-selectdb:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-selectdb test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-snowflake test report (commit 4d03beb76b) - ❌

⏲️ Total pipeline duration: 44mn56s

Step Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-snowflake docker image for platform linux/x86_64
Build airbyte/normalization-snowflake:dev
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-mssql test report (commit 4d03beb76b) - ❌

⏲️ Total pipeline duration: 304mn29s

Step Result
Validate airbyte-integrations/connectors/destination-mssql/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-mssql docker image for platform linux/x86_64
Build airbyte/normalization-mssql:dev
./gradlew :airbyte-integrations:connectors:destination-mssql:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mssql test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-mysql test report (commit 4d03beb76b) - ❌

⏲️ Total pipeline duration: 42mn24s

Step Result
Validate airbyte-integrations/connectors/destination-mysql/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-mysql docker image for platform linux/x86_64
Build airbyte/normalization-mysql:dev
./gradlew :airbyte-integrations:connectors:destination-mysql:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mysql test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-pulsar test report (commit 4d03beb76b) - ✅

⏲️ Total pipeline duration: 01mn48s

Step Result
Validate airbyte-integrations/connectors/destination-pulsar/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-pulsar docker image for platform linux/x86_64
./gradlew :airbyte-integrations:connectors:destination-pulsar:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pulsar test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-oracle-strict-encrypt test report (commit 4d03beb76b) - ❌

⏲️ Total pipeline duration: 34mn05s

Step Result
Validate airbyte-integrations/connectors/destination-oracle-strict-encrypt/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-oracle-strict-encrypt docker image for platform linux/x86_64
Build airbyte/normalization-oracle:dev
./gradlew :airbyte-integrations:connectors:destination-oracle-strict-encrypt:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-oracle-strict-encrypt test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-mongodb test report (commit 4d03beb76b) - ❌

⏲️ Total pipeline duration: 17mn11s

Step Result
Validate airbyte-integrations/connectors/destination-mongodb/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-mongodb docker image for platform linux/x86_64
./gradlew :airbyte-integrations:connectors:destination-mongodb:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mongodb test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-rockset test report (commit 4d03beb76b) - ❌

⏲️ Total pipeline duration: 08mn42s

Step Result
Validate airbyte-integrations/connectors/destination-rockset/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-rockset docker image for platform linux/x86_64
./gradlew :airbyte-integrations:connectors:destination-rockset:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-rockset test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-postgres test report (commit 4d03beb76b) - ❌

⏲️ Total pipeline duration: 31mn09s

Step Result
Validate airbyte-integrations/connectors/destination-postgres/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-postgres docker image for platform linux/x86_64
Build airbyte/normalization:dev
./gradlew :airbyte-integrations:connectors:destination-postgres:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-postgres test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-iceberg test report (commit 4d03beb76b) - ❌

⏲️ Total pipeline duration: 304mn14s

Step Result
Validate airbyte-integrations/connectors/destination-iceberg/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-iceberg docker image for platform linux/x86_64
./gradlew :airbyte-integrations:connectors:destination-iceberg:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-iceberg test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-pubsub test report (commit 4d03beb76b) - ✅

⏲️ Total pipeline duration: 16mn16s

Step Result
Validate airbyte-integrations/connectors/destination-pubsub/metadata.yaml
Connector version semver check
QA checks
Build connector tar
Build destination-pubsub docker image for platform linux/x86_64
./gradlew :airbyte-integrations:connectors:destination-pubsub:integrationTest

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pubsub test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-s3 test report (commit fbaa265019) - ❌

⏲️ Total pipeline duration: 02mn47s

Step Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml
Connector version semver check
Connector version increment check
QA checks
Code format checks
Connector package install
Build source-s3 docker image for platform linux/x86_64
Unit tests

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

@brianjlai Brian Lai (brianjlai) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. presumably the tests will pass once the CDK change that this depends on is merged in

return [path_prefix + path_pattern]
return [path_pattern]
def _create_globs(cls, path_pattern: str) -> List[str]:
if "|" in path_pattern:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's add one test case that we separate legacy glob input with the the or separators and confirm the resulting glob is a list

@clnoll
Catherine Noll (clnoll) force-pushed the file-cdk-v3-v4-file-match-comparison branch 2 times, most recently from d7dad51 to 7b95e0e Compare August 15, 2023 13:18
Base automatically changed from file-cdk-v3-v4-file-match-comparison to master August 15, 2023 16:18
@clnoll
Catherine Noll (clnoll) force-pushed the source-s3-v3-v4-file-match-comparison branch from fbaa265 to e180267 Compare August 15, 2023 18:19
@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-s3 test report (commit e180267cd6) - ❌

⏲️ Total pipeline duration: 03mn32s

Step Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml
Connector version semver check
Connector version increment check
QA checks
Code format checks
Connector package install
Build source-s3 docker image for platform linux/x86_64
Unit tests

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-s3 test report (commit 5494420adb) - ❌

⏲️ Total pipeline duration: 03mn00s

Step Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml
Connector version semver check
Connector version increment check
QA checks
Code format checks
Connector package install
Build source-s3 docker image for platform linux/x86_64
Unit tests

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-s3 test report (commit b3f498b80b) - ✅

⏲️ Total pipeline duration: 13mn43s

Step Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml
Connector version semver check
Connector version increment check
QA checks
Code format checks
Connector package install
Build source-s3 docker image for platform linux/x86_64
Unit tests
Integration tests
Acceptance tests

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/azure-blob-storage connectors/destination/bigquery connectors/destination/bigquery-denormalized connectors/destination/cassandra connectors/destination/clickhouse connectors/destination/clickhouse-strict-encrypt connectors/destination/csv connectors/destination/databricks connectors/destination/dev-null connectors/destination/doris connectors/destination/dynamodb connectors/destination/e2e-test connectors/destination/elasticsearch connectors/destination/elasticsearch-strict-encryp connectors/destination/exasol connectors/destination/gcs connectors/destination/iceberg connectors/destination/kafka connectors/destination/keen connectors/destination/kinesis connectors/destination/local-json connectors/destination/mariadb-columnstore connectors/destination/mongodb connectors/destination/mongodb-strict-encrypt connectors/destination/mqtt connectors/destination/mssql connectors/destination/mssql-strict-encrypt connectors/destination/mysql connectors/destination/mysql-strict-encrypt connectors/destination/oracle connectors/destination/oracle-strict-encrypt connectors/destination/postgres connectors/destination/postgres-strict-encrypt connectors/destination/pubsub connectors/destination/pulsar connectors/destination/redis connectors/destination/redpanda connectors/destination/redshift connectors/destination/rockset connectors/destination/s3-glue connectors/destination/scylla connectors/destination/selectdb connectors/destination/snowflake connectors/destination/starburst-galaxy connectors/destination/teradata connectors/destination/tidb connectors/destination/vertica connectors/destination/yugabytedb connectors/source/chargify connectors/source/onesignal connectors/source/s3 connectors/source/stripe

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants