Source S3: handle legacy path_prefix + path_patterns - #29382
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/source-s3/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ❌ |
| QA checks | ✅ |
| Code format checks | ✅ |
| Connector package install | ✅ |
| Build source-s3 docker image for platform linux/x86_64 | ✅ |
| Unit tests | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-s3 test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-gcs/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-gcs docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-gcs:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-gcs test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-mongodb-strict-encrypt/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-mongodb-strict-encrypt docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-mongodb-strict-encrypt:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mongodb-strict-encrypt test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-bigquery docker image for platform linux/x86_64 | ✅ |
| Build airbyte/normalization:dev | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-mariadb-columnstore/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-mariadb-columnstore docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-mariadb-columnstore:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mariadb-columnstore test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-redis/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-redis docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-redis:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-redis test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-selectdb/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ❌ |
| Build connector tar | ✅ |
| Build destination-selectdb docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-selectdb:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-selectdb test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-snowflake docker image for platform linux/x86_64 | ✅ |
| Build airbyte/normalization-snowflake:dev | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-snowflake test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-mssql/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-mssql docker image for platform linux/x86_64 | ✅ |
| Build airbyte/normalization-mssql:dev | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-mssql:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mssql test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-mysql/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-mysql docker image for platform linux/x86_64 | ✅ |
| Build airbyte/normalization-mysql:dev | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-mysql:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mysql test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-pulsar/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-pulsar docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-pulsar:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-pulsar test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-oracle-strict-encrypt/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-oracle-strict-encrypt docker image for platform linux/x86_64 | ✅ |
| Build airbyte/normalization-oracle:dev | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-oracle-strict-encrypt:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-oracle-strict-encrypt test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-mongodb/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-mongodb docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-mongodb:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mongodb test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-rockset/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ❌ |
| Build connector tar | ✅ |
| Build destination-rockset docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-rockset:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-rockset test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-postgres/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-postgres docker image for platform linux/x86_64 | ✅ |
| Build airbyte/normalization:dev | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-postgres:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-postgres test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-iceberg/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ❌ |
| Build connector tar | ✅ |
| Build destination-iceberg docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-iceberg:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-iceberg test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-pubsub/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-pubsub docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-pubsub:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-pubsub test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/source-s3/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ❌ |
| QA checks | ✅ |
| Code format checks | ✅ |
| Connector package install | ✅ |
| Build source-s3 docker image for platform linux/x86_64 | ✅ |
| Unit tests | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-s3 test
Brian Lai (brianjlai)
left a comment
There was a problem hiding this comment.
lgtm. presumably the tests will pass once the CDK change that this depends on is merged in
| return [path_prefix + path_pattern] | ||
| return [path_pattern] | ||
| def _create_globs(cls, path_pattern: str) -> List[str]: | ||
| if "|" in path_pattern: |
There was a problem hiding this comment.
let's add one test case that we separate legacy glob input with the the or separators and confirm the resulting glob is a list
d7dad51 to
7b95e0e
Compare
fbaa265 to
e180267
Compare
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/source-s3/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ❌ |
| QA checks | ✅ |
| Code format checks | ✅ |
| Connector package install | ✅ |
| Build source-s3 docker image for platform linux/x86_64 | ✅ |
| Unit tests | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-s3 test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/source-s3/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
| Code format checks | ✅ |
| Connector package install | ✅ |
| Build source-s3 docker image for platform linux/x86_64 | ✅ |
| Unit tests | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-s3 test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/source-s3/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
| Code format checks | ✅ |
| Connector package install | ✅ |
| Build source-s3 docker image for platform linux/x86_64 | ✅ |
| Unit tests | ✅ |
| Integration tests | ✅ |
| Acceptance tests | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-s3 test
This is the source-s3 update corresponding with #29389.