Skip to content

🚨 Destination bigquery 1s1t: change raw dataset + table name - #28723

Merged
Octavia The Approver (octavia-approvington) merged 7 commits into
masterfrom
edgao/1s1t/namespace
Jul 27, 2023
Merged

Octavia The Approver (octavia-approvington) merged 7 commits into
masterfrom
edgao/1s1t/namespace

Conversation

@edgao

@edgao Edward Gao (edgao) commented Jul 26, 2023 •

Copy link
Copy Markdown
Contributor
  • raw dataset is now airbyte_internal by default (CatalogParser)
  • TypingDedupingFlag now has logic to get the override config option (arguably overkill, idk); BigQueryDestination consumes the new method
    • Updated integration tests to handle the raw dataset override
  • Rename raw tables to <namespace>_raw_..._stream_<name> (StreamId)
    • and prevent it from generating as _raw_stream, i.e. there are always at least two underscores

new gsm secret https://console.cloud.google.com/security/secret-manager/secret/SECRET_DESTINATION-BIGQUERY_CREDENTIALS_1S1T_STANDARD_RAW_OVERRIDE/versions?project=dataline-integration-testing

closes #28403

see also #28744

@github-actions

github-actions Bot commented Jul 26, 2023 •

Copy link
Copy Markdown
Contributor

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! πŸŽ‰

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

public class CatalogParser {

public static final String DEFAULT_RAW_TABLE_NAMESPACE = "airbyte";
public static final String DEFAULT_RAW_TABLE_NAMESPACE = "airbyte_internal";

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was theoretically a oneliner, the rest is just prepwork for snowflake

@@ -0,0 +1,9 @@
package io.airbyte.integrations.destination.bigquery.typing_deduping;

public class BigQueryStandardInsertsRawOverrideTypingDedupingTest extends AbstractBigQueryTypingDedupingTest {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: I see this pattern a lot in the test suite. Do we need to run all the tests again, or would it be sufficient just to run one of them, just making sure a sync doesn't blow up and the schema we expect was used.

Looking at this test now, I think it would pass if the override wasn't in use at all, and we used the default?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah yeah that's a good point... I guess if we have the test explicitly override getRawDataset to return literal overridden_raw_dataset then we'd catch it (because the test would try to dump raw records from a nonexistent table). I'll make that change.

IMO there's enough interesting tests (i.e. overwrite, incremental dedup, namespace=null all seem useful to run for every config) that it's worth just running the entire suite, but curious if you feel otherwise

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-mssql-strict-encrypt test report (commit af5dba901d) - ❌

⏲️ Total pipeline duration: 28mn44s

Step Result
Validate airbyte-integrations/connectors/destination-mssql-strict-encrypt/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build destination-mssql-strict-encrypt docker image for platform linux/x86_64 βœ…
Build airbyte/normalization-mssql:dev βœ…
./gradlew :airbyte-integrations:connectors:destination-mssql-strict-encrypt:integrationTest ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mssql-strict-encrypt test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-postgres-strict-encrypt test report (commit af5dba901d) - ❌

⏲️ Total pipeline duration: 05mn47s

Step Result
Validate airbyte-integrations/connectors/source-postgres-strict-encrypt/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build source-postgres-strict-encrypt docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:source-postgres-strict-encrypt:integrationTest βœ…
Acceptance tests ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-mssql-strict-encrypt test report (commit af5dba901d) - ❌

⏲️ Total pipeline duration: 16mn38s

Step Result
Validate airbyte-integrations/connectors/source-mssql-strict-encrypt/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build source-mssql-strict-encrypt docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:source-mssql-strict-encrypt:integrationTest ❌
Acceptance tests ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql-strict-encrypt test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-redpanda test report (commit af5dba901d) - ❌

⏲️ Total pipeline duration: 06mn45s

Step Result
Validate airbyte-integrations/connectors/destination-redpanda/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build destination-redpanda docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:destination-redpanda:integrationTest ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redpanda test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-tidb test report (commit af5dba901d) - βœ…

⏲️ Total pipeline duration: 17mn14s

Step Result
Validate airbyte-integrations/connectors/destination-tidb/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build destination-tidb docker image for platform linux/x86_64 βœ…
Build airbyte/normalization-tidb:dev βœ…
./gradlew :airbyte-integrations:connectors:destination-tidb:integrationTest βœ…

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-tidb test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-doris test report (commit af5dba901d) - ❌

⏲️ Total pipeline duration: 03mn53s

Step Result
Validate airbyte-integrations/connectors/destination-doris/metadata.yaml βœ…
Connector version semver check βœ…
QA checks ❌
Build connector tar βœ…
Build destination-doris docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:destination-doris:integrationTest ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-doris test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-postgres test report (commit af5dba901d) - βœ…

⏲️ Total pipeline duration: 14mn11s

Step Result
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build source-postgres docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:source-postgres:integrationTest βœ…
Acceptance tests βœ…

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-mysql test report (commit af5dba901d) - ❌

⏲️ Total pipeline duration: 20mn12s

Step Result
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build source-mysql docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest ❌
Acceptance tests βœ…

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-mssql test report (commit af5dba901d) - ❌

⏲️ Total pipeline duration: 19mn48s

Step Result
Validate airbyte-integrations/connectors/source-mssql/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build source-mssql docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:source-mssql:integrationTest ❌
Acceptance tests βœ…

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-databricks test report (commit af5dba901d) - ❌

⏲️ Total pipeline duration: 50mn24s

Step Result
Validate airbyte-integrations/connectors/destination-databricks/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build destination-databricks docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:destination-databricks:integrationTest ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-databricks test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-e2e-test test report (commit af5dba901d) - βœ…

⏲️ Total pipeline duration: 04mn39s

Step Result
Validate airbyte-integrations/connectors/source-e2e-test/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build source-e2e-test docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:source-e2e-test:integrationTest βœ…

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-e2e-test test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-mysql test report (commit af5dba901d) - ❌

⏲️ Total pipeline duration: 41mn53s

Step Result
Validate airbyte-integrations/connectors/destination-mysql/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build destination-mysql docker image for platform linux/x86_64 βœ…
Build airbyte/normalization-mysql:dev βœ…
./gradlew :airbyte-integrations:connectors:destination-mysql:integrationTest ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mysql test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-redis test report (commit af5dba901d) - ❌

⏲️ Total pipeline duration: 06mn10s

Step Result
Validate airbyte-integrations/connectors/destination-redis/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build destination-redis docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:destination-redis:integrationTest ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redis test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-keen test report (commit af5dba901d) - βœ…

⏲️ Total pipeline duration: 09mn10s

Step Result
Validate airbyte-integrations/connectors/destination-keen/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build destination-keen docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:destination-keen:integrationTest βœ…

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-keen test

@edgao Edward Gao (edgao) changed the title Destination bigquery 1s1t: change default raw dataset Destination bigquery 1s1t: change raw dataset + table name Jul 26, 2023
@octavia-squidington-iii Octavia Squidington III (octavia-squidington-iii) added the area/documentation Improvements or additions to documentation label Jul 26, 2023
@edgao Edward Gao (edgao) changed the title Destination bigquery 1s1t: change raw dataset + table name 🚨 Destination bigquery 1s1t: change raw dataset + table name Jul 26, 2023
connectorType: destination
definitionId: 22f6c74f-5699-40ff-833c-4a879ea40133
dockerImageTag: 1.5.7
dockerImageTag: 1.6.0

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor bump since it's a breaking change for a beta feature...? idk, semver gets a little weird in this case

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-dynamodb test report (commit 7f17394087) - ❌

⏲️ Total pipeline duration: 02mn14s

Step Result
Validate airbyte-integrations/connectors/source-dynamodb/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-dynamodb test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-s3 test report (commit 7f17394087) - βœ…

⏲️ Total pipeline duration: 14mn57s

Step Result
Validate airbyte-integrations/connectors/destination-s3/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build destination-s3 docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:destination-s3:integrationTest βœ…

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-s3 test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-oracle-strict-encrypt test report (commit 7f17394087) - ❌

⏲️ Total pipeline duration: 33mn57s

Step Result
Validate airbyte-integrations/connectors/destination-oracle-strict-encrypt/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build destination-oracle-strict-encrypt docker image for platform linux/x86_64 βœ…
Build airbyte/normalization-oracle:dev βœ…
./gradlew :airbyte-integrations:connectors:destination-oracle-strict-encrypt:integrationTest ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-oracle-strict-encrypt test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-dev-null test report (commit 7f17394087) - ❌

⏲️ Total pipeline duration: 08mn15s

Step Result
Validate airbyte-integrations/connectors/destination-dev-null/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build destination-dev-null docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:destination-dev-null:integrationTest ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-dev-null test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-teradata test report (commit 7f17394087) - βœ…

⏲️ Total pipeline duration: 18mn41s

Step Result
Validate airbyte-integrations/connectors/destination-teradata/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build destination-teradata docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:destination-teradata:integrationTest βœ…

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-teradata test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

destination-mongodb test report (commit 7f17394087) - ❌

⏲️ Total pipeline duration: 16mn20s

Step Result
Validate airbyte-integrations/connectors/destination-mongodb/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build destination-mongodb docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:destination-mongodb:integrationTest ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mongodb test

@octavia-squidington-iii

Copy link
Copy Markdown
Collaborator

source-tidb test report (commit 7f17394087) - ❌

⏲️ Total pipeline duration: 09mn39s

Step Result
Validate airbyte-integrations/connectors/source-tidb/metadata.yaml βœ…
Connector version semver check βœ…
QA checks βœ…
Build connector tar βœ…
Build source-tidb docker image for platform linux/x86_64 βœ…
./gradlew :airbyte-integrations:connectors:source-tidb:integrationTest βœ…
Acceptance tests ❌

πŸ”— View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-tidb test

@edgao

Edward Gao (edgao) commented Jul 27, 2023 •

Copy link
Copy Markdown
Contributor Author

ran airbyte-ci --show-dagger-logs connectors --name destination-bigquery test locally. The only failing tests are known issues (fixed in #28802). merging with approve bot and cancelling the connectors ci action.

@edgao

Copy link
Copy Markdown
Contributor Author

/approve-and-merge reason="ran airbyte-ci locally and the only test failures are from known-buggy tests"

@octavia-approvington

Copy link
Copy Markdown
Contributor

It's time
fine lets go

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/bigquery

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Change Default Airbyte Internal Schema name and Allow Overrides

5 participants