New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
1s1t: Refactor sqlgenerator integration test #28890
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
8917066
to
703f42e
Compare
c2066ae
to
5227fa2
Compare
destination-mssql test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mssql test
destination-starburst-galaxy test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-starburst-galaxy test
destination-tidb test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-tidb test
destination-dynamodb test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-dynamodb test
destination-mysql-strict-encrypt test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mysql-strict-encrypt test
destination-yugabytedb test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-yugabytedb test
destination-mariadb-columnstore test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mariadb-columnstore test
destination-tidb test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-tidb test
destination-mssql test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mssql test
destination-dynamodb test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-dynamodb test
destination-clickhouse-strict-encrypt test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-clickhouse-strict-encrypt test
destination-starburst-galaxy test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-starburst-galaxy test
destination-clickhouse test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-clickhouse test
destination-mssql-strict-encrypt test report (commit
|
Step | Result |
---|
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mssql-strict-encrypt test
// This is not a typical stream ID would look like, but SqlGenerator isn't allowed to make any | ||
// assumptions about StreamId structure. | ||
// In practice, the final table would be testDataset.users, and the raw table would be | ||
// airbyte_internal.testDataset_ab__ab_users. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
raw table name update
// This is not a typical stream ID would look like, but SqlGenerator isn't allowed to make any | ||
// assumptions about StreamId structure. | ||
// In practice, the final table would be testDataset.users, and the raw table would be | ||
// airbyte_internal.testDataset_ab__ab_users. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// airbyte_internal.testDataset_ab__ab_users. | |
// airbyte_internal.testDataset_raw__stream_users. |
destination-bigquery test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ❌ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-bigquery docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery test
destination-bigquery test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ❌ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-bigquery docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery test
destination-bigquery test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-bigquery docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery test
closes #27569; closes #27567
add a BaseSqlGeneratorIntegrationTest, and make BigQuerySqlGeneratorIntegrationTest extend it. Move most test cases into the base class. Similarly to the TypingDedupingTest, some of these test cases depend on an input records file (defined in the base test module) and an expected records file (defined in the destination module). A lot of these tests are only verifying record counts; we might want to beef them up eventually but I think they're sufficient for now.
Also switched the tests to use a composite PK.
A small subset of tests are still defined in the bigquery test implementation:
Also, swap the order of StreamId.finalTableId's parameters, so that the "suffix" param is the last argument 😛 so there's a small amount of runtime code being modified, but it's behaving identically to before.
All tests pass locally: