🧹 Sweep authSpecification in connectors repo - #27996
🧹 Sweep authSpecification in connectors repo#27996Octavia The Approver (octavia-approvington) merged 16 commits into
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
|
Note: keeping this in draft because there are still 2 connectors that need to be migrated off of But it is ready for review now! |
|
/legacy-test connector=bases/connector-acceptance-test
Build PassedTest summary info: |
Alexandre Girard (girarda)
left a comment
There was a problem hiding this comment.
lgtm!
| :returns list of path_in_connector_config paths | ||
| """ | ||
| paths = [] | ||
| for value in schema.values(): |
There was a problem hiding this comment.
nit: the pythonic way to do this is
return [
"/" + "/".join(value["path_in_connector_config"])
for value in schema.values()
]
There was a problem hiding this comment.
Fixed! Thanks for the suggestion
Brian Lai (brianjlai)
left a comment
There was a problem hiding this comment.
lgtm, but once the other two connectors are updated, just make sure to run the legacy-test command and verify the spec test changes still pass
|
/legacy-test connector=bases/connector-acceptance-test
Build PassedTest summary info: |
|
/legacy-test connector=connectors/source-rd-station-marketing
Build FailedTest summary info: |
|
/legacy-test connector=connectors/destination-google-sheets
Build PassedTest summary info: |
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-mssql-strict-encrypt/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-mssql-strict-encrypt docker image for platform linux/x86_64 | ✅ |
| Build airbyte/normalization-mssql:dev | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-mssql-strict-encrypt:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mssql-strict-encrypt test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/source-facebook-marketing/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ❌ |
| QA checks | ✅ |
| Code format checks | ✅ |
| Connector package install | ✅ |
| Build source-facebook-marketing docker image for platform linux/x86_64 | ✅ |
| Unit tests | ✅ |
| Integration tests | ✅ |
| Acceptance tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-facebook-marketing test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-exasol/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ❌ |
| Build connector tar | ✅ |
| Build destination-exasol docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-exasol:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-exasol test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-pubsub/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-pubsub docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-pubsub:integrationTest | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-pubsub test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-databricks/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-databricks docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-databricks:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-databricks test|
/legacy-test connector=connectors/source-google-sheets
|
|
/legacy-test connector=connectors/source-google-sheets
|
|
/legacy-publish connector=bases/connector-acceptance-test
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/legacy-test connector=connectors/source-google-sheets
|
|
/legacy-publish connector=bases/connector-acceptance-test
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/legacy-test connector=connectors/source-google-sheets
|
|
/legacy-test connector=connectors/source-google-sheets
Build FailedTest summary info: |
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/source-google-sheets/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ❌ |
| QA checks | ✅ |
| Code format checks | ✅ |
| Connector package install | ✅ |
| Build source-google-sheets docker image for platform linux/x86_64 | ✅ |
| Unit tests | ✅ |
| Acceptance tests | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-google-sheets test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-r2/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-r2 docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-r2:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-r2 test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/source-snowflake/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build source-snowflake docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:source-snowflake:integrationTest | ❌ |
| Acceptance tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-snowflake test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/source-postgres/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build source-postgres docker image for platform linux/x86_64 | ✅ |
| ./gradlew :airbyte-integrations:connectors:source-postgres:integrationTest | ✅ |
| Acceptance tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-postgres test
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/source-facebook-marketing/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
| Code format checks | ✅ |
| Connector package install | ✅ |
| Build source-facebook-marketing docker image for platform linux/x86_64 | ✅ |
| Unit tests | ✅ |
| Integration tests | ✅ |
| Acceptance tests | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-facebook-marketing test|
/approve-and-merge reason="source-google-sheets connector should be working since this is just unpinning it to a version that had passed tests on its last change; the source test account is just in a broken state right now" |
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/destination-mssql/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Build connector tar | ✅ |
| Build destination-mssql docker image for platform linux/x86_64 | ✅ |
| Build airbyte/normalization-mssql:dev | ✅ |
| ./gradlew :airbyte-integrations:connectors:destination-mssql:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mssql test|
Lake Mossman (@lmossman) Hello! It seems like this change break some more
They are failed on CAT with this: Assuming that |
|
UPDATED: Apparently some sources like "complete_oauth_server_output_specification": {
"type": "object",
"additionalProperties": false,
"properties": {
"client_id": {
"type": "string",
"path_in_connector_config": ["client_id"]
},
"client_secret": {
"type": "string",
"path_in_connector_config": ["client_secret"]
}
}
}Removing this I guess the rest of the failed sources CATs died for the same reason, fixing them now. |
|
Baz (@bazarnov) that sounds right to me; for example, looking at |
|
it is, however the Connectors Team was not aware of this change) It would be very helpful if we know that prior to the merge, if it's possible) |
|
Yes I definitely could have communicated this better, sorry about that! I didn't expect this change to cause new failures in connectors since it should have been testing the same type of things as the old authSpecification test, but in these cases the connectors failed the new test because of an underlying bug in their spec |

What
Relates to airbytehq/airbyte-protocol#39 and https://github.com/airbytehq/airbyte-platform-internal/pull/7595
This PR sweeps all occurrences of
authSpecificationfrom theairbytehq/airbyterepository as part of the project to deprecate that field.How
Remove references to
authSpecification, and update the tests as necessary to test theadvanced_authconfiguration instead.Note: The legacy
authSpecificationstill appears in thecloud_registry.json,oss_registry.json, andconnector_catalog.jsonfiles, but this shouldn't matter since those are only used for testing general behavior that doesn't rely on this field, and will be wiped out the next time those files are refreshed due to a catalog shape change.Recommended reading order
airbyte-api/src/main/openapi/config.yaml- removes authSpecification from the apiairbyte-integrations/bases/connector-acceptance-test/connector_acceptance_test/tests/test_core.py- updates thetest_oauth_flow_parametersCAT test to verify that all paths in theadvanced_authobject exist in the connector spec, rather than testing something similar forauthSpecificationairbyte-integrations/bases/connector-acceptance-test/unit_tests/test_spec.py- updates the CAT unit tests to account for the above change🚨 User Impact 🚨
No breaking changes or user impact