New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🧹 Sweep authSpecification in connectors repo #27996
🧹 Sweep authSpecification in connectors repo #27996
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
Note: keeping this in draft because there are still 2 connectors that need to be migrated off of But it is ready for review now! |
/legacy-test connector=bases/connector-acceptance-test
Build PassedTest summary info:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm!
:returns list of path_in_connector_config paths | ||
""" | ||
paths = [] | ||
for value in schema.values(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: the pythonic way to do this is
return [
"/" + "/".join(value["path_in_connector_config"])
for value in schema.values()
]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed! Thanks for the suggestion
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm, but once the other two connectors are updated, just make sure to run the legacy-test
command and verify the spec test changes still pass
/legacy-test connector=bases/connector-acceptance-test
Build PassedTest summary info:
|
/legacy-test connector=connectors/source-rd-station-marketing
Build FailedTest summary info:
|
/legacy-test connector=connectors/destination-google-sheets
Build PassedTest summary info:
|
destination-mssql-strict-encrypt test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-mssql-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-mssql-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-mssql:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-mssql-strict-encrypt:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mssql-strict-encrypt test
source-facebook-marketing test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-facebook-marketing/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ❌ |
QA checks | ✅ |
Code format checks | ✅ |
Connector package install | ✅ |
Build source-facebook-marketing docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-facebook-marketing test
destination-exasol test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-exasol/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ❌ |
Build connector tar | ✅ |
Build destination-exasol docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-exasol:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-exasol test
destination-pubsub test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-pubsub/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-pubsub docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-pubsub:integrationTest | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-pubsub test
destination-databricks test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-databricks/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-databricks docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-databricks:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-databricks test
/legacy-test connector=connectors/source-google-sheets
|
/legacy-test connector=connectors/source-google-sheets
|
/legacy-publish connector=bases/connector-acceptance-test
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/legacy-test connector=connectors/source-google-sheets
|
/legacy-publish connector=bases/connector-acceptance-test
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/legacy-test connector=connectors/source-google-sheets
|
/legacy-test connector=connectors/source-google-sheets
Build FailedTest summary info:
|
source-google-sheets test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-google-sheets/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ❌ |
QA checks | ✅ |
Code format checks | ✅ |
Connector package install | ✅ |
Build source-google-sheets docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Acceptance tests | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-google-sheets test
destination-r2 test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-r2/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-r2 docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-r2:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-r2 test
source-snowflake test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-snowflake/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-snowflake docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-snowflake:integrationTest | ❌ |
Acceptance tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-snowflake test
source-postgres test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-postgres docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-postgres:integrationTest | ✅ |
Acceptance tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-postgres test
source-facebook-marketing test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-facebook-marketing/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
Code format checks | ✅ |
Connector package install | ✅ |
Build source-facebook-marketing docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-facebook-marketing test
/approve-and-merge reason="source-google-sheets connector should be working since this is just unpinning it to a version that had passed tests on its last change; the source test account is just in a broken state right now" |
destination-mssql test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-mssql/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-mssql docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-mssql:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-mssql:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mssql test
@lmossman Hello! It seems like this change break some more
They are failed on CAT with this:
Assuming that |
UPDATED: Apparently some sources like "complete_oauth_server_output_specification": {
"type": "object",
"additionalProperties": false,
"properties": {
"client_id": {
"type": "string",
"path_in_connector_config": ["client_id"]
},
"client_secret": {
"type": "string",
"path_in_connector_config": ["client_secret"]
}
}
} Removing this I guess the rest of the failed sources CATs died for the same reason, fixing them now. |
@bazarnov that sounds right to me; for example, looking at |
it is, however the Connectors Team was not aware of this change) It would be very helpful if we know that prior to the merge, if it's possible) |
Yes I definitely could have communicated this better, sorry about that! I didn't expect this change to cause new failures in connectors since it should have been testing the same type of things as the old authSpecification test, but in these cases the connectors failed the new test because of an underlying bug in their spec |
What
Relates to airbytehq/airbyte-protocol#39 and https://github.com/airbytehq/airbyte-platform-internal/pull/7595
This PR sweeps all occurrences of
authSpecification
from theairbytehq/airbyte
repository as part of the project to deprecate that field.How
Remove references to
authSpecification
, and update the tests as necessary to test theadvanced_auth
configuration instead.Note: The legacy
authSpecification
still appears in thecloud_registry.json
,oss_registry.json
, andconnector_catalog.json
files, but this shouldn't matter since those are only used for testing general behavior that doesn't rely on this field, and will be wiped out the next time those files are refreshed due to a catalog shape change.Recommended reading order
airbyte-api/src/main/openapi/config.yaml
- removes authSpecification from the apiairbyte-integrations/bases/connector-acceptance-test/connector_acceptance_test/tests/test_core.py
- updates thetest_oauth_flow_parameters
CAT test to verify that all paths in theadvanced_auth
object exist in the connector spec, rather than testing something similar forauthSpecification
airbyte-integrations/bases/connector-acceptance-test/unit_tests/test_spec.py
- updates the CAT unit tests to account for the above change🚨 User Impact 🚨
No breaking changes or user impact