CAT: Add scenario that OAuth is default method and fields are marked as airbyte_secret

[darynaishchenko]

What

resolved: #33564
As part of certification criteria we should be ensuring that OAuth on cloud is the first option presented to users and that sensitive values are designated as secrets.

How

OAuth is default: Extended existing TestSpec.test_oauth_flow_parameters() with validation for case when credentials object do have an oneOf list and the first method listed is OAuth method.
OAuth fields marked as secret: TestSpec.test_secret_is_properly_marked has been already implemented. Updated unit test for it with OAuth method in test cases spec.

This has been tested on a handful of different sources:

• source-facebook-pages
• source-google-ads
• source-facebook-marketing
• source-github

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jan 25, 2024 11:05am

[maxi297]

Just a couple of questions

[darynaishchenko]

@maxi297 updated testing flow

• connector health engineer will be responsible for fixing failed connectors
• added skip with bypass reason options that can be added in CAT config

[maxi297]

Just one clarification and I'm good to go

[brianjlai]

two thoughts here:

Do we need a separate CAT acceptance config at all? Rather than force each connector to explicitly define this field, could we just always run the test no matter what connector? And we either skip or pass the test if a connector's spec doesn't have any form of OAuth to validate.

And for the connectors that do have OAuth in the spec, then we verify it's the first option. That seems like the least impact on our existing connectors. And if we're concerned about this test failing for uncertified connectors, we can use operational_certification_test fixture to skip the test when it quality level is not at the right value

I'm curious if we have seen any use cases for why OAuth shouldn't be first option and it requires a bypass reason?

[darynaishchenko]

@brianjlai , For current implementation a separate CAT acceptance config is optional, the test always runs as part of TestSpec scenario. The test is skipped in case 1. no oauth in connector. 2. no oneOf (only one way to auth), if oauth is present 3. auth_default_method.oauth == False with bypass reason provided in CAT acceptance config (A connector health engineer(secondary oncall) will be responsible to fix oauth default method or add skip and bypass in cat config.)

Do we need a separate CAT acceptance config at all? Rather than force each connector to explicitly define this field, could we just always run the test no matter what connector? And we either skip or pass the test if a connector's spec doesn't have any form of OAuth to validate.

It's no problem to add operational_certification_test fixture, but the reason why it wasn't added was mentioned in the task #33564 (Although this is a higher operational certification level, this is a relatively trivial fix even if it were to block community level connectors.) For example zendesk-talk is certified and generally_available but OAuth is not default option, I didn't find the reason in certification checklist, it needs more investigation that a secondary oncall can do while his work on connector health and provide a fix/bypass.

And for the connectors that do have OAuth in the spec, then we verify it's the first option. That seems like the least impact on our existing connectors. And if we're concerned about this test failing for uncertified connectors, we can use operational_certification_test fixture to skip the test when it quality level is not at the right value

[brianjlai]

One naming correction on the config validator and there's the merge conflict on the CHANGELOG. Everything else looks good to me.

I think we're in alignment that it will just incorporated w/o the QL taken into account and primary/secondary can fix them if the do come up

[darynaishchenko]

@maxi297, I have approve from @brianjlai. Is everything good from your side?