airbyte-ci: enable the use of local connector secrets

[alafanechere]

Motivation

Closes #25621
As airbyte-ci is now the recommended way to build and test connectors we should enable community users to use it.
So far one blocker for community use was the fact that the connectors command group did not support the use of local connector secrets and relied on our GSM secrets store.

We want the airbyte-ci connectors command to be runnable without the GCP_GSM_CREDENTIALS env var and have a sensible default wether to use remote or local secrets if the --use-local-secret/--use-remote-secrets flags are not passed.

CLI behavior

• If --use-local-secret/--use-remote-secrets flags are not passed:

  • Use remote secrets if the GCP_GSM_CREDENTIALS env var is set
  • Fall back to local secrets if GCP_GSM_CREDENTIALS is not set

• If --use-local-secrets is passed:
  We'll mount secrets from the connector secrets directory.
  We log a warning if this directory is empty and does not exist.
  Unit/Integration/CAT tests will likely fail if these file are missing, we let them handle the error handling when the secret file is missing. (We have some connectors, like source-pokeapi, that do not require secret files).

• If --use-remote-secrets:
  We check that GCP_GSM_CREDENTIALS is set or raise a click.UsageError exception otherwise.

🚨 User Impact 🚨

Community users should be able to run the test command without access to our secret store. test will use their local connector secrets.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 24, 2023 5:08pm

[alafanechere]

Current dependencies on/for this PR:

• master
  • PR airbyte-ci: enable the use of local connector secrets #31766 👈

This comment was auto-generated by Graphite.

[bazarnov]

Awesome improvement! LGTM.

[marcosmarxm]

We'll mount secrets from the connector secrets directory.
We log a warning if this directory is empty and does not exist.
Unit/Integration/CAT tests will likely fail if these file are missing, we let them handle the error handling when the secret file is missing. (We have some connectors, like source-pokeapi, that do not require secret files).

Usually community connectors have only one secret file and it's easy to share a single credential with the contributor. This feature will help a lot!

[marcosmarxm]

This is the case if the community member or us want to publish the connector from local?

[alafanechere]

Nope, it's a check to make sure that these env var are set in the CI

[erohmensing]

Didn't test it myself, but code looks good! A few nits, but nothing blocking

[erohmensing]

🥳