Airbyte should support external secret stores

[michel-tricot]

Tell us about the problem you're trying to solve

I would like Airbyte to store connector secrets in external secret stores (HashiCorp Vault, AWS Secrets Manager, GCS Secrets Manager...)

Target Secret Stores

• - Google Secrets Manager (done) - issue docs
• External Secrets Store Support: HashiCorp Vault #10519
• External Secrets Store Support: AWS Secrets Manager #10518

[ChristopheDuong]

FYI this is handled in Airflow and i guess commonly used so if we switch/integrate somehow, this could benefit from it too:
https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html

[chethanuk]

+1 Nice to have an alternative secrets backend, even just the Hashicorp vault will be better.
Mainly it solves:

• Central secrets managements
• Reduce manual rotation of secrets in all the systems.

Helps in Automating secrets management end to end.

[tovbinm]

+1 plugable secret store

[prasrvenkat]

Not sure if it's under consideration, but, If possible, reading from existing stores would be huge benefit. In our case we have a different flow keeping our tokens up-to-date elsewhere since we use them as well for different use cases in our product so simply mentioning a way to read from existing external stores would be nice. Happy to work with Airbyte on any/all part of it.

[iamzjk]

Would be great if it can read from env variables too.

[vbhamidipati]

+1 - this is an important security capability

[raphaelauv]

look like you can use the GCP secret manager to store the secret with the setting -> SECRET_PERSISTENCE_TYPE

[novotl]

Hi, any updates / estimates on this?

We would like to run Airbyte OSS on Kubernetes, but unencrypted secrets is a big security hazard for us. Environment variables or HashiCorp Vault would suit us the best.

[anand-srinivaas]

+1 this would be really helpful !

[cavanaug]

FWIW. This is an interesting project as well for folks wanting to expose secrets in K8s. It sort of provides the abstraction layer, so you dont necessarily write code for each vaulting system. It is something we are looking to utilize internally in our own k8s infrastructure.

https://github.com/kubernetes-sigs/secrets-store-csi-driver

[cgardens]

We support GCP secrets store and AWS secrets manager. It is possible to add other external secrets stores.

[Sayon0]

Hey it looks like you guys support writing to a secrets store for any secrets, however, how do I set airbyte to read a secret from an existing secret store.

More specifically, I'm setting up a source that requires and API key, and I don't want to have my team type that API key in, I would like for them to just state the name of the secret in the secret store

[pgrant87]

We've had a request to add CyberArk secret support from a customer in a support case, Feature request for this at: #32981

[ecofer-zip]

Has anyone integrated Airbyte with Azure KeyVault for storing key/pwd/uid info securely? This could be a blocker for us to use Airbyte.

[VeeraswamyGatta]

Hi Team,

To follow up on previous question on Key Vault , does any one integrated Key Vault? if yes any reference link.

[tovbinm]

Using AWS Systems Manager with encrypted params is also way cheaper than AWS Secrets Manager