Support for AWS Secrets Manager

[janwillies]

What problem are you facing?

I want to manage AWS Secrets Manager via crossplane provider-aws, e.g. create, update and delete secrets

How could Crossplane help solve your problem?

Implement a Secrets Manager resource: https://docs.aws.amazon.com/sdk-for-go/v2/api/service/secretsmanager

[Knappek]

I am planning to work on this and thinking about how to implement it. Check out the related discussion on slack.

I am thinking about two use cases:

1. automatically sink secrets from crossplane managed infrastructure resources to AWS secrets manager: currently crossplane writes secrets (like RDS connection string etc.) to Kubernetes secrets via writeConnectionSecretToRef. This can be extended to have the option to write to AWS Secrets Manager (or any external secret store).
2. manually creating secrets via kubectl: This is especially useful if you want to store secrets of external services (resources that are not managed by crossplane), e.g. credentials of an external database.

Let's first consider use case 1: @hasheddan's proposal was to use a new type CredentialsSink that allows something like provider-vault to basically surface to other providers that it can be a sink for credential information to be stored. The idea is to not write any secrets to Kubernetes secrets at all! The tricky part thereby is that you don't want to write any secrets to the new type CredentialsSink either. On the other hand, you want to have an independent controller that watches the CredentialsSink type that is still be able to somehow have access to these secrets which it should write to the external secret store.

I have thought about this for a while and this is my proposal:

1. crossplane still creates Kubernetes secrets as provided in writeConnectionSecretToRef (without needing to know that these secrets might be stored in an external secret store as well)
2. if you want to reconcile secrets to an external secret store, you create a new type with kind: Credential and apiVersion: secretsmanager.aws.crossplane.io/v1alpha1. A controller implemented in the provider-aws will watch this new type. This type stores all information needed in order to create a secret in AWS secrets manager, having all kinds of parameters in forProvider as defined in the aws secretsmanager go sdk.
3. the new type Credential tells the secretsmanager-controller to also watch the corresponding Kubernetes secret name: rds-credentials-my-app; namespace: my-app-ns. By watching the corresponding Kubernetes secret, we make sure that changes to this secret get reconciled to AWS secrets manager.

A couple of pros/cons to this approach:

• pros:
  • no need to change anything in crossplane core
  • easy implementation of the new secretsmanager controller that watches a new type
• cons:
  • customers might want to not store any secrets in Kubernetes secrets at all. In order to change this, we need to make fundamental changes in crossplane core and extend the implementation of writeConnectionSecretToRef

@hasheddan and @janwillies, it would be great to have your opinion on that.

[janwillies]

If my understanding is correct this would be a straight forward implementation of the CRUD operations of the Secrets Manager Resource with the exception that the SecretString-value is a reference to a Kubernetes Secret and the actual value is picked up from there. That makes sense to me b/c this way it can benefit from the envelope encryption of Kubernetes secrets. Does that work for you @hasheddan @muvaf?

A nitpick: it should be kind: Secret if we want to adopt the AWS vocabulary

[muvaf]

I think the case where users don't want to store creds in Kubernetes Secret will be addressed with a general approach in crossplane/crossplane#1424

Meanwhile, I agree that we can have it just like any other managed resource. The difference would be that the data source would be a referenced Secret which is something we already do for sensitive data.

[negz]

I think this is a great approach - thanks @Knappek! I really like using the regular connection secrets as an intermediary store that can in turn be shipped off to another secret store.

I'm curious how many orgs actually have a mandate to ensure credentials are never persisted as a Kubernetes secret at all. If we could get away with using this approach for e.g. provider-vault that would certainly scale better. This way every provider just needs to know how to write a secret, and we can add many decoupled backends that can sync those secrets to secret stores. The alternative - having every provider support every possible secret store in its native code - seems like it would not scale well.