Workloads get new dynamic secrets by Vault when new replicas are created

[braybaut]

Hi, wonderful people.

I'm using Vault to provide dynamic AWS secrets to on-prem Kubernetes clusters and reach AWS components, I created the storage class to get the credentials and I attach the volume to workload, so far so good, however when the workloads scale up to X replicas, the vault CSI get dynamic secrets to each new replica if I have 50 replicas I will have 50 new access key but, the vault secret object just keep the last AWS credentials.

I think it's a bug because doesn't make sense to have an access key for each replica,

is it expected behavior? should the storage class just get a dynamic secret for a workload with N replicas and rotate it through pull interval and TTL?

Thanks in advance.

[hamdifourati]

We are having the same issue with database dynamics credentials.

[tomhjp]

This was filed earlier, but I commented on #151 which shares a lot in common with this issue. I'd be interested to get thoughts on my comment there, as it seems like there could be some unintended consequences to sharing dynamic secrets among pods if it's done naively, but I'm open to feedback on that.

One thing #151 doesn't cover is the fact that the Kuberentes secret object gets re-written a lot with the different values from each pod, which is a bit gross. Thanks for highlighting it.

[Freyert]

@braybaut, just to be clear every single pod that is referencing the vaul-csi-provider volume has the same AWS Credential? This being the last AWS credential provisioned?

I had imagined that each pod would get its own credential, but what you're describing does sound like a bug if they all have the same credential. Plus the 49 unused credentials.

[tomhjp]

Update: We recently released the Vault Secrets Operator, which has a much better model for satisfying a "1 secret per deployment" pattern - you can define a VaultDynamicSecret custom resource to create a Kubernetes secret managed by the operator, and all pods can mount the same leased secret into a volume/env var. Deployments can also opt in to rolling restarts by using the RolloutRestartTarget field.