Clarity: secrets store CSI driver vs external secrets... what to use?

[bitsofinfo]

Hi - thanks for this project. Currently looking for the best way to seed/bridge vault/azure-kv stored secrets and provision/access them from k8s. I came across the Secrets Store CSI project yesterday and today came across this project.

What are the main differences? is the CSI (or this project) becoming more part of an official standard? Just trying to determine which route to go.

external-secrets/kubernetes-external-secrets#722

[knelasevero]

The main differences:

• ESO synchronizes secrets from a cloud provider to secrets in k8s, so you can keep using k8s secrets if you are used to that.

• SSCSID mounts the external secret as a volume in a pod directly, and having a k8s secret in the cluster is optional

• ESO focuses on having configuration on the CRD, what you create on the secret store in the provider is only the secret value itself

• SSCSID requires the entire config/secret to be stored in the provider directly as the application will need to consume it. This may be too difficult to use with larger configurations that have some secrets embedded.

• ESO secrets can be used with any resource in k8s natively, that's obvious, but 👇

• SSCSID needs a pod webhook to really have it work well. You can not easily use SSCSID secrets to reference them in ingress, or dockerconfig for pulling images, since their goal is just to mount to a pod. Even if you want to enable k8s secret sync, you need to first mount the secret to a pod to sync it.

• In ESO, Since we sync secrets with k8s native secrets, if you have connectivity problems, you can still access the secret that is present in your cluster, when you re-connect, it will just continue to re-sync

• with SSCSID, if you loose connectivity, your csi driver mounts stop working if you get some restarts and all that. They are thinking about that, not sure what is the progress there: doc. You would need to check with them.

• ESO will be just one operator deployment in your cluster

• SSCSID will have a privileged provider daemonset that will be responsible to make the mounts in your pods

If you need to follow any compliance and need to avoid having kubernetes native secrets in your cluster, you need to go with SSCSID, then you can mount secrets from external provider directly as volumes in your pods. If you don't need that, you can probably give ESO a go.

We have in our roadmap the plan to integrate with them: #336

We have some discussions going: #461

But it can get quite complicated and we lack the engineering time to pursue that right now.

[bitsofinfo]

Thanks a lot @knelasevero this helps a lot!

[knelasevero]

Closing the issue. Feel free to comment or ask us to re-open :)

[ebuildy]

A big difference (I may be wrong): "ESO will decouple the secret provider with the workload, because it creates k8s secrets. SSCSID will call the secret provider every time a workload unit (aka pod) is created."

SSCSID is more similar to "vault k8s injector"

[tuxerrante]

@knelasevero hi!
I see this answer is referenced by the official documentation but after two years new features were implemented (like now CSI also supports secrets syncing), could this be updated on the docs please?
https://external-secrets.io/latest/introduction/faq/#differences-to-csi-secret-store

[knelasevero]

Secret Sync was available since the creation of this issue (for SSCSID) it is not new in that regard. The thing is that to perform a sync you still need to perform a pod mount. So if the secret is not really meant for workloads (img pull secrets, some certs, etc), you would still need to mount this into a dummy pod.

In any case, SSCSID folks are working on separating this functionality from the main driver into a new simple controller. Look at:

https://docs.google.com/document/d/1Ylwpg-YXNw6kC9-kdHNYD3ZKskj9TTIopwIxz5VUOW4/edit#heading=h.n3xa8h2b1inm

https://secrets-store-csi-driver.sigs.k8s.io/design-docs

And https://www.youtube.com/watch?v=CqySC9HP85M&list=PL69nYSiGNLP0VMOZ-V7-5AchXTHAQFzJw&index=1&ab_channel=Kubernetes 40 min mark and 50 min mark.

With this separation we could also work on making ESO replace that other controller if users want the feature-full nature of ESO and take the best of both worlds. But this is not planned work yet, so no promises, just outlining possibilities.

[knelasevero]

Also,

SSCSID mounts the external secret as a volume in a pod directly, and having a k8s secret in the cluster is optional

The having a k8s secret in the cluster is optional was probably what you were looking for, in the perspective of the documentation, and it is already there in the issue response.

[joebowbeer]

One thing that might be stressed more from a security point of view, is that Kubernetes Secrets are best avoided. This is spelled out CIS Benchmarks for Kubernetes, AWS Security Best Practices, and numerous other sources. It stems from the lack of RBAC specificity.

So, in my view, the choice is more like: "if you must have secrets, then use ESO"

That said, one thing that is not clear wrt SSCSID is that the quality and features vary per implementation. The AWS impl. for example, does not provide a way to fetch SSM Parameters by path, and it does not cache. Every pod that needs those parameters is going to execute its own fetches (which, in my case, nullifies the very reason I wanted to use ASCP).

[knelasevero]

I wouldn't stress that out on the documentation as I don't agree with that (and I believe others maintainers agree with me). Of course, open to discuss and change that perspective. I briefly wrote a bit about that (without much rigor, and within an opinionated article, but still), and I think that when thinking about threat modeling avoiding k8s secrets does not give necessarily more security, and this is one of those 'best practices' that sound right only on the surface, without analyzing all the details.

If you are synchronizing secrets from an external providers into your cluster it does not matter much if they are injected by a sidecar, stored as ephemeral volumes, or k8s native secrets, threat modeling from an attacker perspective is pretty similar and you are a subject of lack of RBAC specificity either way as far as I understand.

I still think the general idea is 'if you need to avoid k8s native secrets, look for other solutions' and not the other way around.

That said, one thing that is not clear wrt SSCSID is that the quality and features vary per implementation. The AWS impl. for example, does not provide a way to fetch SSM Parameters by path, and it does not cache. Every pod that needs those parameters is going to execute its own fetches (which, in my case, nullifies the very reason I wanted to use ASCP).

Yeah, agree

This is an interesting discussion, but maybe better moved to the 'discussion' tab here on github? Not sure here is the best place to discuss that.

PS: When CIS Kubernetes Benchmark talks about secrets it basically talks about RBAC, avoiding them as env vars and preferring files. When saying it might be a good idea to avoid K8S secrets, it talks about reaching external providers through their APIs directly, and not using any sync method (but then you are vendor locked in, and will have to change application code).

[joebowbeer]

@knelasevero I think it is worth mentioning because these benchmarks & best practices are what the security auditors and pen testers will be using to assess the security of one's clusters. It's a legit difference because of the extra attention it will generate, even if in practice reasonable minds can provide an explanation. (Whether the findings presented by the auditors are relevant is another matter, that I agree is best discussed elsewhere, but I can say from experience that every Kubernetes Secret will be scrutinized.)

[gusfcarvalho]

Hi @joebowbeer ! Security benchmarks and the need to comply is something that varies from company to company. ESO itself is just tool, and IMO, should not drive that type of conclusions. The use (or not) of any tool from a security perspective is always prone to evaluation (that’s why people have security departments to analyze what on the benchmark makes sense and what doesn’t).

I really don’t want to drive the projects documentation to only a sector of the whole Cloud native landscape (regulated environments).

[knelasevero]

I am all for avoiding K8S secrets if it will make your life easier during audits! I even wrote that in that article. I would not write that directly in the docs as this can sound like a pinch on how we tackle security nowadays, but I think I am comfortable mentioning it here on the issue :D That is a very valid point on being pragmatic on everything you will have to do in your company, not everything is technical, and we have processes to go through. So +1 (even though it is frustrating, and most of the time it won't make sense)

[tuxerrante]

I love to see I've triggered an interesting discussion 😁

If you like to move this on the discussion page I'd love to try to contribute as soon as I read a little more from the docs.

In brief: installing an external operator is a much much bigger risk than using secrets which are allowed to be read by an attacker which has already RBAC or AD permissions to spawn a container in the cluster + Get/List permissions to see them 😬
Is the container built from scratch, with 0 cves, denied privileged execution, restricted by seccomp? (Sorry still AFK)

[knelasevero]

Discussion: #2680

[gusfcarvalho]

In brief: installing an external operator is a much much bigger risk than using secrets which are allowed to be read by an attacker which has already RBAC or AD permissions to spawn a container in the cluster + Get/List permissions to see them

not sure if I understood your point here! My point is that you can’t evaluate risk outside of implementation. Automatic rotation of sensitive information (that you can do with ESO) would be less riskier to having a separate process to inject that same static information (and have it compromised anywhere in your stack)

Is the container built from scratch, with 0 cves, denied privileged execution, restricted by seccomp? (Sorry still AFK)

containers are built from distroless/ubi images, have 0 high/critical CVEs. As any other container, we can’t really control how people deploy it in terms of privileged execution/seccomp. We do provide sensible defaults.

[MohitBishesh]

One of the differences is that the CSI driver runs as a daemon set in the cluster whereas ESO does not, so it's better to use ESO as it's easy to manage and save resources as well.

[tuxerrante]

One of the differences is that the CSI driver runs as a daemon set in the cluster whereas ESO does not, so it's better to use ESO as it's easy to manage and save resources as well.

How could an operator with its complex control loop and custom resources reconciliation save resources with respect to a native daemonset implementing a much simpler control loop?

[knelasevero]

One of the differences is that the CSI driver runs as a daemon set in the cluster whereas ESO does not, so it's better to use ESO as it's easy to manage and save resources as well.

How could an operator with its complex control loop and custom resources reconciliation save resources with respect to a native daemonset implementing a much simpler control loop?

You can just install both projects into a cluster and check how much CPU and RAM are being eaten to answer that question :). (In my experience, this single deployment eats less resources, but yeah, dependant on context)

Not sure what you mean by "Native" Daemonset, or what makes the other logic simpler, as it is dependant of several components.

This difference is also already mentioned in the original answer up above.

Btw, I'd like us to stop necrobumping this closed issue and instead start opening different discussions in the 'Discussions' tab of GitHub on this repo. We can even link them here, but I think this can branch out too much for these specific sub-topics inside the project's differences.