-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Confusions about AKS secrets encryption at rest #99
Comments
Hello @dhei,
When you use
Using |
@nilekhc We're being asked about encryption at rest of etcd for an internal security review of my team's applications hosted in AKS, and I ran across this thread in doing some research. As far as I know, we cannot use the procedure you linked to query etcd, as we don't have access to the master nodes. Am I mistaken? Or if not, is there some way to verify that the contents of etcd are encrypted? |
@nilekhc with base64 yes. We are adding kms to AKS next month. |
@nilekhc thank you for the into and info. @miwithro do you mean it's base64 encoded? My experience with base64 encoding is you can decode it with CLI / other tools...is there some other base64 scheme you are talking about that uses keys or something else? From my experience, security standards treat base64 encoded text as plaintext. |
@clauney etcd by default is encrypted with base64 encoding. So we are adding KMS to make it more secure, and until that is there we recommend to not put secrets in the cluster for the reason you eluded to. |
@miwithro thanks for that additional info. We don't really need the ability to bring our own key; this is more about the need to just have the content of secrets be encrypted to where an attacker with filesystem access couldn't just harvest the etcd content from the filesystem and access the secrets. But if we have to use KMS integration to get that and then deal with the customer managed key thing, we could do that. |
BTW - it would be better, IMO, to make this more clear in AKS documentation. Base64 encoding isn't encryption, in that it can be reversed by anyone without any other info like a key. It's also easily recognized - moreso if you find it in the context of an etcd db, since kube defaults to storing secrets that way. Anyway, if someone assumes their data is encrypted and relies on that as a control, makes such assertion to an auditor, etc. there could be ramifications for their organization. I have seen source control for Microsoft documentation in the past, and if you think whoever reviews end user PRs would be amenable, I will look for where the AKS documentation is and try to come up with an update and submit a PR. If I'm remembering wrong or if that documentation isn't in open source scope, I can just drop it and focus on KMS for our own stuff. |
@miwithro is there a plan to have AKS implement the aks-engine enableDataEncryptionAtRest feature described here? From a look at https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/, that does implement a process to encrypt secrets with a key, though it seems static; perhaps the Keyvault integration is needed for rotation, etc. Overall it's hard to map AKS features to aks-engine configs and capabilities, both current and roadmap state. But if KMS is the direction, we'll focus on that. |
@clauney actually I recommend customers use CSI Secret Store https://docs.microsoft.com/en-us/azure/aks/csi-secrets-store-driver |
Thanks! We'll check that out too, especially the synced option. I was looking to limit latency / etc. so wanted to avoid direct hits to KV, and the sync described here still fetches on demand, but at least for secrets shared among pods / services within a namespace, it may give us what we want. |
Hi @clauney couple more pointer..
|
Is there any update on enabling |
@ahmad-hamade We will have a Public Preview within the next few weeks of KMS. This will not include key rotation support, as that is targeted for GA. |
@miwithro base64 is not encrypting anyone can do https://www.base64decode.org/ or w/e to decrypt your secret this is far from being encrypted that's like having it in plaintext... |
@LiorAlafiArmo we are releasing KMS etcd encrytion for AKS in Public Preview in the next few weeks as I eluded too above. I will refactor the document at that time. |
@miwithro Is there a planned timeline or roadmap for etcd encryption GA release? |
Tentatively Aug 2022. |
Hi @miwithro, thanks for your help in the thread so far. I'm trying out the new preview KMS encryption feature by following the steps in the documentation you provided above, however at the last step "Update an exiting AKS cluster to enable KMS etcd encryption" I encountered the following error:
The specific command I used was Hoping to get some further info on this error message as I was unable to find any documentation/discussion on it. Thanks so much in advance. |
@lireanne I will add a note to the document about this. Please run az aks update --name <> --resource-group <> --enable-managed-identity --assign-identity $IDENTITY_RESOURCE_ID Then run az aks update --name --resource-group --enable-managed-identity --assign-identity $IDENTITY_RESOURCE_ID --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $KEY_ID |
Thanks @miwithro , I tried that but unfortunately still it returned |
@lireanne Are you using Keyvault with Private Link? If so, we just checked that code in and it will be available in a few weeks. |
@miwithro This is an initial POC so I actually did not configure private link access to the KV. But thank you for the info, will try again later with a private link when possible. |
@miwithro I encounter same issue with this, and I didn't configure Private link to my keyvault. Do you have any idea on this?
|
Hi @miwithro and @nilekhc , revisiting this thread with some additional questions. Please provide guidance as I'm sure we're all concerned about the security of our secrets hosted on Azure. Thanks in advance. I followed the documentation here - Set up Secrets Store CSI Driver to enable NGINX Ingress Controller with TLS - to sync an AKV certificate to a cluster. As expected, this created a secret, however when accessed with kubectl it's still returning as unencrypted data (b64-encoded only). According to Azure documentations and earlier replies, this is because "the secret is decrypted when fetched by Could you clarify:
|
Secrets Store CSI driver today does not have built in encryption.
@miwithro could you confirm this?
Yes. We recommend using KMS etcd encryption to protect Synced K8s Secrets. It would still return decrypted data with |
Just FYI, the https://docs.microsoft.com/en-us/azure/aks/use-kms-etcd-encryption feature on AKS is now general available. |
Hi @nilekhc, I'm still confused. Are AKS secrets encrypted at rest (with AKS/Azure managed keys) if we have NOT setup Thank you. |
@miwithro Could you confirm encryption at rest for AKS? @jdstone Our general recommendation is to use the KMS plug-in when synching secrets. |
Without the KMS plugin enabled on AKS, the secrets are not encrypted at rest in etcd. |
Hi @ritazh I'd like to request further clarification on the topic, as in your previous statement you confirmed that secrets are not encrypted at rest in etcd; however, the documentation of this project at this link suggests otherwise, mentioning platform managed encryption is in place.
Could you please confirm if this is or is not the case, and if so, can we have the documentation updated to be one source of truth that we can guide customers with? Thanks |
@mateuszdrab Thanks for raising this. In order to encrypt kubernetes resources at rest in etcd, you need to enable the KMS plugin feature on AKS with
This is incorrect. I have opened #201 to update the readme. |
@ritazh that's very sad to hear. From user/customer perspective we do want to include KMS. All we want is that secrets persisted (at rest) are encrypted. I was hoping Azure would think about usability and allow usage of Azure managed key. Will you consider that? |
@xiwenc You can enable encryption at rest today with a key from AKV. Are you referring to enabling the feature with a key and keyvault managed by Azure? |
@aramase indeed that's what I mean. With the current implementation using KMS a KeyVault is required. Our environment is very strict with KV's behind private endpoints. So we do prefer Azure to focus on easy of use/consumption. Managed identities for instance is a good example where it's better thought out. |
@xiwenc Thanks for the feedback. Could you open an issue here for this? |
Hi kubernetes kms team,
I have questions about AKS encryption at rest and hope to get some clarity on that.
We use Azure Key Vault Provider for Secrets Store CSI Driver on AKS to manage our secrets. Recently we noticed that the base64-encoded unencrypted kubernetes secrets can be accessed via
kubectl
commands or from azure portal. So I have done some reading on the topic and found confusing information.k8s official docs (link) says k8s secrets are unencrypted in
etcd
by default and recommends usingEncryptionConfiguration
with--encryption-provider-config
flag.AKS docs (link) says
Kubernetes secrets are stored in etcd, a distributed key-value store. Etcd store is fully managed by AKS and data is encrypted at rest within the Azure platform
. This contradicts with the unencrypted secrets we saw fromkubectl
commands or from azure portal.This repo's README (link) mentioned
AKS does encrypt secrets at rest, but keys are managed by the service and users cannot bring their own
.So my questions are:
kubectl
and portal? Does it mean that secrets are encrypted at the disk level but authorized users to the AKS cluster would still be able to read the unencrypted secrets?Thank in advance for any help or insights.
The text was updated successfully, but these errors were encountered: