kubelet does not refresh immutable secrets after recreation as documentation says

[mbrancato]

What happened?

The docs say this about immutable Secrets
https://kubernetes.io/docs/concepts/configuration/secret/#secret-immutable-create

Note:
Once a Secret or ConfigMap is marked as immutable, it is not possible to revert this change nor to mutate the contents of the data field. You can only delete and recreate the Secret. Existing Pods maintain a mount point to the deleted Secret - it is recommended to recreate these pods.

However, we're seeing different behavior. After the "delete and recreate" we see a mix of behavior. First off, if we recreate the Secret with new keys. New pods cannot mount these keys, but they can mount previously existing keys. If we add a new key in the newly created Secret (with the same name), we'll get an error like this:

Error: couldn't find key FOO in Secret test/mysecret

But if we view the Secret with kubectl the new key exists. There is also a more difficult to reproduce problem where some pods get old values in the existing keys, and some pods get new values. I think all the symptoms stem from the same issue, which is that the kubelet caches the immutable secret forever. The documentation indicates that on Pod create, this would be refreshed, but that does not seem to happen.

Is the documentation incorrect in that the kubelet persists the data forever and may never update it?

As a pretty harsh workaround, I recreated nodes in the cluster and these fresh Nodes pulled the updated Secret and values.

What did you expect to happen?

After recreating the immutable secret, new pods would be able to access the updated values.

How can we reproduce it (as minimally and precisely as possible)?

• Create an immutable Secret
• Start and mount the Secret to a Pod
• Delete the immutable Secret
• Recreate the immutable Secret with new values
• Start and mount the Secret to a new Pod that is scheduled on the same node as the previous Pod

Anything else we need to know?

Likely related issue: #124701

Kubernetes version

Details

$ kubectl version
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.30.10-gke.1070000

Cloud provider

Details

GKE

OS version

Details

Google COS

Install tools

Details

Container runtime (CRI) and version (if applicable)

Details

Related plugins (CNI, CSI, ...) and versions (if applicable)

Details

[mbrancato]

/sig storage

[liggitt]

cc @wojtek-t

[liggitt]

See also the discussion in #124701 (comment)

At the very least we should update the documentation to match what actually happens, even if what currently happens is not what we want long-term (referencing issues tracking the desired changes in documentation mentioning sharp edges is useful)

[carlory]

/sig node

[kei01234kei]

Hi @liggitt
May I take on it?

[liggitt]

Hi @liggitt
May I take on it?

Updating the documentation now would be a help. Tag @wojtek-t for review of the website PR.

Before trying to change the behavior, pull in node and scalability folks involved in the original feature (https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/1412-immutable-secrets-and-configmaps/kep.yaml#L3-L16)... the entire point of immutable configmaps and secrets was to improve kubelet performance by letting it assume those objects would never change. Anything we do to make the kubelet invalidate and refetch those objects at any point seems like it will undo what that feature was trying to accomplish, at least a little.