ConfigMaps and Secrets mounted with subPath do not update when changed

[thesandlord]

/kind bug

What happened:

I wanted to mount a ConfigMap and a Secret directly as a file and didn't want to mount it as a full directory, so I used subPath to do so:

        volumeMounts:
          - name: my-config
            mountPath: /usr/src/app/config/config.json
            subPath: config.json
          - name: my-secret
            mountPath: /usr/src/app/secret/secret.json
            subPath: secret.json
      volumes:
      - name: my-config
        configMap:
          name: my-config
      - name: my-secret
        secret:
          secretName: my-secret

When the pod is created, it mounts the ConfigMap and Secret correctly. However, if I change them, the updates are not projected into the currently running pods. New pods get the updated file. According to the documentation, changes to a ConfigMap should be automatically propagated to running containers that mount them.

However, if I don't use subPath and instead mount the ConfigMap and Secret as a directory:

        volumeMounts:
          - name: my-config
            mountPath: /usr/src/app/config
          - name: my-secret
            mountPath: /usr/src/app/secret
      volumes:
      - name: my-config
        configMap:
          name: my-config
      - name: my-secret
        secret:
          secretName: my-secret

Then the files are updated inside the container when the underlying ConfigMap and Secret are updated and everything works as expected.

Anything else we need to know?:

In both cases, the files are being updated on the host VM. @kelseyhightower and I tried to debug this, and the only conclusion we could come up with is that subPath is using a different method to mount the files (I think it is using symlinks), and these either aren't or can't be updated for whatever reason.

Action:
The behavior that files mounted with subPath don't get updated needs to be documented, or it needs to be fixed so that subPath mounts are updated when the underlying ConfigMap or Secret changes.

Environment: GKE

• Kubernetes version (use kubectl version): 1.6.7
• Cloud provider or hardware configuration**: GKE n1-standard-1
• OS (e.g. from /etc/os-release): Container-Optimized OS 59 9460.64.0
• Kernel (e.g. uname -a): 4.4.52+

[thesandlord]

/sig storage
/sig docs

[wongma7]

Yes, symlinks are involved. Atomic writer relies on symlinks to update configmaps, secrets and such: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/atomic_writer.go. On the host, the pod volume directory looks something like this:

$ pwd
/var/lib/kubelet/pods/uuid/volumes/kubernetes.io~secret/my-secret*
$ tree -a
.
├── ..8988_09_08_14_21_15.788262705
│   └── secret.json
├── ..data -> ..8988_09_08_14_21_15.788262705
└── secret.json -> ..data/secret.json

When you subPath the secret file, that secret.json file is bind mounted into the container. But secret.json is actually a symlink to another secret.json in a timestamped folder (..8988_09_08_14_21_15.788262705). When the secret gets updated, the symlinks get changed around but the file bind mounted into the container remains the same.

Not sure what the solution is. Hope this helps somebody else think of one though :)

e: Disregarding the symlinking complexity of the atomic writer algorithm for the moment...the tool we have to atomically update the file is a rename, in that case is a solution even possible? The bind mount file will always refer to the same inode yes?

[supereagle]

@thesandlord Maybe your use case is Add ConfigMap data to a specific path in the Volume.

I follow the documentation, and it works as expected.
My configmap:

apiVersion: v1
data:
  config.json: |
    {
      "special.level": "good",
      "special.type": "charm"
    }
kind: ConfigMap
metadata:
  creationTimestamp: 2017-08-10T01:21:28Z
  name: my-config
  namespace: test
  resourceVersion: "2368592"
  selfLink: /api/v1/namespaces/test/configmaps/my-config
  uid: 3bde7d2d-7d6a-11e7-9d4a-fa163ed438e9

My pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
  namespace: test
spec:
  containers:
    - name: test-container
      image: nginx:latest
      volumeMounts:
      - name: config-volume
        mountPath: /usr/src/app/config
  volumes:
    - name: config-volume
      configMap:
        name: my-config
        items:
        - key: config.json
          path: config.json
  restartPolicy: Never

[kfox1111]

@supereagle that is not the same thing. sometimes you want to merge in a few config files into a magic config dir, like, /etc/condor/config.d, but leave the rest of the dir open for other things to drop things in. subPath is the right way to do that I think. but should be able to get updates still.

[barleyer]

@SCheng1