ConfigMaps and Secrets mounted with subPath do not update when changed

[thesandlord]

/kind bug

What happened:

I wanted to mount a ConfigMap and a Secret directly as a file and didn't want to mount it as a full directory, so I used subPath to do so:

        volumeMounts:
          - name: my-config
            mountPath: /usr/src/app/config/config.json
            subPath: config.json
          - name: my-secret
            mountPath: /usr/src/app/secret/secret.json
            subPath: secret.json
      volumes:
      - name: my-config
        configMap:
          name: my-config
      - name: my-secret
        secret:
          secretName: my-secret

When the pod is created, it mounts the ConfigMap and Secret correctly. However, if I change them, the updates are not projected into the currently running pods. New pods get the updated file. According to the documentation, changes to a ConfigMap should be automatically propagated to running containers that mount them.

However, if I don't use subPath and instead mount the ConfigMap and Secret as a directory:

        volumeMounts:
          - name: my-config
            mountPath: /usr/src/app/config
          - name: my-secret
            mountPath: /usr/src/app/secret
      volumes:
      - name: my-config
        configMap:
          name: my-config
      - name: my-secret
        secret:
          secretName: my-secret

Then the files are updated inside the container when the underlying ConfigMap and Secret are updated and everything works as expected.

Anything else we need to know?:

In both cases, the files are being updated on the host VM. @kelseyhightower and I tried to debug this, and the only conclusion we could come up with is that subPath is using a different method to mount the files (I think it is using symlinks), and these either aren't or can't be updated for whatever reason.

Action:
The behavior that files mounted with subPath don't get updated needs to be documented, or it needs to be fixed so that subPath mounts are updated when the underlying ConfigMap or Secret changes.

Environment: GKE

• Kubernetes version (use kubectl version): 1.6.7
• Cloud provider or hardware configuration**: GKE n1-standard-1
• OS (e.g. from /etc/os-release): Container-Optimized OS 59 9460.64.0
• Kernel (e.g. uname -a): 4.4.52+

[thesandlord]

/sig storage
/sig docs

[wongma7]

Yes, symlinks are involved. Atomic writer relies on symlinks to update configmaps, secrets and such: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/atomic_writer.go. On the host, the pod volume directory looks something like this:

$ pwd
/var/lib/kubelet/pods/uuid/volumes/kubernetes.io~secret/my-secret*
$ tree -a
.
├── ..8988_09_08_14_21_15.788262705
│   └── secret.json
├── ..data -> ..8988_09_08_14_21_15.788262705
└── secret.json -> ..data/secret.json

When you subPath the secret file, that secret.json file is bind mounted into the container. But secret.json is actually a symlink to another secret.json in a timestamped folder (..8988_09_08_14_21_15.788262705). When the secret gets updated, the symlinks get changed around but the file bind mounted into the container remains the same.

Not sure what the solution is. Hope this helps somebody else think of one though :)

e: Disregarding the symlinking complexity of the atomic writer algorithm for the moment...the tool we have to atomically update the file is a rename, in that case is a solution even possible? The bind mount file will always refer to the same inode yes?

[supereagle]

@thesandlord Maybe your use case is Add ConfigMap data to a specific path in the Volume.

I follow the documentation, and it works as expected.
My configmap:

apiVersion: v1
data:
  config.json: |
    {
      "special.level": "good",
      "special.type": "charm"
    }
kind: ConfigMap
metadata:
  creationTimestamp: 2017-08-10T01:21:28Z
  name: my-config
  namespace: test
  resourceVersion: "2368592"
  selfLink: /api/v1/namespaces/test/configmaps/my-config
  uid: 3bde7d2d-7d6a-11e7-9d4a-fa163ed438e9

My pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
  namespace: test
spec:
  containers:
    - name: test-container
      image: nginx:latest
      volumeMounts:
      - name: config-volume
        mountPath: /usr/src/app/config
  volumes:
    - name: config-volume
      configMap:
        name: my-config
        items:
        - key: config.json
          path: config.json
  restartPolicy: Never

[kfox1111]

@supereagle that is not the same thing. sometimes you want to merge in a few config files into a magic config dir, like, /etc/condor/config.d, but leave the rest of the dir open for other things to drop things in. subPath is the right way to do that I think. but should be able to get updates still.

[barleyer]

@SCheng1

[bnelz]

Bump

[QiuMike]

Any update for this issue?
When I create a volume using hostPath with a file, it also has this problem.
That is: modified the host file, Pod's doesn't changed.

apiVersion: v1
kind: ReplicationController
metadata:
   name: cloud52
   namespace: kube-system
spec:
  replicas: 1
  selector:
     name: cloud52
  template:
    metadata:
     labels:
       name: cloud52
    spec:
      volumes:
      - name: hosts-volume
        hostPath:
          path: /root/k8s/hosts
      hostname: cloud52
      subdomain: cloud52
      containers:
      - image: DockerRegistry:5000/centos
        command:
          - sleep
          - "3600"
        name: cloud52
        volumeMounts:
        - name: hosts-volume
          mountPath: /etc/hosts
---
apiVersion: v1
kind: Service
metadata:
  name: cloud52
  namespace: kube-system
spec:
  selector:
    name: cloud52
  ports:
  - name: foo # Actually, no port is needed.
    port: 23
    targetPort: 23

[liggitt]

cc @kubernetes/sig-storage-bugs

[jberkus]

@maeb do you see this getting done in the next 5 days for 1.10?

[jberkus]

This issue and its related PRs will be removed from 1.10 at Code Freeze on Monday, unless they are all updated with status/approved-for-milestone and with a progress update. If that's fine, do nothing; if you are still targeting 1.10, please let us know what's going on and update the labels. Thanks!

[jberkus]

As a reminder, this issue is about to be kicked out of 1.10 tracking in 6 hours, because it doesn't have the required labels. If this is actually an 1.10 issue, please update it! @maeb @thesandlord

[msau42]

I think we just need to document this as a known limitation.

Because subpaths are bind mounted by docker, if it was a symlink, then it gets resolved to the actual path during the bindmount.

[msau42]

/assign