Nested data volumes (e.g. /secrets & /secrets/more) cause unexpected behavior

[joelsmith]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

Data volumes backed by the API such as secret, configmap, downwardAPI and projected volumes, appear to do strange things when nested. Since each mounted volume is intended to correspond 1:1 with the API object data. Any change to the data causes the kubelet to load a new version of the data and to replace all of the old volume data with the newly-loaded data.

1. When the kubelet asks Docker to mount the nested volume into the base volume, Docker creates a mount point in the base directory upon which it can mount the nested volume.

2. The kubelet views the existence of the mount point directory in the base directory as a change that puts the volume out of sync with the data.

3. The kublet attempts to remove the old versions of the secrets and re-symlink the data.
   a. on an old-enough kernel (e.g. 3.10.0-514.21.1.el7.x86_64), the remove attempt fails because the directory is an in-use mount point and gives an error of "Device or resource busy".
   b. on a newer kernel (e.g. 3.10.0-693.5.2.el7.x86_64), the remove succeeds despite the directory being an active mount point and the nested mount is no longer visible from within the container, despite its being mounted.

4. On the old kernel, the kubelet continuously repeats step 3 because every re-sync iteration makes it think that it needs a refresh due to the constant presence of the mountpoint directory. Additionally, every iteration leaves behind an old copy of the data since an error is encountered prior to its removal. For example, you might end up with 100 directories that look like /secrets/..YYYY_MM_DD_HH_MM_SS.XXXXXXXXX after a few hours.

What you expected to happen:
As long as the name of a nested volume doesn't interfere with the name of a data item from a parent volume, I would expect the nested volume to work.

How to reproduce it (as minimally and precisely as possible):

1. Create a yaml file with 2 secrets and a pod that makes volumes for the 2 secrets, with one of them mounted within the other:

apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: Secret
  data:
    secret: bXkgc2VjcmV0Cg==
  metadata:
    name: secrets
  type: Opaque
- apiVersion: v1
  kind: Secret
  data:
    secret: c2VjcmV0Cg==
  metadata:
    name: moresecrets
  type: Opaque
- apiVersion: v1
  kind: Pod
  metadata:
    name: busybox-pod
  spec:
    containers:
    - image: gcr.io/google_containers/busybox:latest
      name: busybox
      command: ["/bin/sh"]
      args: ["-c", "while sleep 3600; do true; done"]
      volumeMounts:
      - mountPath: /secrets
        name: secrets
      - mountPath: /secrets/more
        name: moresecrets
    restartPolicy: OnFailure
    volumes:
    - name: secrets
      secret:
        secretName: secrets
    - name: moresecrets
      secret:
        secretName: moresecrets

2. Create the 2 secrets and the pod. Assuming you named the above file pod.yaml:

kubectl create -f pod.yaml

3. Verify that the bug is happening. Wait a few minutes to make sure that a sync cycle has happened, then look at the containers /secrets directory:

kubectl exec -ti busybox-pod -- sh -c 'find /secrets; echo -n Secret:; cat /secrets/more/secret'

If you have a new kernel, /secrets/more/secret won't exist. If you have an old kernel, it will exist but you should see several timestamp directories in /secrets (one for each sync iteration) that aren't being cleaned up.

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.90+157fb7b66cff84", GitCommit:"157fb7b66cff84e77f02110718509facd13681c5", GitTreeState:"clean", BuildDate:"2017-12-19T20:57:27Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.90+157fb7b66cff84", GitCommit:"157fb7b66cff84e77f02110718509facd13681c5", GitTreeState:"clean", BuildDate:"2017-12-19T20:57:27Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

[joelsmith]

/sig storage

[joelsmith]

/sig storage
/sig node
/assign joelsmith