Kubelet silently deletes any files written to secret volume

[joelsmith]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:
When a container has a secret volume and creates or updates a file on the secret volume, the changes disappear within a minute or two. This behavior is confusing and surprising to container creators. Because the volume acts read-only (by reverting any changes made), a better solution would be do disallow writes to the volume so that applications can fail early if they attempt to write to the volume.

What you expected to happen:
Containers should be prevented from writing to secret volumes (and other volumes like them like downwardApi, configMap or projected volumes).
How to reproduce it (as minimally and precisely as possible):

Create a file pod.yaml with this content:

apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: Secret
  data:
    secret: bXkgc2VjcmV0Cg==
  metadata:
    name: secrets
  type: Opaque
- apiVersion: v1
  kind: Pod
  metadata:
    name: busybox-pod
  spec:
    containers:
    - image: gcr.io/google_containers/busybox:latest
      name: busybox
      command: ["/bin/sh"]
      args: ["-c", "sleep 10; echo before; ls -l /secrets; touch /secrets/myfile; echo checking; ls -l /secrets; sleep 120; echo checking again; ls -l /secrets; while sleep 3600; do true; done"]
      volumeMounts:
      - mountPath: /secrets
        name: secrets
    restartPolicy: OnFailure
    volumes:
    - name: secrets
      secret:
        secretName: secrets

Then create a secret and pod via kubectl create -f pod.yml

Watch the logs for 2 minutes via kubectl logs -f busybox-pod
You should see the pod create myfile and then 2 minutes later when it checks again, myfile will be gone.

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.110+3d652cae0391cf-dirty", GitCommit:"3d652cae0391cfeff81f3e96f1ed7d558f171626", GitTreeState:"dirty", BuildDate:"2018-01-23T16:21:15Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.110+3d652cae0391cf-dirty", GitCommit:"3d652cae0391cfeff81f3e96f1ed7d558f171626", GitTreeState:"dirty", BuildDate:"2018-01-23T16:21:15Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

[joelsmith]

/assign

[joelsmith]

/sig node
/sig storage

[gertcuykens]

Config volumes and secret volumes on a subdirectory of the config mount path also have this side effect, see ref 458

       volumeMounts:
        - name: tls-volume
          mountPath: /etc/nats/tls
        - name: config-volume
          mountPath: /etc/nats
      volumes:
      - name: tls-volume
        secret:
          secretName: tls-nats-server
      - name: config-volume
        configMap:
          name: nats-config

Secrets will first appear in your program and then disappear

[joelsmith]

@gertcuykens I think that should be fixed now that #57422 has merged. I don't know what you mean by ref 458. (Edit: oh, you're referring to an issue in another repo).