configMap and secrets volumeMount are always mounted readOnly in 1.9.6

[primeroz]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:
After upgrading from 1.9.4 to 1.9.6 configMap and secrets volumes are always mounted ReadOnly even when the deployment specs don't set the option and "kubectl describe pod" show the mount ad rw

Deployment specs and kubectl describe show RW:

$ kubectl get deployment -n infra-services ldaps-proxy -o yaml | egrep -A 6 "volumeMounts:"
        volumeMounts:
        - mountPath: /etc/openldap/slapd.conf
          name: config
          subPath: slapd.conf
        - mountPath: /etc/openldap/ssl
          name: secrets
      dnsPolicy: ClusterFirst


 $ kubectl describe pod -n infra-services ldaps-proxy-99b569895-f7xpm               
...
    Mounts:
      /etc/openldap/slapd.conf from config (rw)
      /etc/openldap/ssl from secrets (rw)

Docker inspect and actual mountpoint inside of the running container show RO

kubectl exec -t -i -n infra-services ldaps-proxy-99b569895-f7xpm sh
sh-4.2# mount | grep etc                                                                                                                  
...                                   
tmpfs on /etc/openldap/ssl type tmpfs (ro,relatime,seclabel)                                          
/dev/mapper/vg_core-kube on /etc/openldap/slapd.conf type xfs (ro,relatime,seclabel,attr2,inode64,noquota)

docker inspect
 "Binds": [
 "/var/lib/kubelet/pods/2383afb0-37df-11e8-b64a-525400d41f7e/volume-subpaths/config/slapd/0:/etc/openldap/slapd.conf:ro,Z,rslave",
 "/var/lib/kubelet/pods/2383afb0-37df-11e8-b64a-525400d41f7e/volumes/kubernetes.io~secret/secrets:/etc/openldap/ssl:ro,Z,rslave",
...
...
 ],

What you expected to happen:

Mountpoints should be RW as they used to be at least up to 1.9.4 ( I never went through 1.9.5 )

How to reproduce it (as minimally and precisely as possible):

Have a 1.9.6 cluster, apply the following spec and verify that mounts are RO rather than RW

apiVersion: v1
data:
  file.conf: "i am a file"
kind: ConfigMap
metadata:
  labels:
    app: test-mount-ro
  name: test-mount-ro
  namespace: kube-system
---
apiVersion: v1
data:
  secret.conf: VGVzdCBzZWNyZXQK
kind: Secret
metadata:
  labels:
    app: test-mount-ro
  name: test-mount-ro
  namespace: kube-system
type: Opaque
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: test-mount-ro
  name: test-mount-ro
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-mount-ro
  template:
    metadata:
      labels:
        app: test-mount-ro
    spec:
      containers:
      - args:
        - "3600"
        command:
        - sleep
        image: busybox
        imagePullPolicy: Always
        name: busybox
        resources:
          limits:
            cpu: 100m
            memory: 250Mi
          requests:
            cpu: 100m
            memory: 250Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp/config
          name: config
          subPath: file.conf
          readOnly: false
        - mountPath: /tmp/secret
          name: secrets
          subPath: secret.conf
          readOnly: false
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 0666
          name: test-mount-ro
        name: config
      - name: secrets
        secret:
          defaultMode: 0666
          secretName: test-mount-ro

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6+coreos.0", GitCommit:"c2aac694c2c7373dfe3ad81f47fc3cbc70a5a8fa", GitTreeState:"clean", BuildDate:"2018-03-21T21:54:22Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

Baremetal and KVM Vms

• OS (e.g. from /etc/os-release):

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1632.3.0
VERSION_ID=1632.3.0
BUILD_ID=2018-02-14-0338
PRETTY_NAME="Container Linux by CoreOS 1632.3.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

• Kernel (e.g. uname -a):

Linux worker01.dev3.london.k8s.mintel.ad 4.14.19-coreos #1 SMP Wed Feb 14 03:18:05 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5320 @ 1.86GHz GenuineIntel GNU/Linux

• Install tools: Matchbox
• Others:
  It used to work fine on 1.9.4 , i upgraded to 1.9.6 due to Detect backsteps correctly in base path detection #61080 and problem started.

Tested with minikube 1.9.3 and it works as expected , the configmap and secrets are mounted RW

[dims]

/sig storage

[gnufied]

This was intentional. More details https://github.com/kubernetes/kubernetes/pull/60258/files . If you really want configMap or secrets to be rw then you can disable the feature gate - ReadOnlyAPIDataVolumes.

[liggitt]

making atomic writer volumes (secret, configmap, downwardAPI, projected) readonly was part of the fix for #60814

Fix impact:
Secret, configMap, downwardAPI and projected volumes will be mounted as read-only volumes. Applications that attempt to write to these volumes will receive read-only filesystem errors. Previously, applications were allowed to make changes to these volumes, but those changes were reverted at an arbitrary interval by the system. Applications should be re-configured to write derived files to another location.

/close

[primeroz]

Ok Great, i must have missed it in the changelog . Already did work around it so all good

thanks!

[nitindwivedicloud]

@primeroz - what is the work aboud?

-Thanks

[komljen]

My approach is to copy the config map to emptyDir volume if I need rw capabilities helm/charts@e4b7d0b

[mingfang]

I don't agree with this.
I should have the option to control readOnly flag.
In fact when I describe the pod the confmap and secrets do show rw.