chown: /consul/config: Read-only file system

[tscolari]

I'm getting this when I try to start the statefulset

NAME       READY     STATUS             RESTARTS   AGE       IP              NODE
consul-0   0/1       CrashLoopBackOff   1          12s       10.200.100.13   k8s-node-2

kubectl logs consul-0
chown: /consul/config: Read-only file system

I didn't see any other issue here so I'm assuming I did something really wrong,
but /consul/config is the mount point for the configmap, so it makes sense that it's read only I guess.

[lsgrep]

You have to change mount path from /consul/config to /consul/myconfig, then you can mount ConfigMap to this directory. I guess mounting does not work if that directory already exists. I changed the mount path and it worked.

[dinar-dalvi]

@lsgrep good one, thanks it worked for me as well. I am using k8s 1.8.10.

[mgyong]

@dinar-dalvi @lsgrep Can you guys be more explicit on the change that was done in the yaml file? I don't fully understand where precisely is the change

[martwetzels]

@mgyong I just ran into the same issue. Look at the statefulset/consul.yaml file at the volumeMounts.

volumeMounts:
            - name: data
              mountPath: /consul/data
            - name: config
              mountPath: /consul/myconfig <-- this one was originally /consul/config
            - name: tls
              mountPath: /etc/tls

Don't forget to update the config-file args in the same file:

- "-config-file=/consul/myconfig/server.json"

Hope that helps.

[mgyong]

@martwetzels It works now. Thks. I forgot to update the config-file args in the same file

[innovia]

to anyone getting stuck with this, you don't have to use a config map
here's the full explanation:

since 1.8.9 the option to readwrite from configmap is disabled unless you configure the kubelet like so

kubelet:
    featureGates:
      ReadOnlyAPIDataVolumes: "true"

this is not going to work if you already in a failed state.

now for the fix:

you need to remove the config map volume mount but do not remove the config-file flag from the yaml

step 1: add the config as json string to the nev var CONSUL_LOCAL_CONFIG

env: 
  - name: CONSUL_LOCAL_CONFIG
    value: '{
                "ca_file": "/etc/tls/ca.pem",
                "cert_file": "/etc/tls/consul.pem",
                "key_file": "/etc/tls/consul-key.pem",
                "verify_incoming": true,
                "verify_outgoing": true,
                "verify_server_hostname": true,
                "ports": {
                  "https": 8443
                }
       }'

step 2 point to the generated file:

now docker entry point will auto create a file under /consul/config/local.json so you must update the -config-file flag to match under the args
- "-config-file=/consul/config/local.json"

this instructs consul to get its config from an env var instead of a file where the docker image will attempt to chown the config for the console user.

[kelseyhightower]

Fixed on master.