Calico kube-policy-controller IOError: : [Errno 30] Read-only file system: '/etc/hosts'

[BlueMonday]

1. What kops version are you running? use kops version

   Version 1.8.0-beta.1 (git-2b2b5971b)

2. What Kubernetes version are you running? use kubectl version

   Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"922a86cfcd65915a9b2f69f3f193b8907d741d9c", GitTreeState:"clean", BuildDate:"2017-07-21T08:08:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?

   AWS

---

I updated my cluster using kops built from source and the calico policy controller is now crash looping.

╭─serenst@unicron  ~/go/src/k8s.io/kops  ‹master*› 
╰─$ kubectl logs -n kube-system calico-policy-controller-309338636-7hmnz
2017-11-15 23:24:51,804 7 INFO Configuring /etc/hosts
Traceback (most recent call last):
  File "<string>", line 555, in <module>
  File "<string>", line 531, in configure_etc_hosts
IOError: [Errno 30] Read-only file system: '/etc/hosts'
controller returned -1

This seems to be caused by this recent change: #3848

/etc/hosts/ is now mounted as readOnly into the kube-policy-controller:

• https://github.com/kubernetes/kops/blob/master/upup/models/cloudup/resources/addons/networking.projectcalico.org/k8s-1.6.yaml.template#L299

However the CONFIGURE_ETC_HOSTS environment variable is also set to true:

• https://github.com/kubernetes/kops/blob/master/upup/models/cloudup/resources/addons/networking.projectcalico.org/k8s-1.6.yaml.template#L292

I am guessing that the fix would be to either:

• set CONFIGURE_ETC_HOSTS tofalse (or not set it at all, the default is false)
• mount /etc/hosts/ with write permissions
• not mount /etc/hosts at all?

I don't actually know which one is the correct solution.

[BlueMonday]

@justinsb @marshallbrekka

[chrislovecnm]

#3867 is fixed in my Calico PR