Calcio > 3.8.x and CoreOS fail

[sstarcher]

**1. What kops version are you running? kops 1.15.0.beta1
**2. What Kubernetes version are you running? 1.14.8

3. What cloud provider are you using? aws and CoreOS

kops update cluster
kops rolling-update cluster

Calico fails with

 Warning  FailedMount  30s (x8 over 94s)  kubelet, ip  MountVolume.SetUp failed for volume "flexvol-driver-host" : mkdir /usr/libexec/kubernetes: read-only file system

Workaround
https://docs.projectcalico.org/v3.10/reference/faq#are-the-calico-manifests-compatible-with-coreos

Issue on calco side
projectcalico/calico#2712

[z1oy]

kops version
Version 1.15.0 (git-9992b4055)
kubernetesVersion: 1.15.6

OS: CoreOS

I have the same problem, too

[mazzy89]

This PR I've worked on should solve the thing #7545 but unfortunately it never got the right love

[r00twayne]

also seeing this issue on an upgrade from k8s v1.14.9 -> v1.15.5 . (kops 1.15.0)

[AlexLast]

Also seeing this k8s 1.14.9 => 1.15.6 running kops 1.15.0

[hakman]

I think this was fixed by #8062.

[phspagiari]

For AWS users, we added these lines on our cluster deployment script until the release of v1.15.1:

# After kops update
aws s3 cp s3://<STATE-STORE-BUCKET>/<CLUSTER-NAME>/addons/networking.projectcalico.org/k8s-1.12.yaml
sed -i 's#/usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds#/var/lib/kubelet/volumeplugins/nodeagent~uds#g' k8s-1.12.yaml
aws s3 cp k8s-1.12.yaml s3://<STATE-STORE-BUCKET>/<CLUSTER-NAME>/addons/networking.projectcalico.org/k8s-1.12.yaml
# Before kops rolling update

[ifosch]

I found the same issue when creating a cluster from scratch using kops 1.15.2, k8s 1.15.10 on AWS with latest CoreOS AMI:

Warning  FailedMount  113s (x11 over 8m5s)  kubelet, ip-192-168-9-165.ec2.internal  MountVolume.SetUp failed for volume "flexvol-driver-host" : mkdir /usr/libexec/kubernetes: read-only file system

I'm going to check on @phspagiari workaround, to see if it works in my setup.

[mazzy89]

this has been already fixed. please can you share how which is the name of the AMI you use?

[ifosch]

The AMI I'm using (us-east-1) is CoreOS-stable-2303.4.0-hvm (ami-0f2d95e41c7dac6b4)

[mazzy89]

could you please share how you pass it to kops?

[ifosch]

I use something like this:

export COREOS_AMI=$(curl -s https://coreos.com/dist/aws/aws-stable.json | jq -r '.["us-east-1"].hvm')
kops create cluster --kubernetes-version 1.15.10 --cloud=aws --image ${COREOS_AMI} ... ${NAME}

Along with some other options.

[mazzy89]

Yeah indeed. this produces an AMI ID. From kops 1.15, it expects to work correctly and mounting the volume automatically that you pass as base image in the format

075585003325/Flatcar-stable-2303.4.0-hvm

So automatically kops will mount for you that dir plugin in the right place.

[ifosch]

My bad, I understood these were interexchangeable. Thank you very much, and sorry for my mistake.

[hakman]

Thanks for clarifying this @mazzy89. If I get some extra time will make it more reliable in the future.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[hakman]

This is fixed and also CoreOS will be EOL soon.
/close

[k8s-ci-robot]

@hakman: Closing this issue.

In response to this:

This is fixed and also CoreOS will be EOL soon.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.