Mounting/unmounting volume error on GKE

[rwyyr]

Greetings all. I'm running my cluster on the GKE and after upgrading to 1.3.3 I've been having errors of the following format:

Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message

8m 4m 20 {default-scheduler } Warning FailedScheduling no nodes available to schedule pods
3m 3m 1 {default-scheduler } Normal Scheduled Successfully assigned mongodb-321322295-mnmo7 to gke-test-cluster-default-pool-f56cf9ee-f2kb
1m 1m 1 {kubelet gke-test-cluster-default-pool-f56cf9ee-f2kb} Warning FailedMount Unable to mount volumes for pod "mongodb-321322295-mnmo7_dev(20c985df-5882-11e6-ac8b-42010af00014)": timeout expired waiting for volumes to attach/mount for pod "mongodb-321322295-mnmo7"/"dev". list of unattached/unmounted volumes=[mongodb-data]
1m 1m 1 {kubelet gke-test-cluster-default-pool-f56cf9ee-f2kb} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "mongodb-321322295-mnmo7"/"dev". list of unattached/unmounted volumes=[mongodb-data]

Evidently, that's not something specific to mongodb-data volume. I get these same errors for other volumes too (majority of them are secrets). I'm anticipating the possibility of upgrading to 1.3.4 release on GKE as I've read that it possibly resolves the problem but it's not available yet.

But for now I'd like to know if there're some workarounds/hacks for the problem that could allow me to continue deploying my services?

Also, huge thanks to the whole Kubernetes community. You're awesome 🖖🏻

[mwuertinger]

I'm experiencing the same problems on multiple clusters, all running 1.3.3 on the nodes. I'm seeing this with secrets as well as with persistent disks.

Here's what I get in /var/log/kubelet.log in a recent incident (Removed some unrelated logs from the output):

E0804 10:37:23.625498    4237 goroutinemap.go:155] Operation for "kubernetes.io/gce-pd/jenkins-co" failed. No retries permitted until 2016-08-04 10:37:24.125485506 +0000 UTC (durationBeforeRetry 500ms). error: MountVolume.WaitForAttach failed for volume "kubernetes.io/gce-pd/jenkins-co" (spec.Name: "jenkins-data") pod "03bdc3f3-5a2e-11e6-91e2-42010af0000b" (UID: "03bdc3f3-5a2e-11e6-91e2-42010af0000b") with: Could not find attached GCE PD "jenkins-co". Timeout waiting for mount paths to be created.
I0804 10:37:24.181044    4237 reconciler.go:179] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/jenkins-co" (spec.Name: "jenkins-data") pod "38981380-5a2f-11e6-91e2-42010af0000b" (UID: "38981380-5a2f-11e6-91e2-42010af0000b")
I0804 10:37:24.184064    4237 operation_executor.go:959] Controller successfully attached volume "kubernetes.io/gce-pd/jenkins-co" (spec.Name: "jenkins-data") pod "38981380-5a2f-11e6-91e2-42010af0000b" (UID: "38981380-5a2f-11e6-91e2-42010af0000b") devicePath: "/dev/disk/by-id/google-jenkins-co"
I0804 10:37:24.281385    4237 reconciler.go:253] MountVolume operation started for volume "kubernetes.io/gce-pd/jenkins-co" (spec.Name: "jenkins-data") to pod "38981380-5a2f-11e6-91e2-42010af0000b" (UID: "38981380-5a2f-11e6-91e2-42010af0000b"). 
I0804 10:37:24.281468    4237 operation_executor.go:629] Entering MountVolume.WaitForAttach for volume "kubernetes.io/gce-pd/jenkins-co" (spec.Name: "jenkins-data") pod "38981380-5a2f-11e6-91e2-42010af0000b" (UID: "38981380-5a2f-11e6-91e2-42010af0000b") DevicePath: "/dev/disk/by-id/google-jenkins-co"
E0804 10:37:54.107339    4237 kubelet.go:1930] Unable to mount volumes for pod "jenkins-8r2qa_jenkins-co(38981380-5a2f-11e6-91e2-42010af0000b)": timeout expired waiting for volumes to attach/mount for pod "jenkins-8r2qa"/"jenkins-co". list of unattached/unmounted volumes=[jenkins-data]; skipping pod
E0804 10:37:54.107392    4237 pod_workers.go:183] Error syncing pod 38981380-5a2f-11e6-91e2-42010af0000b, skipping: timeout expired waiting for volumes to attach/mount for pod "jenkins-8r2qa"/"jenkins-co". list of unattached/unmounted volumes=[jenkins-data]
I0804 10:38:09.277105    4237 reconciler.go:253] MountVolume operation started for volume "kubernetes.io/secret/ssl" (spec.Name: "ssl") to pod "38981380-5a2f-11e6-91e2-42010af0000b" (UID: "38981380-5a2f-11e6-91e2-42010af0000b"). Volume is already mounted to pod, but remount was requested.
I0804 10:38:09.277236    4237 reconciler.go:253] MountVolume operation started for volume "kubernetes.io/secret/default-token-h26dn" (spec.Name: "default-token-h26dn") to pod "38981380-5a2f-11e6-91e2-42010af0000b" (UID: "38981380-5a2f-11e6-91e2-42010af0000b"). Volume is already mounted to pod, but remount was requested.
I0804 10:38:09.282795    4237 operation_executor.go:720] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/default-token-h26dn" (spec.Name: "default-token-h26dn") pod "38981380-5a2f-11e6-91e2-42010af0000b" (UID: "38981380-5a2f-11e6-91e2-42010af0000b").
I0804 10:38:09.283012    4237 operation_executor.go:720] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ssl" (spec.Name: "ssl") pod "38981380-5a2f-11e6-91e2-42010af0000b" (UID: "38981380-5a2f-11e6-91e2-42010af0000b").

[mwuertinger]

I solved it temporarily by deleting the affected node.

[rwyyr]

Just a quick update: I temporarily resolved the problem by downgrading my cluster's nodes to 1.2.6 and haven't experienced the problem yet albeit I've found in similar topics that it doesn't resolve the problem. But so far it works.

[mwuertinger]

Maybe it was just resolved because the node got recreated like in my case?

[rwyyr]

@mwuertinger yes, I assume that it might have been the case.

[mwuertinger]

@mesmerizingr do you happen to have some background information about this issue so that we can reference it here?

[mwuertinger]

This seems to be related: #29617

[rwyyr]

@mwuertinger take a look on this thread: #28750

Also, one may find that the problem was acknowledged in the following release notes: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md/#v133

The most reassuring comment I've got was: #28750 (comment)

So, if you aren't vendor-locked on GKE and have complete control on your cluster including its master I should upgrade it to 1.3.4.

[jingxu97]

@mwuertinger @mesmerizingr, thank you for reporting your issue. Yes, we
recently fixed a few issues for mounting/umounting volumes in v1.3.3. The
fix is already in v1.3.4. Please upgrade to v1.3.4 and kindly let us know
if you still have problems.

Jing

On Thu, Aug 4, 2016 at 7:46 AM, Roman notifications@github.com wrote:

@mwuertinger https://github.com/mwuertinger take a look on this thread:
#28750 #28750

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#29903 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASSNxSOTF3WlyjFGuOv-zrLj-wNuMwEeks5qcfs1gaJpZM4JaVMN
.

• Jing

[mwuertinger]

Thanks for confirming, @jingxu97
We're on Google Container Engine so we have to wait until they approve v1.3.4 there.

[mheese]

@mwuertinger, if you are on GKE, downgrading to 1.3.2 seemed to have solved the issue on at least one of my clusters ... however, I didn't really follow up anymore (by restarting pods with PDs and secret volumes) as I was just happy that the cluster was healthy again.

[rwyyr]

@mheese thank for sharing this information. I'll just wait for 1.3.4 upgrade. Downgrading cluster's nodes to 1.2.6 resolved the problem.

[mheese]

@mesmerizingr: false alarm: it was still happening today .... only the downgrade to 1.2.6 really fixes it ... thx for the hint 👍