Unable to mount volume

[r-moiseev]

Is this a bug report or feature request?

• Bug Report

Bug Report

What happened:
I have deployed rook-operator with helm chart and very basic cluster and block-storage pool from doc.
When i created PVC, PV was created, but can not be mounted to a pod. I have this:

Unable to mount volumes for pod ***": timeout expired waiting for volumes to attach/mount for pod "***". list of unattached/unmounted volumes=[data]

I can see that volume was successfully created in operator logs

What you expected to happen:
Volume mounted to a pod

How to reproduce it (minimal and precise):
Deploy rook-alpha/rook with helm
Create Cluster as described in quickstart
Create Blockstorage Pool as described in quickstart
Create SC for this Pool
Create PVC

Environment:

• OS ubuntu 15.04
• Kernel (e.g. uname -a): Linux dev1-master-0 4.13.0-1018-azure Apply crush map topology settings from the command line #21-Ubuntu SMP Thu May 17 13:58:38 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Cloud provider or hardware configuration: AzureRM
• Rook version (use rook version inside of a Rook Pod): v0.7.1
• Kubernetes version (use kubectl version): 0.9.5
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Kubespray on Azure
• Ceph status (use ceph health in the Rook toolbox): HEALTH_OK

[galexrt]

@r-moiseev Can you check your kubelet logs for volume mount errors such as no volume plugin matched?
Is your kubelet configured to use the same flexvolume directory as the Rook operator? See https://rook.io/docs/rook/v0.7/flexvolume.html.

[galexrt]

@zhwenh Seems to be a Kubernetes issue/PR description you have posted. Could you elaborate on your comments? I don't see how this is directly related to this issue.

[r-moiseev]

I have this error in kubelet:

Error processing volume "data" for pod "drone-drone-server-865fbf7fbd-njjt9_ci(8a659735-6f13-11e8-9190-000d3a22a084)": error processing PVC "ci"/"drone-drone": failed to fetch PVC ci/drone-drone from API server. err=persistentvolumeclaims "drone-drone" not found

And yes, 'no volume plugin matched' is there

How do i fix this on ubuntu?

[galexrt]

@r-moiseev The no volume plugin matched means that the kubelet can't find the Rook volume plugin in the flexvolume directory.
Can you check that your flexvolume plugin dir path contains a directory named rook.io~rook. Check that the Rook operator has the FLEXVOLUME_DIR_PATH environment variable set to the same path as the kubelet flexvolume plugin dir path. In both cases try deleting the rook-agent DaemonSet and restarting the Rook operator Pod and see if after the rook-agent DaemonSet has been created a gain the error has gone away.

[r-moiseev]

@galexrt rook-operator has no FLEXVOLUME_DIR_PATH variable at all

[galexrt]

@r-moiseev But your kubelets have the flexvolume dir flag set? If so, delete the rook-agent DaemonSet and add the FLEXVOLUME_DIR_PATH env var to the operator.
See https://rook.io/docs/rook/v0.7/flexvolume.html.

[r-moiseev]

Manually added this variable to helm chart, redeployed operator, recreated agent, and its working now.

Thank you.

[galexrt]

@r-moiseev To document here the helm chart also has an option for it https://rook.io/docs/rook/v0.7/helm-operator.html#configuration agent.flexVolumeDirPath.

[aziemski]

for k8s v1.11.0 & ubuntu 16.04 setting FLEXVOLUME_DIR_PATH to default kubernetes volume plugin directory helps

        - name: FLEXVOLUME_DIR_PATH
          value: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec"

[allen13]

I'm running on openshift-enterprise 3.9.33 with rook v0.8 and I'm having this issue even when the FLEXVOLUME_DIR_PATH is set.

journalctl -f -u atomic-openshift-node

Jul 22 19:28:18 ocp-06 atomic-openshift-node[2438]: E0722 19:28:18.930350 2462 desired_state_of_world_populator.go:289] Failed to add volume "alertmanager-main-db" (specName: "pvc-f282459a-8e1f-11e8-b0ef-0800270fad8c") for pod "f2844a0f-8e1f-11e8-b0ef-0800270fad8c" to desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pvc-f282459a-8e1f-11e8-b0ef-0800270fad8c" err=no volume plugin matched

$ ls /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
ceph.rook.io~rook  ceph.rook.io~rook-ceph-system  rook.io~rook  rook.io~rook-ceph-system

I've deleted everything and started from scratch just to make everthing got restarted including the agent deamonsets.

Also, I've made sure to follow every single step exactly from here https://github.com/rook/rook/blob/master/Documentation/openshift.md

Willing to go the extra mile to get this working.

[allen13]

I was able to resolve this by changing the FLEXVOLUME_DIR_PATH var to /etc/origin/kubelet-plugins/volume/exec .

Apparently, the type of openshift install you use affects where the plugin path resides: https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_flex_volume.html

[jbw976]

This speaks to the big disappointment of no longer being able to programmatically "discover" the right path the Kubelet will be looking in for flexvolumes, as discussed in #1330 (comment). We cannot do it programmatically from the agent, but I'm wondering if we can improve our documentation to help users figure out where the Kubelet will be looking, so that they know what exactly to set FLEXVOLUME_DIR_PATH to in operator.yaml.

[galexrt]

@jbw976 I'm working on an overhaul of the flexvolume docs which will also add OpenShift doc links and readd Atomic.

[galexrt]

The docs have been overhauled, please give it a look here: https://rook.io/docs/rook/v0.8/flexvolume.html.
I'm closing this for now as I believe the docs have been updated to better the user experience.
Please create a new issue if you think the flexvolume docs can be further improved. Thank you!

[ToroNZ]

Future travelers:

I had all these symptoms but in my case it was resolved by restarting the kubelet.
K8s 1.11
Fedora 28

I followed an old guide for k8s 1.7 https://github.com/rook/rook/blob/master/Documentation/common-issues.md#kubelet-restart

[wallnerryan]

thanks @ToroNZ on kubernetes 1.12 and rook/ceph 0.8 and had to restart kubelet as well on 1/3 workers. All plugin directories were there otherwise, and restart fixed a stuck ContainerCreating

[Harold2017]

same issue, but with wrong fs type. i have to change fstype from xfs to ext4 to sucecssfully mount.