Unable to attach storage to pod due to timeout issues #54
Comments
|
It sounds exactly like the installation of Flexvolume driver wasn't done properly. We've covered that in our troubleshooting section. Can you check https://github.com/rancher/longhorn#volume-can-be-attacheddetached-from-ui-but-kubernetes-podstatefulset-etc-cannot-use-it to see if it helps? Also, what's your Kubernetes version and guest os version? And can you check the log of |
|
Kubernetes Version - 1.9.5 ( Also confirmed on 1.8.0 not working ) longhorn-flexvolume-driver-deployer longhorn-flexvolume-driver |
|
longhorn-manager |
|
@errorsandwarnings For 1.9.5, you need to follow https://github.com/rancher/longhorn#volume-can-be-attacheddetached-from-ui-but-kubernetes-podstatefulset-etc-cannot-use-it to check what's the path used by kubelet. Especially: |
|
@yasker : I do not think it is the same issue. I did check that one. |
|
ps aux | grep kubelet
|
|
@errorsandwarnings it's indeed sounds like the flexvolume plugin wasn't called at all. The flexvolume plugin will perform attach/detach operation on behavior of kubernetes. If the volume can be manually attach/detach through UI but cannot be attach/detach by kubernetes, it's most likely due to kubelet cannot find flexvolume plugin. Can you check the log for kubelet see if it reports error on finding longhorn flexvolume plugin? |
|
@yasker : Where will I find the kubelet log in rancher, Tried to find everywhere. I can not find it anywhere. |
|
Is kubelet deployed as a container? If not, try Which version of Rancher you're using? |
|
I am using Rancher 1.6.5 Using Rancher UI - I went to kubelets and from their logs it shows below. |
|
OK, Rancher is deploying kubelet as a container, and it lacks necessary bind mounts to make Flexvolume driver working. You need to bind-mount You may not need to bind-mount |
|
@yasker : I use Rancher in combination with cloud plugins to add hosts on fly. If I do this manually then how am I going to scale up dynamically. I will have to do this for each host I add in future then. Right ? That seems to require a fix. |
|
@yasker : How come this is working for everyone ? I am pretty much on a default rancher settings. |
|
For now, you can update the binding at https://github.com/rancher/rancher-catalog/blob/v1.6-development/infra-templates/k8s/45/docker-compose.yml.tpl#L69:26 Sorry that we missed it on 1.6. We will provide a guideline on how to enable Flexvolume on 1.6 soon. |
|
@yasker : Waiting for you to provide me a solution better than stopping these kubelet manually in each node and starting again with commands. How to do that with hundreds of nodes ? |
This is related to issue in longhorn longhorn/longhorn#54 Affects other storage drivers too like OpenEBS
|
@errorsandwarnings Just a reminder that the yaml file is the latest one, you may want to use your own version as the base rather than the latest one to prevent unintended upgrades. |
|
@yasker : Thanks, Adding a pull request for this to be merged with master. rancher/rancher-catalog#1117 |
|
@yasker : Got it. |
|
@yasker : I tried copying rancher.io~longhorn dir to the /var/lib/kubelet/volumeplugins and upgrading kubelets by the --volume-plugin-dir=/var/lib/kubelet/volumeplugins Now what I see is another issue |
|
After changing the properties of yaml by removing "" for size value it works. But permission error are coming on the volume directory. For instance jenkins container fails with the below error. |
|
Can you log in to the pod and check if the directory is writeable? It's maybe a Jenkins issue. |
|
Also you can try the latest driver at: https://raw.githubusercontent.com/yasker/longhorn-manager/work/deploy/02-components/04-driver.yaml Remind that you would need to update the FLEXVOLUME_DIR in the file to After updated FLEXVOLUME_DIR, run: This will only upgrade the driver. |
|
I, too, am running into this issue. Is there a simple way to resolve this issue, or do I need to wait until the next release of Rancher? I am running the rancher container. |
|
@wattwood You can workaround it for now:
|
|
@yasker I am on a baremetal installation with rancher:server running on a VM as a docker container. With this in mind, not using GKE, how would I modify the volume plugin directory setting, and, why move it away from the default? Right now, the default location has a file: If Kubernetes is configured to use that by default, while the output (error) is the same, is this a different issue? Do I still need to mount the default location since it's not showing up in the kubelet? |
|
I found where to modify it. |
|
Alright, my kubelet is updated: I still have the error. Should I now switch to /var/lib/kubelet/volumeplugins? The folder doesn't exist on my worker nodes. 4/3/2018 4:50:06 PME0403 22:50:06.383842 20770 desired_state_of_world_populator.go:286] Failed to add volume "volv" (specName: "pvc-56cd3c9f-3776-11e8-b95f-02d1e7c5d723") for pod "4f3bcb8a-3790-11e8-b95f-02d1e7c5d723" to desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pvc-56cd3c9f-3776-11e8-b95f-02d1e7c5d723" err=no volume plugin matched I also updated longhorn-flexvolume-driver-deployer to include: |
|
@wattwood You need to change directory to |
|
@wattwood : Wrong command. |
|
@wattwood : Same goes for flexvolume DIR. |
|
@errorsandwarnings & @yasker: It all fell into place on why it wasn't a good idea to add the missing mount to the kubelet. I did need to make sure in the longhorn-flexvolume-driver-deployer that the value had a / at the end, otherwise K8S threw a JSON error: 4/3/2018 10:51:42 PME0404 04:51:42.449075 64840 driver-call.go:237] Failed to unmarshal output for command: mount, output: "", error: unexpected end of JSON input |
|
Can you manually attach the volume to the host through longhorn UI? The error means something is wrong with the mount call. What guest OS you're using? Can you post the log of longhorn-manager? |
This is related to issue in longhorn longhorn/longhorn#54 Affects other storage drivers too like OpenEBS Adding changes by Review from @yasker
This is related to issue in longhorn longhorn/longhorn#54 Affects other storage drivers too like OpenEBS Adding changes by Review from @yasker
|
PR to the Rancher 1.6 Kubernetes has been merged. Close this issue. |
Hi,
Using longhorn as storage system. I am able to attach using longhorn UI to host but the pods can not use the storage volume. Same issue with OpenEBS storage driver too.
Storage Class
The text was updated successfully, but these errors were encountered: