hostpath.csi.k8s.io not found in the list of registered CSI drivers

[eldritch-dream]

While running through the example with microK8s kubernetes I keep running into this error when trying to actually start the my-csi-app pod

Warning FailedMount 5s (x6 over 22s) kubelet, rkimmel-dt MountVolume.MountDevice failed for volume "pvc-4d70c760-a4b0-11e9-8376-1866da0e836f" : driver name hostpath.csi.k8s.io not found in the list of registered CSI drivers

Looks like for some reason the hostpath csi driver is not getting registered but I'm not sure where to look to fix that or find out why it's happening.

Any help with this would be greatly appreciated

[pohly]

Use the example from https://github.com/kubernetes-csi/csi-driver-host-path/releases/tag/v1.1.0.

We are in a transition from the old driver name without domain part to "hostpath.csi.k8s.io" and your deployment still seems to use the old name.

[eldritch-dream]

Thanks for your earlier reply I appreciate you taking the time to respond so quickly! After a few different attempts at using the 1.1.0 release I am still having roughly the same issue unfortunately. Overall it looks like for some reason the driver registration is not happening at all even with this release. Is there somewhere I can look for for why this might be happening?

Output of get pods

NAME READY STATUS RESTARTS AGE
csi-hostpath-attacher-0 1/1 Running 0 4m52s
csi-hostpath-provisioner-0 1/1 Running 0 4m50s
csi-hostpath-snapshotter-0 1/1 Running 0 4m48s
csi-hostpath-socat-0 1/1 Running 0 4m47s
csi-hostpathplugin-0 3/3 Running 0 4m51s
my-csi-app 0/1 ContainerCreating 0 2m58s

Output of get pv

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-90d2bb47-a4df-11e9-9e80-1866da0e836f 1Gi RWO Delete Bound default/csi-pvc csi-hostpath-sc 2m52s

Output of get pvc

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
csi-pvc Bound pvc-90d2bb47-a4df-11e9-9e80-1866da0e836f 1Gi RWO csi-hostpath-sc 4m31s

Output of describe my-csi-app

Name: my-csi-app
Namespace: default
Priority: 0
PriorityClassName:
Node: rkimmel-dt/172.18.22.1
Start Time: Fri, 12 Jul 2019 15:59:55 -0400
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"my-csi-app","namespace":"default"},"spec":{"affinity":{"podAffinity":...
Status: Pending
IP:
Containers:
my-frontend:
Container ID:
Image: busybox
Image ID:
Port:
Host Port:
Command:
sleep
1000000
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
Mounts:
/data from my-csi-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4lrh7 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
my-csi-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: csi-pvc
ReadOnly: false
default-token-4lrh7:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-4lrh7
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

---

Warning FailedScheduling 5m22s (x3 over 5m22s) default-scheduler persistentvolumeclaim "csi-pvc" not found
Warning FailedScheduling 4m48s (x6 over 5m10s) default-scheduler pod has unbound immediate PersistentVolumeClaims
Normal Scheduled 4m48s default-scheduler Successfully assigned default/my-csi-app to rkimmel-dt
Normal SuccessfulAttachVolume 4m47s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-90d2bb47-a4df-11e9-9e80-1866da0e836f"
Warning FailedMount 29s (x10 over 4m40s) kubelet, rkimmel-dt MountVolume.MountDevice failed for volume "pvc-90d2bb47-a4df-11e9-9e80-1866da0e836f" : driver name csi-hostpath not found in the list of registered CSI drivers
Warning FailedMount 28s (x2 over 2m44s) kubelet, rkimmel-dt Unable to mount volumes for pod "my-csi-app_default(8959f6cd-a4df-11e9-9e80-1866da0e836f)": timeout expired waiting for volumes to attach or mount for pod "default"/"my-csi-app". list of unmounted volumes=[my-csi-volume]. list of unattached volumes=[my-csi-volume default-token-4lrh7]

Output of api-resources

NAME SHORTNAMES APIGROUP NAMESPACED KIND
bindings true Binding
componentstatuses cs false ComponentStatus
configmaps cm true ConfigMap
endpoints ep true Endpoints
events ev true Event
limitranges limits true LimitRange
namespaces ns false Namespace
nodes no false Node
persistentvolumeclaims pvc true PersistentVolumeClaim
persistentvolumes pv false PersistentVolume
pods po true Pod
podtemplates true PodTemplate
replicationcontrollers rc true ReplicationController
resourcequotas quota true ResourceQuota
secrets true Secret
serviceaccounts sa true ServiceAccount
services svc true Service
mutatingwebhookconfigurations admissionregistration.k8s.io false MutatingWebhookConfiguration
validatingwebhookconfigurations admissionregistration.k8s.io false ValidatingWebhookConfiguration
customresourcedefinitions crd,crds apiextensions.k8s.io false CustomResourceDefinition
apiservices apiregistration.k8s.io false APIService
controllerrevisions apps true ControllerRevision
daemonsets ds apps true DaemonSet
deployments deploy apps true Deployment
replicasets rs apps true ReplicaSet
statefulsets sts apps true StatefulSet
tokenreviews authentication.k8s.io false TokenReview
localsubjectaccessreviews authorization.k8s.io true LocalSubjectAccessReview
selfsubjectaccessreviews authorization.k8s.io false SelfSubjectAccessReview
selfsubjectrulesreviews authorization.k8s.io false SelfSubjectRulesReview
subjectaccessreviews authorization.k8s.io false SubjectAccessReview
horizontalpodautoscalers hpa autoscaling true HorizontalPodAutoscaler
cronjobs cj batch true CronJob
jobs batch true Job
certificatesigningrequests csr certificates.k8s.io false CertificateSigningRequest
leases coordination.k8s.io true Lease
events ev events.k8s.io true Event
daemonsets ds extensions true DaemonSet
deployments deploy extensions true Deployment
ingresses ing extensions true Ingress
networkpolicies netpol extensions true NetworkPolicy
podsecuritypolicies psp extensions false PodSecurityPolicy
replicasets rs extensions true ReplicaSet
networkpolicies netpol networking.k8s.io true NetworkPolicy
poddisruptionbudgets pdb policy true PodDisruptionBudget
podsecuritypolicies psp policy false PodSecurityPolicy
clusterrolebindings rbac.authorization.k8s.io false ClusterRoleBinding
clusterroles rbac.authorization.k8s.io false ClusterRole
rolebindings rbac.authorization.k8s.io true RoleBinding
roles rbac.authorization.k8s.io true Role
priorityclasses pc scheduling.k8s.io false PriorityClass
volumesnapshotclasses snapshot.storage.k8s.io false VolumeSnapshotClass
volumesnapshotcontents snapshot.storage.k8s.io false VolumeSnapshotContent
volumesnapshots snapshot.storage.k8s.io true VolumeSnapshot
storageclasses sc storage.k8s.io false StorageClass
volumeattachments storage.k8s.io false VolumeAttachment

[pohly]

So kubelet didn't the driver neither under the old nor the new name, i.e. not at all.

What kind of cluster did you run this on? If kubelet finds a CSI driver, it adds a label for it, so "kubectl describe node/" is a good first check. You probably don't have anything related to csi hostpath there, so next check the kubelet log. Perhaps that shows something.

The log of the driver-registrar container in the csi-hostpathplugin-0 pod may also provide some information.

What kind of cluster do you run this on?

OpenShift with SELinux enabled was problematic before (issue #32).

[eldritch-dream]

Interestingly enough looks like the driver registration is attempted just fine but maybe it never gets a response and that's why i'm not seeing a finish message in this driver-registrar log? I'm using microk8s on fedora 27 you can see my kubectl version output below the logs from the driver-registrar. (Edit) I am running in an SELinux possible environment but I have it disabled so I don't think that's it necessarily.

rkimmel@rkimmel-dt csi-driver-host-path-1.1.0 $ microk8s.kubectl logs csi-hostpathplugin-0 node-driver-registrar I0715 15:39:09.696267 1 main.go:110] Version: v1.1.0-0-g80a94421 I0715 15:39:09.696309 1 main.go:120] Attempting to open a gRPC connection with: "/csi/csi.sock" I0715 15:39:09.696324 1 connection.go:151] Connecting to unix:///csi/csi.sock I0715 15:39:14.956646 1 main.go:127] Calling CSI driver to discover driver name I0715 15:39:14.957122 1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo I0715 15:39:14.957198 1 connection.go:181] GRPC request: {} I0715 15:39:14.959216 1 connection.go:183] GRPC response: {"name":"csi-hostpath","vendor_version":"v1.1.0-0-g670a96b1"} I0715 15:39:14.959769 1 connection.go:184] GRPC error: <nil> I0715 15:39:14.959803 1 main.go:137] CSI driver name: "csi-hostpath" I0715 15:39:14.959854 1 node_register.go:54] Starting Registration Server at: /registration/csi-hostpath-reg.sock I0715 15:39:14.959919 1 node_register.go:61] Registration Server started at: /registration/csi-hostpath-reg.sock

rkimmel@rkimmel-dt csi-driver-host-path-1.1.0 $ microk8s.kubectl version Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6", GitCommit:"abdda3f9fefa29172298a2e42f5102e777a8ec25", GitTreeState:"clean", BuildDate:"2019-05-08T13:53:53Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6", GitCommit:"abdda3f9fefa29172298a2e42f5102e777a8ec25", GitTreeState:"clean", BuildDate:"2019-05-08T13:46:28Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

[pohly]

Looks like kubelet never finds /registration/csi-hostpath-reg.sock. Check your deployment: does it use /var/lib/kubelet/plugins_registry as registration directory
as in

csi-driver-host-path/deploy/kubernetes-1.13/hostpath/csi-hostpath-plugin.yaml

Lines 125 to 128 in 46a3521

	- hostPath:
	path: /var/lib/kubelet/plugins_registry
	type: Directory
	name: registration-dir

?

Does you kubelet use that same path?

[eldritch-dream]

I think you may have hit the nail on the head here my path uses this directory /var/snap/microk8s/common/var/lib/kubelet/ so I'm going to try changing all the yaml in the hostpath dir to use the right path and I'll get back to you! Thanks again for your help looking at this!

(EDIT) This totally worked so thank you so much for helping me get through that!

[pohly]

We should document this better, therefore I am keeping this issue open.

@ivorywang: perhaps you can do a PR for this?

[pohly]

/help
/good-first-issue

[k8s-ci-robot]

@pohly:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/help
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[msau42]

/lifecycle frozen

[nemo83]

Looks like kubelet never finds /registration/csi-hostpath-reg.sock. Check your deployment: does it use /var/lib/kubelet/plugins_registry as registration directory
as in

csi-driver-host-path/deploy/kubernetes-1.13/hostpath/csi-hostpath-plugin.yaml

Lines 125 to 128 in 46a3521

	- hostPath:
	path: /var/lib/kubelet/plugins_registry
	type: Directory
	name: registration-dir

?
Does you kubelet use that same path?

Just here to say thank you! 🙏 I now have CSI driver up and running on my microk8s cluster hosted on raspberry pi4.

[EDIT] Just a quick update to say I've managed to create a handy helm chart for other people to easily start to play around. Repo is https://github.com/speedwing/csi-hostpath-helm

[pohly]

/assign