k8s 1.8 flexvolume driver is not recognized

[binocarlos]

When running a k8s v1.8.7-gke.1 cluster on GKE - with all of the dotmesh pods running:

$ kubectl get po -n dotmesh
NAME                                           READY     STATUS    RESTARTS   AGE
dotmesh-5z7t7                                  1/1       Running   0          4h
dotmesh-dynamic-provisioner-5b8548c84b-hw2f8   1/1       Running   0          1h
dotmesh-etcd-cluster-0000                      1/1       Running   0          4h
dotmesh-etcd-cluster-0001                      1/1       Running   0          4h
dotmesh-etcd-cluster-0002                      1/1       Running   0          4h
etcd-operator-56b49b7ffd-n42tf                 1/1       Running   0          4h

And the flexvolume binary present in /usr/libexec/kubernetes/kubelet-plugins/volume/exec/dotmesh.io~dm

and having done sudo systemctl restart kubelet on the host lead to this error in the kubelet:

Feb 06 17:52:18 gke-dotmesh-gke-cluster-default-pool-bfb89fc5-d1bn kubelet[20547]: E0206 17:52:18.236820   20547 desired_state_of_world_populator.go:288] Failed to add volume "test-storage" (specName: "pvc-db880397-0b65-11e8-9e61-42010a9a0fd9") for pod "dbca7409-0b65-11e8-9e61-42010a9a0fd9" to desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pvc-db880397-0b65-11e8-9e61-42010a9a0fd9" err=no volume plugin matched

Reading this part of the release notes - it seems that the plugin discovery mechanism has changed. Update the flexvolume driver to work with k8s >= 1.8

[lukemarsden]

I don't think our flex plugin installer is broken, we already do the atomic rename from a file starting with . trick in http://gitlab.dotmesh.io:9999/dotmesh/dotmesh/blob/master/cmd/dotmesh-server/pkg/main/kubernetes.go#L25

[lukemarsden]

so there must be some other reason why you got that error...

[lukemarsden]

when this is fixed and we start working with Kubernetes 1.8, hopefully we also start working with 1.9 (which is what ships with latest Docker for Mac)

[alaric-dotmesh]

We do almost exactly what the docs @binocarlos refers to says:

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/flexvolume-deployment.md#recommended-driver-deployment-method

I shall attempt to work out what's wrong.

[alaric-dotmesh]

Ok, I've tried this on k8s 1.9 and everything works flawlessly (with no kubelet restart!) in the "hello dotmesh on k8s" tutorial.

On k8s 1.8 (1.8.8 to be precise) I'm getting a problem, but it's not with the flexvolume driver being picked up. The driver is being invoked correctly (with no kubelet restart), but logging this into /var/log/dotmesh-flexvolume.log:

22239: 2018/02/21 10:34:19.840566 RPC FAIL: {Namespace:admin Name:moby_counter2@newbranch Subdot:__default__} -> DotmeshRPC.Procure -> A volume called admin/moby_counter2 already exists with id b50d97a7-78ff-4fae-7ed1-8305e42d452a
22239: 2018/02/21 10:34:19.840789 MOUNT: Procure of admin/moby_counter2@newbranch.__default__ failed: Couldn't decode response '{"jsonrpc":"2.0","error":{"code":-32000,"message":"A volume called admin/moby_counter2 already exists with id b50d97a7-78ff-4fae-7ed1-8305e42d452a","data":null},"id":5577006791947779410}
': A volume called admin/moby_counter2 already exists with id b50d97a7-78ff-4fae-7ed1-8305e42d452a

As you might guess from it being called moby_counter2 in there, it did this originally with moby_counter and I tried again with a new name in case it was just broken state left from an earlier experiment, but it instantly starts failing.

So: FV driver is working fine, but DM is giving errors back from the Procure call on k8s 1.8. Investigating...

[alaric-dotmesh]

False alarm, that was because I'd left my redis-pvc.yaml pinned to a branch, and was getting:

Error procuring filesystem: Cannot use branch-pinning syntax (docker run -v volume@branch:/path) to create a non-existent volume with a non-master branch, pausing and trying again...

...but that was in docker logs dotmesh-server-inner, it wasn't getting passed back up to the FV driver!

#300 raised to improve flexvolume error reporting.

[alaric-dotmesh]

After a bit of fiddling with stuff purely local to my setup, it works fine for kubernetes 1.8 with the stock dotmesh-k8s-1.8.yaml YAML!

So: This may be a GKE-specific problem. Now investigating a GKE cluster...

[alaric-dotmesh]

Well, one problem is that GKE uses a different place for FV plugins:

https://github.com/rancher/longhorn#troubleshooting

• they go in /home/kubernetes/flexvolume, but I'm struggling to get it working nonetheless.

[alaric-dotmesh]

Ok, it needs the socket moved as well, and I can't hack that with mv as it's on a different filesystem, so I need to rebuild the server to install it differently.

[alaric-dotmesh]

Ok, fixed it! We now have a YAML parameter to choose where the FV plugins go, and separate .gke.yaml files packaged that have it set for GKE.

$ kubectl apply -f https://get.dotmesh.io/unstable/263-gke-flexvolume/yaml/dotmesh-k8s-1.8.gke.yaml
serviceaccount "dotmesh" created
clusterrole "dotmesh" created
clusterrolebinding "dotmesh" created
service "dotmesh" created
daemonset "dotmesh" created
serviceaccount "dotmesh-provisioner" created
clusterrole "dotmesh-provisioner-runner" created
clusterrolebinding "dotmesh-provisioner" created
deployment "dotmesh-dynamic-provisioner" created
storageclass "dotmesh" created
$ kubectl get po -n dotmesh
NAME                                          READY     STATUS    RESTARTS   AGE
dotmesh-7pl9w                                 1/1       Running   1          1m
dotmesh-dynamic-provisioner-f796dd6df-88m7k   1/1       Running   0          1m
dotmesh-g9j22                                 1/1       Running   1          1m
dotmesh-lx687                                 1/1       Running   1          1m
etcd-operator-56b49b7ffd-rltzp                1/1       Running   0          1d
$ kubectl apply -f . 
deployment "redis" created
persistentvolumeclaim "redis-pvc" created
service "redis" created
deployment "web" created
service "web" created
$ kubectl get po
NAME                     READY     STATUS    RESTARTS   AGE
redis-7d85fc658d-krl8b   1/1       Running   0          15s
web-57c7d94cb9-b8jf6     1/1       Running   0          15s

Tada!