Document ceph-csi drivers implementation

[mickymiek]

Description of your changes:
Documented how to implement ceph-csi drivers in a k8s 1.12+ cluster with Rook.

Which issue is resolved by this Pull Request:
Resolves #2233

Checklist:

• Documentation has been updated, if necessary.
• Pending release notes updated with breaking and/or notable changes, if necessary.
• Upgrade from previous release is tested and upgrade user guide is updated, if necessary.
• Code generation (make codegen) has been run to update object specifications, if necessary.
• Comments have been added or updated based on the standards set in CONTRIBUTING.md

[skip ci]

[travisn]

Just some minor comments. Can't wait until this is fully integrated with Rook. Lots of steps for now! @rootfs what if we mark this doc as experimental until the full integration is completed?

[travisn]

do we only need these feature gates if it was pre k8s 1.12?

[mickymiek]

Depends on the feature gate. KubeletPluginsWatcher is enabled by default in 1.12, but CSINodeInfo, CSIDriverRegistry and VolumeSnapshotDataSource aren't. Should I be more precise on this step?

[rootfs]

Let's align with 1.12 requirement

[BlaineEXE]

How strict is the v1.12 requirement? Could I theoretically get this working on v1.10 if I find the right feature gates?

[travisn]

It would be great to have someone independently run these steps to verify, since there is no test automation until after the full integration is done. I'll try to do it by tomorrow.

[travisn]

@mickymiek @rootfs When I ran these steps to start the csi driver, I am not able to get it working. As soon as I deploy either csi-cephfsplugin.yaml or csi-rbdplugin.yaml, the docker daemon dies and the cluster completely stops working. Did I miss enabling some feature gate? Since I'm running k8s v1.12.2 I was hoping by default it would work.

docker describe csi-cephfsplugin-g5l96 on the csi plugin shows the following event:

  Warning  Failed     25s   kubelet, minikube  Error: failed to start container "csi-cephfsplugin": 
Error response from daemon: OCI runtime create failed: open /var/run/docker/runtime-runc/moby/csi-cephfsplugin/state.json: no such file or directory: unknown

if I connect to the minikube, I can't use the docker daemon anymore. Even docker images dies with

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

[mickymiek]

@travisn I tested this on kubeadm and haven't encountered this. Maybe this problem is related to minikube? Some issues on kubernetes-csi reports similar behaviors: (#37, #99)

[travisn]

@mickymiek i see the related minikube issue. looks like there is a workaround in one of the issues you referenced.
@rootfs could you review this PR and approve as well?

[rootfs]

thanks @mickymiek for putting together all the instructions! I left some comments, let me know when you are ready.

[mickymiek]

Sorry for the delay! I resolved most of your comments, but still have a question concerning snapshots:
Should I make a PR on ceph-csi to store these yaml? Maybe in examples/rbd/snapshot?

For RBAC details I am not sure, I took them as is from external-snapshotter repo and it worked fine for me. I'll do tests to confirm!

@rootfs

[mickymiek]

@rootfs I modified the doc to match the new csi-snapshotter yamls in ceph-csi repo

[travisn]

The CSI driver pods are failing to start for both CephFS and RBD. Am I missing something in the setup before this? I'm running in minikube and I believe the feature gates are enabled.

I see there are a number of host paths in the yaml files. Perhaps I need to modify those for my environment? I haven't looked closely at which host paths, but hopefully we can cut back on them. Do we really need them all? This is really a question for ceph-csi rather than this doc.

Here is the error in the pod description for the CephFS driver

  Warning  Failed     8m8s  kubelet, minikube  Error: failed to start container "csi-cephfsplugin": Error response from daemon: OCI runtime create failed: open /var/run/docker/runtime-runc/moby/csi-cephfsplugin/state.json: no such file or directory: unknown

Here is the RBD driver pod error:

  Warning  FailedCreatePodSandBox  3m4s (x26 over 8m26s)  kubelet, minikube  Failed create pod sandbox: open /etc/resolv.conf: no such file or directory

[mickymiek]

The CSI driver pods are failing to start for both CephFS and RBD. Am I missing something in the setup before this? I'm running in minikube and I believe the feature gates are enabled.

I'm not sure it is related but could you try running in kubeadm to confirm if minikube is the source of your errors or not? If it is we should do something about it

I see there are a number of host paths in the yaml files. Perhaps I need to modify those for my environment?

You shouldn't have to modify it, as you can see here and here, /var/lib/kubelet/plugins/* are created if not present. I don't think this is related to your issue. As for the hostPath not related to the kubelet plugin dirs, @rootfs surely knows better than me.

[rootfs]

@travisn might be a docker issue, can you restart docker? see moby/moby#30984

[rootfs]

thanks @mickymiek

[travisn]

After the weight is updated, I would just suggest that the commits be squashed and we can go ahead and merge it. I haven't had time to investigate why it was failing for me, but since it's working for multiple other contributors, let's go ahead and merge so we can get broader feedback.

[travisn]

Other topics have been moved around. Could you change this to weight 32? That will put it after the Ceph CRD help topics and before the upgrade guide. Sound ok?

[jbw976]

something we did in the Crossplane project was to make the weights much bigger values, like in the hundreds, so there was more room to move things around without having conflicts. Take for example this doc with a weight of 350.

It might be near time to do that in Rook soon too :)

[travisn]

Yeah that's great if you want to increase the numbers. What I'm really hoping for someday is drag and drop to order the topics. Can you help with that? ;)