Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rook CSI RBD Plugin can't start on host nodes that lack the findmnt binary #3923

Closed
rajha-korithrien opened this issue Sep 19, 2019 · 19 comments · Fixed by #3931
Closed

Rook CSI RBD Plugin can't start on host nodes that lack the findmnt binary #3923

rajha-korithrien opened this issue Sep 19, 2019 · 19 comments · Fixed by #3931
Labels

Comments

@rajha-korithrien
Copy link

@rajha-korithrien rajha-korithrien commented Sep 19, 2019

Is this a bug report or feature request?

  • Bug Report
    There is only one other issue that mentions findmnt #3726 but I don't think it is related.

Deviation from expected behavior:
csi-rdbplugin fails to start on any node in the cluster

Expected behavior:
csi-rbdplugin should start on all nodes in the cluster

How to reproduce it (minimal and precise):
Deploy Kubernetes via Rancher RKE onto some number of RancherOS (bare metal) nodes.
Some things to note:

  • This will deploy kubernetes via Docker using hyperkube
  • RancherOS (by default) doesn't supply a findmnt binary or a stat binary
  • Using Rook via the Flex Volume system works fine on Rancher

Ensure you pass "extra_binds" arguments so kubelet gains access to host directories (much as is done when using the Flex volume system).

extra_binds:
      - "/usr/libexec/kubernetes/kubelet-plugins/volume/exec:/usr/libexec/kubernetes/kubelet-plugins/volume/exec"
      - "/var/lib/kubelet/plugins_registry:/var/lib/kubelet/plugins_registry"
      - "/var/lib/kubelet/pods:/var/lib/kubelet/pods:shared,z"

I had a previously working Rook 1.0.6/Ceph 14.2.x installation and followed these upgrade instructions to move from 1.0.6 to 1.1.0. I had not used the Ceph CSI driver before now.

https://rook.io/docs/rook/v1.1/ceph-upgrade.html

At the step where the Rook operator is updated (I had to manually fix some minor RBAC issues see #3868 ) the result is csi-rdbplugin fails to deploy on each cluster host node. The related driver-registrar and liveness-prometheus deploy correctly.

File(s) to submit:

  • Crashing pod(s) logs, if necessary
    This is the log from csi-rdbplugin
I0919 19:27:26.963369 19679 cephcsi.go:103] Driver version: v1.2.0 and Git version: c420ee6de9e2f90fcce97b2700c0fd57845225c3
E0919 19:27:26.965139 19679 cephcsi.go:128] Failed to get the PID limit, can not reconfigure: open /sys/fs/cgroup/pids/docker/e7caa2de7b1e1640df0b2382774a3bd653343364391fdfc99aa572dfaa6160f4/kubepods/besteffort/pod48425a55-daef-11e9-9ffd-e0db5570db32/8a9f453be111b13efed122df4bb56528d94f6c4a0662d56cd222ced80f4b3b39/pids.max: no such file or directory
I0919 19:27:26.965342 19679 cephcsi.go:158] Starting driver type: rbd with name: paas-rook-ceph.rbd.csi.ceph.com
I0919 19:27:26.976889 19679 mount_linux.go:170] Cannot run systemd-run, assuming non-systemd OS
I0919 19:27:26.976926 19679 mount_linux.go:171] systemd-run failed with: exit status 1
I0919 19:27:26.976953 19679 mount_linux.go:172] systemd-run output: Failed to create bus connection: No such file or directory
F0919 19:27:26.977175   19679 driver.go:145] failed to start node server, err unable to find findmnt

Discussion
I have tracked this down to the use of quay.io/cephcsi/cephcsi:v1.2.0 which eventually calls into Kubernetes code (in nsenter) that checks for the tools it needs:

binaries := []string{
		"mount",
		"findmnt",
		"umount",
		"systemd-run",
		"stat",
		"touch",
		"mkdir",
		"sh",
		"chmod",
		"realpath",
}
// search for the required commands in other locations besides /usr/bin
for _, binary := range binaries {
	// check for binary under the following directories
	for _, path := range []string{"/", "/bin", "/usr/sbin", "/usr/bin"} {
		binPath := filepath.Join(path, binary)
		if _, err := os.Stat(filepath.Join(ne.hostRootFsPath, binPath)); err != nil {
			continue
		}
		ne.paths[binary] = binPath
		break
	}
	// systemd-run is optional, bailout if we don't find any of the other binaries
	if ne.paths[binary] == "" && binary != "systemd-run" {
		return fmt.Errorf("unable to find %v", binary)
	}
}

Which I am fairly sure gets its "hostRootFsPath" from the yaml Rook used to deploy csi-rbdplugin

env:
- name: HOST_ROOTFS
   value: /rootfs
...
volumeMounts:
- mountPath: /rootfs
   name: host-rootfs
...
volumes:
- hostPath:
      path: /
      type: ""
    name: host-rootfs

The binaries needed exist in the quay.io/cephcsi/cephcsi:v1.2.0 image. Is there a way to configure the CephCSI driver to look for the binaries in the container rather than in a filesystem mounted from the host?

Environment:

  • OS (e.g. from /etc/os-release):
NAME="RancherOS"
VERSION=v1.5.1
ID=rancheros
ID_LIKE=
VERSION_ID=v1.5.1
PRETTY_NAME="RancherOS v1.5.1"
HOME_URL="http://rancher.com/rancher-os/"
SUPPORT_URL="https://forums.rancher.com/c/rancher-os"
BUG_REPORT_URL="https://github.com/rancher/os/issues"
BUILD_ID=
  • Kernel (e.g. uname -a):
    Linux host-0 4.14.85-rancher #1 SMP Sat Dec 1 12:40:08 UTC 2018 x86_64 GNU/Linux
  • Cloud provider or hardware configuration:
    Private bare metal
  • Rook version (use rook version inside of a Rook Pod):
    rook: v1.1.0
  • Storage backend version (e.g. for ceph do ceph -v):
    ceph version 14.2.3 (0f776cf838a1ae3130b2b73dc26be9c95c6ccc39) nautilus (stable)
  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.9", GitCommit:"3e4f6a92de5f259ef313ad876bb008897f6a98f0", GitTreeState:"clean", BuildDate:"2019-08-05T09:22:00Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:16Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
    Using Rancher 2.2.8
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):
[root@host-0 /]# ceph health
HEALTH_OK
[root@host-0 /]#
@martinlindner

This comment has been minimized.

Copy link

@martinlindner martinlindner commented Sep 20, 2019

I'm seeing exactly the same behavior with a fresh installation (via the helm chart) of rook-ceph v1.1.0 on a Rancher 2.2.8-provisioned lab cluster.

Is there any workaround to get this running on RancherOS (except for going non-CSI with v1.0.x)?

  • OS:
NAME="RancherOS"
VERSION=v1.5.4
ID=rancheros
ID_LIKE=
VERSION_ID=v1.5.4
PRETTY_NAME="RancherOS v1.5.4"
HOME_URL="http://rancher.com/rancher-os/"
SUPPORT_URL="https://forums.rancher.com/c/rancher-os"
BUG_REPORT_URL="https://github.com/rancher/os/issues"
BUILD_ID=
  • Kernel:
Linux lab-worker1 4.14.138-rancher #1 SMP Sat Aug 10 11:25:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Rook version:
rook: v1.1.0
  • Storage backend version:
ceph version 14.2.3 (0f776cf838a1ae3130b2b73dc26be9c95c6ccc39) nautilus (stable)
  • Kubernetes version:
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T12:36:28Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:16Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
  • Storage backend status:
HEALTH_OK
@Madhu-1

This comment has been minimized.

Copy link
Contributor

@Madhu-1 Madhu-1 commented Sep 20, 2019

@martinlindner @ajha-korithrien I have never tried Rancher is there I way to deploy it using vagrant so that I can reproduce locally and to fix and test it

there is a flag in csi to disable use of nsenter, can you try it out https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/csi/template/rbd/csi-rbdplugin.yaml#L57
edit the daemon set and set the above value to false (this is just a temp fix to try out) let me know if it works will make it as configurable

@rajha-korithrien

This comment has been minimized.

Copy link
Author

@rajha-korithrien rajha-korithrien commented Sep 20, 2019

@Madhu-1 I have never used vargrant to setup Rancher, but the project has instructions for how to do it.

https://rancher.com/docs/rancher/v2.x/en/quick-start-guide/deployment/quickstart-vagrant/

However I doubt it is Rancher the Kubernetes as a Service project that is the issue, I believe it is RancherOS the tiny Linux distro. There is an out of support Github project for Vargrant and RancherOS

https://github.com/rancher/os-vagrant

Which says they officially support docker-machine with a number of different hypervisors.

https://rancher.com/docs/os/v1.x/en/installation/running-rancheros/workstation/docker-machine/

@Madhu-1

This comment has been minimized.

Copy link
Contributor

@Madhu-1 Madhu-1 commented Sep 20, 2019

#3931 removes the use of nsenter which checks for binaries in rbd

@rajha-korithrien

This comment has been minimized.

Copy link
Author

@rajha-korithrien rajha-korithrien commented Sep 20, 2019

@Madhu-1 @martinlindner I can confirm that setting --containerized=false from the current default of --containerized=true does cause csi-rbdplugin to deploy properly. I will test that it correctly allocates volumes are report back.

@BlaineEXE BlaineEXE added the ceph - label Sep 20, 2019
@rajha-korithrien

This comment has been minimized.

Copy link
Author

@rajha-korithrien rajha-korithrien commented Sep 20, 2019

@Madhu-1 I doubt this is related but while I have running csi-rdbplugin pods, rook-ceph-osd-prepare-* pods fail with the following error in their logs:

2019-09-20 16:45:59.209876 I | exec: Running command: stdbuf -oL ceph-volume lvm batch --prepare --bluestore --yes --osds-per-device 1 --block-db-size 5242880000 /dev/zd0 --db-devices /dev/zd16 --report
2019-09-20 16:46:16.393841 I | --> AttributeError: 'MixedType' object has no attribute 'total_available_db_space'
failed to configure devices. failed to configure devices with ceph-volume. failed to initialize devices. failed ceph-volume report. Failed to complete '': exit status 1.

At first glance this seems related to #2659 but it is not as I had a perfectly working 1.0.6 deployment before the upgrade. If you feel this is unrelated shall I close this ticket and open a new one?

Update

This specific issue of AttributeError: 'MixedType' object has no attribute 'total_available_db_space' was caused by Linux changing which block device was pointed to by which /dev entry. My data and metadata devices switched /dev entries (/dev/zd0 became /dev/zd16, and /dev/zd16 became /dev/zd0). Once I straightened out the device order, the error went away and the cluster finished its update.

@rajha-korithrien

This comment has been minimized.

Copy link
Author

@rajha-korithrien rajha-korithrien commented Sep 22, 2019

@Madhu-1 I can confirm that after fixing the "AttributeError: 'MixedType' issue, your fix for --containerized=false works correctly.

PVC are now correctly bound however they do not make it into the pod. With the error:

driver name rook-ceph.rbd.csi.ceph.com not found in the list of registered CSI drivers

Is this issue related?

@Madhu-1

This comment has been minimized.

Copy link
Contributor

@Madhu-1 Madhu-1 commented Sep 23, 2019

@rajha-korithrien Thanks for testing this one. once #3931 is merged this will be fixed

@Madhu-1

This comment has been minimized.

Copy link
Contributor

@Madhu-1 Madhu-1 commented Sep 23, 2019

@Madhu-1 I can confirm that after fixing the "AttributeError: 'MixedType' issue, your fix for --containerized=false works correctly.

PVC are now correctly bound however they do not make it into the pod. With the error:

driver name rook-ceph.rbd.csi.ceph.com not found in the list of registered CSI drivers

Is this issue related?

seeing this for the first time. can you check the driver-register log in rbd plugin pod?

@martinlindner

This comment has been minimized.

Copy link

@martinlindner martinlindner commented Sep 23, 2019

@Madhu-1 @rajha-korithrien
I finally had some time to play with this, too, and I can confirm that --containerized=false fixes the initial deployment issue with csi-rdbplugin.

Furthermore I'm also seeing the driver .. not found issue, for both cephfs and rbd.

Apparently this is fairly specific to RKE clusters:
RKE runs kubelet in a container with --root-dir=/opt/rke/var/lib/kubelet and matching bind mount to the same dir on the host. Since rook's csi-*plugin have /var/lib/kubelet/ paths hardcoded, the CSI drivers and sockets end up in a location where the kubelet can't see them. As a quick test, i patched all paths in the csi-*plugin daemonsets and things started working for me ..

For reference, it looks like the Rancher devs ran into a similar issue with their own Longhorn storage engine a while ago:
longhorn/longhorn#198

Apparently their fix was to auto-detect the kubelet's root-dir during deployment:
longhorn/longhorn-manager#242

Do you think something similar would be possible for Rook's CSI driver deployers?

@Madhu-1

This comment has been minimized.

Copy link
Contributor

@Madhu-1 Madhu-1 commented Sep 23, 2019

@martinlindner we made kubelet path configurable during deployment in rook see #3927

@martinlindner

This comment has been minimized.

Copy link

@martinlindner martinlindner commented Sep 23, 2019

@Madhu-1:
Great. Sorry, I did't notice you had already added that.

I just did a a fresh deployment from master, with the following added to operator.yaml:

- name: FLEXVOLUME_DIR_PATH
  value: "/var/lib/kubelet/volumeplugins"

- name: ROOK_CSI_KUBELET_DIR_PATH
  value: "/opt/rke/var/lib/kubelet"

After putting --containerized=false into the csi-rbdplugin daemonset, things seem to work fine.

I ran a quick test with rbd.csi and cephfs.csi storage classes and the PVs did provision and attach to pods as expected.

So, here's hoping for your fix for the original nsenter issue to make it into master soon.

@jeffreyvdhondel

This comment has been minimized.

Copy link

@jeffreyvdhondel jeffreyvdhondel commented Sep 23, 2019

Having the same error when deploying to a digitalocean cluster 1.0 works fine.
Will wait for the 1.x fix for this 'problem'.

@Madhu-1

This comment has been minimized.

Copy link
Contributor

@Madhu-1 Madhu-1 commented Sep 23, 2019

@martinlindner Thanks for testing it out, removal of nsenter is still under discussion in ceph-csi. will update PR once it's fixed.

@rajha-korithrien

This comment has been minimized.

Copy link
Author

@rajha-korithrien rajha-korithrien commented Sep 23, 2019

@martinlindner Thanks for tackling where RKE puts stuff. I was still trying to use the extra_binds configuration with /var/lib/kubelet and had not even realized that RKE deployed kublet to /opt/rke/var/lib/kubelet.

@Madhu-1 I can confirm that with the changes you provided for --containerized=false on csi-rbdplugin and setting ROOK_CSI_KUBELET_DIR_PATH as provided by martinlinder, everything works as expected.

@Madhu-1

This comment has been minimized.

Copy link
Contributor

@Madhu-1 Madhu-1 commented Sep 24, 2019

@rajha-korithrien @martinlindner @jeffreyvdhondel this is now fixed in the master is it possible to give a try?

@martinlindner

This comment has been minimized.

Copy link

@martinlindner martinlindner commented Sep 25, 2019

@Madhu-1 Sure, I wanted to try out the helm chart anyways.

I can confirm that this works great with Rancher v2.2.8 (RKE v0.2.8) + RancherOS v1.5.4:

helm install \
  --namespace rook-ceph rook-master/rook-ceph \
  --version v1.1.0-beta.0.175.g9bcdbea \
  --set agent.flexVolumeDirPath=/var/lib/kubelet/volumeplugins \
  --set csi.kubeletDirPath=/opt/rke/var/lib/kubelet

Thanks a lot for getting this fixed so quickly.

@FinnGu

This comment has been minimized.

Copy link

@FinnGu FinnGu commented Oct 15, 2019

Having the same error when deploying to a digitalocean cluster 1.0 works fine.
Will wait for the 1.x fix for this 'problem'.

Did you get rook to work with digitalocean eventually?

@midacts

This comment has been minimized.

Copy link

@midacts midacts commented Oct 20, 2019

I am running Rancher k3s and I ran into this issue as well.
Updating the operator.yaml file with the below settings fixed the issue for me.

#4133 Thanks Madhu-1

- name: ROOK_CSI_KUBELET_DIR_PATH
  value: "/var/lib/rancher/k3s/agent/kubelet"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.