/dev/disk/by-uuid not present in lxd container

[dhain]

Trying to run cri-containerd in a lxd container, and it refuses to start because /dev/disk/by-uuid is not found:

Nov 06 16:46:21 k8s1 cri-containerd[6439]: I1106 16:46:21.612009    6439 cri_containerd.go:130] Run cri-containerd &{Config:{ContainerdConfig:{RootDir:/var/lib/containerd Snapshotter:overlayfs Endpoint:/run/containerd/containerd.sock Runtime:io.containerd.runtime.v1.linux RuntimeEngine: RuntimeRoot:} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d} SocketPath:/var/run/cri-containerd.sock RootDir:/var/lib/cri-containerd StreamServerAddress: StreamServerPort:10010 CgroupPath: EnableSelinux:false SandboxImage:gcr.io/google_containers/pause:3.0 StatsCollectPeriod:10 SystemdCgroup:false OOMScore:-999} ConfigFilePath:/etc/cri-containerd/config.toml}
Nov 06 16:46:21 k8s1 cri-containerd[6439]: Error: failed to create CRI containerd service: failed to get imagefs uuid of "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": open /dev/disk/by-uuid: no such file or directory

This is a privileged container, and the overlay module is loaded, if that's relevant.

[dhain]

$ cri-containerd version
1.0.0-alpha.1

[mikebrow]

Is your root partition mounted by UUID?

[dhain]

Here's the fstab on the host. It's just a digitalocean droplet:

LABEL=cloudimg-rootfs	/	 ext4	defaults	0 0

[mikebrow]

thx.. could you try to mark your lxd container as unprivileged and see if the lxd defaults are successful in that case?

[Random-Liu]

@dhain Linuxkit had similar problem. /cc @ijc

1. Does your host have /dev/disk/by-uuid?
2. Does LXD mount /dev for privileged container?

[dhain]

@mikebrow I retried using an unprivileged container. Same result.
@Random-Liu host does have it, and here's my mount output in the container (looks like it mounts specific entries, but not the whole enchilada):

lxd/containers/ktest on / type zfs (rw,relatime,xattr,noacl)
none on /dev type tmpfs (rw,nodev,relatime,size=492k,mode=755,uid=100000,gid=100000)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nodev,relatime)
udev on /dev/fuse type devtmpfs (rw,nosuid,relatime,size=8206704k,nr_inodes=2051676,mode=755)
udev on /dev/net/tun type devtmpfs (rw,nosuid,relatime,size=8206704k,nr_inodes=2051676,mode=755)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
tmpfs on /dev/lxd type tmpfs (rw,relatime,size=100k,mode=755)
tmpfs on /dev/.lxd-mounts type tmpfs (rw,relatime,size=100k,mode=711)
lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
udev on /dev/null type devtmpfs (rw,nosuid,relatime,size=8206704k,nr_inodes=2051676,mode=755)
udev on /dev/zero type devtmpfs (rw,nosuid,relatime,size=8206704k,nr_inodes=2051676,mode=755)
udev on /dev/full type devtmpfs (rw,nosuid,relatime,size=8206704k,nr_inodes=2051676,mode=755)
udev on /dev/urandom type devtmpfs (rw,nosuid,relatime,size=8206704k,nr_inodes=2051676,mode=755)
udev on /dev/random type devtmpfs (rw,nosuid,relatime,size=8206704k,nr_inodes=2051676,mode=755)
udev on /dev/tty type devtmpfs (rw,nosuid,relatime,size=8206704k,nr_inodes=2051676,mode=755)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,relatime,gid=100005,mode=620,ptmxmode=666)
devpts on /dev/ptmx type devpts (rw,relatime,gid=100005,mode=620,ptmxmode=666)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,uid=100000,gid=100000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755,uid=100000,gid=100000)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,uid=100000,gid=100000)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755,uid=100000,gid=100000)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)

[Random-Liu]

@dhain I think that is the problem.

Both /var/lib/containerd and /dev/disk are not mounted.

What happens is that:

1. Kubelet wants to know what is the uuid of the disk used by image filesystem.
2. cri-containerd checks which device is /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs mounted on.
3. cri-containerd tries to find the uuid of the device by reading /dev/disk/by-uuid.

So even inside a container, cri-containerd needs to have access to those directories.
/cc @ijc How did we solve this problem in LinuxKit?
@dhain

1. Why do you want to run cri-containerd inside a LXD container instead of on the host?
2. Do you also run containerd inside a container?

[dhain]

@Random-Liu thanks for the analysis. I'm trying to set up a k8s cluster on LXD containers to test deployment config before moving to bare metal. I'm running containerd inside the container as well. I'm treating each container like a separate physical machine, so running the whole stack inside each.

[Random-Liu]

@dhain Hm, I'm not familiar with how LXD works, e.g. whether each container has separate mount namespace.

If each LXD container has a separate mount namespace, you need to let containerd and cri-containerd share the same /var/lib/containerd. Probably mount host path, and set mount propagation properly.

@ijc may know more about how to containerize cri-containerd.

[Random-Liu]

@dhain We added skip-imagefs-uuid flag, and will be included in v1.0.0-beta.1. If the current uuid based approach doesn't work, user could use that flag to skip that step. It should not prevent user from trying out this.

I'll close this issue. Further discussion will be carried on in the kubernetes issue kubernetes/kubernetes#57356.