Creating a snapshot of a clone with unprivileged containers fails when using overlayfs.

[thsnr]

Suppose I have an unprivileged container named base. I create a clone of it named clone with the following command:

lxc-clone -s -B overlayfs base clone

Next, I delete the file /tmp/something from clone and attempt to create a snapshot of it with:

lxc-snapshot -n clone

This results in the following error:

rsync: mknod "/home/tiit/.local/share/lxcsnaps/clone/snap0/delta0/tmp/something" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]
lxc_container: bdev.c: rsync_delta: 2295 rsyncing /home/tiit/.local/share/lxc/clone/delta0 to /home/tiit/.local/share/lxcsnaps/clone/snap0/delta0
lxc_container: bdev.c: overlayfs_clonepaths: 2450 copying overlayfs delta
lxc_container: bdev.c: bdev_copy: 3115 failed getting pathnames for cloned storage: overlayfs:/home/tiit/.local/share/lxc/base/rootfs:/home/tiit/.local/share/lxc/clone/delta0
lxc_container: lxccontainer.c: copy_storage: 2489 Error copying storage
lxc_container: lxccontainer.c: lxcapi_snapshot: 2923 clone of /home/tiit/.local/share/lxc:clone failed
lxc_container: lxc_snapshot.c: do_snapshot: 55 Error creating a snapshot

I'm guessing that the cause of this is that overlayfs creates a whiteout character device /tmp/something with the device number 0/0 in the upper layer and when snapshotting, rsync attempts to copy this to the snapshot directory. This fails because we don't have permissions to create special files (as we are unprivileged).

The only workaround I can think of is skipping the whiteouts when rsyncing, followed by deleting the files again in the snapshot, but that does not feel like a very clean solution.

[hallyn]

Thanks, I can confirm this.

'ab' is created as a character device. In a userns we cannot create those. (the original whiteout was created by the kernel with special privilege).

This is a problem... I'll raise it on #ubuntu-kernel

[brauner]

This would be a patch specific to the Ubuntu kernel or upstream kernel?

[hallyn]

It's specific to the ubuntu kernel because afaik upstream kernel doesn't allow unprivileged users to mount overlayfs.

I'll try to find time to post the fix today. It'll simply be to do the rsync on the mounted overlays rather than separately rsyncing the delta0.

[brauner]

Ah, unfortunate. The upstream kernel will probably not want to support this.