lxc-copy of user containers fails.

[linas]

I am unable to make snapshot copies of user-land containers.

lxc-copy -s -n foo-base -N bar -B overlayfs

gives:

lxc-copy: bdev/lxcoverlay.c: ovl_mount: 430 Operation not permitted - Overlayfs: Error mounting /home2/linas/lxc-local-containers/foo-base/rootfs onto /usr/lib/x86_64-linux-gnu/lxc/rootfs with options upperdir=/home/linas/.local/share/lxc/bar/delta0,lowerdir=/home2/linas/lxc-local-containers/foo-base/rootfs: Operation not permitted.
clone failed

Attempts to hack around this by changing the ownership and permissions on /usr/lib/x86_64-linux-gnu/lxc/rootfs does not resolve the problem. For example:

$ ls -la /usr/lib/x86_64-linux-gnu/lxc/rootfs
total 12
drwxrwxrwx 2 linas linas 4096 Dec 21 11:06 .
drwxrwxrwx 4 linas linas 4096 Dec 21 11:06 ..
-rw-r--r-- 1 root  root   189 Nov 24 01:07 README

System: Debian unstable, as of today's date. So

$ lxc-info --version
2.0.6
$ cat /proc/version
Linux version 4.7.0-1-amd64 (debian-kernel@lists.debian.org) (gcc version 5.4.1 20160904 (Debian 5.4.1-2) ) #1 SMP Debian 4.7.6-1 (2016-10-07)
$ /sbin/cgmanager --version
cgmanager 0.41
$ zless /usr/share/doc/libpam-cgm/changelog.Debian.gz
cgmanager (0.41-2) unstable; urgency=medium

[linas]

possibly duplicated in: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=845014

[stgraber]

"Operation not permitted" here suggests that the issue is that overlayfs in your kernel cannot be mounted by unprivileged users. I believe Ubuntu's kernel carries or at least used to carry a patch to enable this.

If your kernel doesn't allow it, there's not much lxc can do about it, though maybe we can show a clearer error message when we get the EPERM from the kernel (rather than just surface it as was done above).

[linas]

Thanks. Hmm. I can confirm that the following does not work: it should have; it does work on ubuntu trusty.

linas $ cd /tmp
linas $ lxc-usernsexec -- /bin/bash
root # mkdir lower upper work overlay
root # mount -t overlay -o lowerdir=lower,upperdir=upper,workdir=work none overlay
mount: permission denied

strace shows its the mount system call:

mount("none", "/tmp/foo/overlay", "overlay", MS_MGC_VAL, "lowerdir=/tmp/foo/lower,upperdir"...) = -1 EPERM (Operation not permitted)

Its a recent kernel:

cat /proc/version
Linux version 4.7.0-1-amd64 (debian-kernel@lists.debian.org) (gcc version 5.4.1 20160904 (Debian 5.4.1-2) ) #1 SMP Debian 4.7.6-1 (2016-10-07)

googling for mount overlayfs as unprivileged user promptly points at CVE's of all sorts, eg.
http://www.openwall.com/lists/oss-security/2015/06/16/2
https://lwn.net/Articles/671641/

It is possible that the reason that the above "obvious" mount fails might be due to some effort to patch these CVE's .. or not. Its hard to tell. There seems to be multiple, similar but different reports, and a variety of patches.

So, yes, it seems its not an lxc issue per-se, but an overlayfs issue. So perhaps this issue should closed.

[brauner]

As @stgraber pointed out mounting overlay filesystems as an unprivileged user requires a patched overlayfs module. To my knowledge, a plain Debian system does not carry this patch whereas Ubuntu does. So I'm closing this as there is nothing we can do about it. :)

[linas]

OK. A status report for user-ns support in the kernel block layer is here: https://www.spinics.net/lists/linux-fsdevel/msg105877.html -- apparently a big patchset was accepted into kernel 4.8 Maybe more work is needed in fuse. I haven't tried the newer kernel.

[PhilippWendler]

For the record: The patch that Ubuntu adds is this one. It is not present in upstream until at least 4.9.

[linas]

For the record: whatever patches are in 4.8, its not enough: I just booted

$ cat /proc/version
Linux version 4.8.0-2-amd64 (debian-kernel@lists.debian.org) (gcc version 5.4.1 20161202 (Debian 5.4.1-4) ) #1 SMP Debian 4.8.15-2 (2017-01-04)

and get the same error message as before.