Failed to create backing store type btrfs

[chiasamen]

The template below is mostly useful for bug reports and support questions.
Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

Required information

• Distribution:

Ubuntu

• Distribution version:

16.04.2 LTS

• The output of
  • lxc-start --version:

2.0.7

• lxc-checkconfig

Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-4.4.0-63-generic
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
Bridges: enabled
Advanced netfilter: enabled
CONFIG_NF_NAT_IPV4: enabled
CONFIG_NF_NAT_IPV6: enabled
CONFIG_IP_NF_TARGET_MASQUERADE: enabled
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled
FUSE (for use with lxcfs): enabled

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

• uname -a

Linux host 4.4.0-63-generic #84-Ubuntu SMP Wed Feb 1 17:20:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• cat /proc/self/cgroups

cat: /proc/self/cgroups: No such file or directory

• cat /proc/1/mounts

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=2003320k,nr_inodes=500830,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=404664k,mode=755 0 0
/dev/sda1 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs rw,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event,release_agent=/run/cgmanager/agents/cgm-release-agent.perf_event 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids,release_agent=/run/cgmanager/agents/cgm-release-agent.pids 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset,clone_children 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=24,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/sdb1 /var/lxc btrfs rw,relatime,space_cache,user_subvol_rm_allowed,subvolid=5,subvol=/ 0 0
/dev/sdc1 /var/lxc-s btrfs rw,relatime,space_cache,user_subvol_rm_allowed,subvolid=5,subvol=/ 0 0
cgmfs /run/cgmanager/fs tmpfs rw,relatime,size=100k,mode=755 0 0
lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=404664k,mode=700,uid=1000,gid=1000 0 0
tmpfs /run/user/1012 tmpfs rw,nosuid,nodev,relatime,size=404664k,mode=700,uid=1012,gid=1012 0 0
tmpfs /run/user/1001 tmpfs rw,nosuid,nodev,relatime,size=404664k,mode=700,uid=1001,gid=1001 0 0

Issue description

Steps to reproduce

from root

su

to unpriv. user

su lxcbob
lxc-create -f ~/.config/lxc/default.conf -B btrfs -n test -t none --logfile test.log --logpriority debug
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
lxc-create: lxccontainer.c: do_bdev_create: 1042 Failed to create backing store type btrfs
lxc-create: lxccontainer.c: do_lxcapi_create: 1518 Error creating backing store type btrfs for test
lxc-create: tools/lxc_create.c: main: 318 Error creating container test
cat test.log
lxc-create 20170423184033.100 INFO lxc_confile - confile.c:config_idmap:1531 - read uid map: type u nsid 0 hostid 951968 range 65536
lxc-create 20170423184033.100 INFO lxc_confile - confile.c:config_idmap:1531 - read uid map: type g nsid 0 hostid 986434 range 65536
lxc-create 20170423184033.100 INFO lxc_caps - caps.c:lxc_caps_up:102 - Last supported cap was 36
lxc-create 20170423184033.115 INFO lxcbtrfs - bdev/lxcbtrfs.c:btrfs_subvolume_create:251 - btrfs: snapshot create ioctl returned -1
lxc-create 20170423184033.115 ERROR lxc_container - lxccontainer.c:do_bdev_create:1042 - Failed to create backing store type btrfs
lxc-create 20170423184033.115 ERROR lxc_container - lxccontainer.c:do_lxcapi_create:1518 - Error creating backing store type btrfs for test
lxc-create 20170423184033.116 ERROR lxc_create_ui - tools/lxc_create.c:main:318 - Error creating container test

cat .config/lxc/default.conf
lxc.include = /etc/lxc/default.conf
lxc.id_map=u 0 951968 65536
lxc.id_map=g 0 986434 65536
lxc.network.type=veth
lxc.network.link=lxcbr1
lxc.logfile = /var/lxc-s/lxc.log
lxc.network.ipv4 = 0.0.0.0
lxc.rootfs = /var/lxc-s/test/rootfs
lxc.mount = /var/lxc-s/.config/lxc/test.fstab
lxc.loglevel = TRACE
lxc.start.auto = 1
lxc.start.delay = 5

cat /var/lxc-s/.config/lxc/test.fstab
none /var/lxc-s/rootfs/dev/pts devpts defaults 0 0
none /var/lxc-s/rootfs/proc proc defaults 0 0
none /var/lxc-s/rootfs/sys sysfs defaults 0 0

[brauner]

Can you create a subvolume using the btrfs tools directly?

btrfs subvolume create /var/lxc-s/test-subvol

[chiasamen]

Yes indeed I can (which suprises me)
btrfs subvolume create /var/lxc-s/test-subvol
Create subvolume '/var/lxc-s/test-subvol'

[robincee]

When i try this without a config file and with the -t none i do not get the error (but nothing is created besides lxc config obviously).

 lxc-create 20170428031558.884 INFO     lxc_create_ui - tools/lxc_create.c:main:324 - container  created

Once i set the type to (for example) ubuntu/debian/etc it fails with the following message:

lxc-create -B btrfs -n test -t ubuntu --logfile test.log --logpriority debug

lxc-create: lxccontainer.c: do_bdev_create: 1042 Failed to create backing store type btrfs
lxc-create: lxccontainer.c: do_lxcapi_create: 1518 Error creating backing store type btrfs for test3
lxc-create: tools/lxc_create.c: main: 318 Error creating container test3

Same story for manually creating a volume, that works fine. (Testing as root / Ubuntu 14 + lxc from backports)

[chiasamen]

With another (unprivileged) user I am able to use lxc-create correctly. Do I have to tell lxc something in order to be able to create a container as "user2"?

[brauner]

Could you please tell me the usernames and the contents of your /etc/subuid, /etc/subgid files?

[robincee]

@chiasamen
Yes.
https://stgraber.org/2014/01/17/lxc-1-0-unprivileged-containers/
https://www.cyberciti.biz/faq/how-to-create-unprivileged-linux-containers-on-ubuntu-linux/
etc etc, lots of guides exist for unprivileged containers.

Use something like this to get the sub uid/gid info:

sudo grep container_username /etc/sub{gid,uid}
Where container_username is the account you want to run the container.

@brauner
For my case: creating containers without btrfs works fine, the uid mapping works and strace doesnt make me any smarter for now when using btrfs. (might missed a step when setting this up, just need to find it) TBC

[chiasamen]

I don't know why, but it worked when I tried it again. I can only imagine that it's related to a reboot since thats the only relevant action which could have an effect to the user.

@robincee : That was exactly my problem - the basics where clear to me (since I already have several unpriv. containers running (with other users))

[robincee]

@chiasamen thanks!

I can't confirm the reboot was required, also was unable to remove the btrfs module and reload it without a reboot, but i don't think that was my problem.

I followed a more general guide when setting btrfs up, while lxc expects the volume in /var/lib/lxc (by default). Mine was located in /mnt/folder. Whoopse. Once i changed lxc.lxcpath = /var/lib/lxc/ in /etc/lxc/lxc.conf to /mnt/folder things started working as expected.

My 2 cents: A simple check in the code if this folder isn't a btrfs volume with a neat comment would do better then the error you get now. And an older version of the lxc-create command listed way more info for the -B option, including a bit about /var/lib/lxc/ that's missing from the codebase atm.

Long story short: when using btrfs be sure to set lxc.lxcpath to a mounted btrfs volume OR /var/lib/lxc should be a btrfs mount already. I confirmed both work just fine. Thanks!

[brauner]

A bunch of problems around btrfs and other storage drivers are fixed in newer liblxc releases.