Btrfs quota correctly enabled but disk size into the container isn't changed

[BenoitGEORGELIN]

Required information

• Distribution: Ubuntu 16.04

• Distribution version: 16.04

• The output of "lxc info" or if that fails:

  driver: lxc
  driverversion: 2.0.1
  kernel: Linux
  kernelarchitecture: x86_64
  kernelversion: 4.5.7-040507-generic
  server: lxd
  serverpid: 1166
  serverversion: 2.0.2
  storage: btrfs
  storageversion: "4.4"

Issue description

I would like to set a quota on the subvolume used by the container and relfect that size into the container.

When I use this command :
lxc config device set my-container root size 10GB

The quota is configured but the disk size into the container remain the one of the host.

Steps to reproduce, exemple with 10GB

+init a new container :
lxc init images:ubuntu/xenial/amd64 bendisk -p default

+set the device root size :
Disk size inside container
lxc config device set bendisk root size 10GB

+start the container:
Start container
lxc start bendisk

Check inside container :
lxc exec bendisk -- df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 100G 9.1G 88G 10% /
none 492K 0 492K 0% /dev
/dev/sdb 100G 9.1G 88G 10% /dev/lxd
cgroup_root 10M 0 10M 0% /sys/fs/cgroup
udev 3.9G 0 3.9G 0% /dev/tty
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 8.1M 3.9G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock

Check quota:
qgroupid rfer excl max_rfer max_excl parent child
0/493 369.91MiB 12.08MiB 10.00GiB 10.00GiB --- ---

[stgraber]

Yeah, that's unfortunately the way disk space reporting works with btrfs...

Unlike zfs, it doesn't show a separate mountpoint per subvolume so the reported disk space is that of the block device, not of the quota.

It "may' be possible to do something about this in github.com/lxc/lxcfs though that'd come with a very big overhead I'd expect.

Other than that, the right way to fix this would be for btrfs to behave more like zfs at the kernel level, but I doubt that's going to happen (would require significant design changes).

Anyway, I'm closing this issue as there's nothing LXD can do about this in either case. We do properly set the quota, it's just that nothing actually reports this information until it's too late (out of disk space error).

[BenoitGEORGELIN]

Thanks

[tim-seoss]

Since df uses statfs(), and this is called on a path, I've suggested on the btrfs mailing list that it might be possible to return quota usage and limits in statfs() - and there does seem to be some interest in this:

https://marc.info/?l=linux-btrfs&m=147585928207456&w=2

XFS (see previous messages in the thread above) already does this with directory ("project") quotas. If you'd like to check that - it should be possible by using the lxd directory storage backend on an XFS filesystem - and I suppose it'd be possible for lxd to manage such quotas on XFS filesystems too?

http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en-US/html/ch08s02.html

[stgraber]

Sounds good. And yeah, that means that in theory we could do XFS quotas when using the directory backend. We'd certainly be happy to review a pull request from someone who knows how that works :)

[ThomasLeister]

Thanks @BenoitGEORGELIN for reporting this issue. Just did some experimenting for a major poject and faced the same issue. Unfortunately for me it is absolutely mandatory to retrieve the real disk usage and limits in a container (given a quota on that container), therefore this problem is now really a show stopper.

I've tried ZFS as an alternative, but there are other problems with it, so I can't use it as well (MariaDB can't be initialized with Alpine container and ZFS storage backend: "fallocate()" call is not supported on mysql_initialize_db)

Where is the correct place to go and encourage developers to improve the situation? As far as I understand, it's BTRFS missing a feature, right? What can I / we do about it (as a non- btrfs developer)? This issue might gather some more importance soon, as containerized environments get more popular quickly. 😟

[brauner]

You can check with David Sterba on the btrfs kernel mailing list whether there has been any movement on this or if it's on their roadmap. He was the one who was interested in implementing it. He's at SUSE and since they support btrfs and try to have a container story it would make sense for them to implement this.

[brauner]

You can Cc lxc-devel@lists.linuxcontainers.org on this.

[brauner]

But please, do keep in mind that they all have a lot to do and SUSE is one of the few companies that actually still actively develop btrfs upstream. So be kind. :)

[ThomasLeister]

Thanks a lot, @brauner ! I've sent an email on both mailing lists regarding the issue.

[tim-seoss]

FWIW, ext4 also has project quota support https://lwn.net/Articles/623835/, so you could use xfs or ext4 with that. They will both display the lowest of the project (i.e. container) quota remaining or the actual filesystem free space in statfs() (df etc.). In the case of ext4 you have to enable the 'project' feature with tune2fs and mkfs.ext4

AFAIK, there's no lxd support at the moment, but it should be reasonably straightforward to add, and you could just implement it outside of lxd in the meantime or to trial it.

Obviously if you need some other feature which xfs/ext4 doesn't yet have, then that's a no-go...