CGroup blkio.throttle does not contains added block device.

[jcfaracco]

The template below is mostly useful for bug reports and support questions.
Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

Required information

• Distribution: ubuntu

• Distribution version: 19.10

• The output of

  • lxc-start --version
    3.2.1-devel

  • lxc-checkconfig

Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-5.3.0-23-generic
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
newuidmap is not installed
newgidmap is not installed
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points:
/sys/fs/cgroup/systemd
/sys/fs/cgroup/cpu,cpuacct
/sys/fs/cgroup/perf_event
/sys/fs/cgroup/blkio
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/net_cls,net_prio
/sys/fs/cgroup/rdma
/sys/fs/cgroup/memory
/sys/fs/cgroup/freezer
/sys/fs/cgroup/hugetlb
/sys/fs/cgroup/devices
/sys/fs/cgroup/pids

Cgroup v2 mount points:
/sys/fs/cgroup/unified

Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, loaded
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/local/bin/lxc-checkconfig

• uname -a
  Linux julio 5.3.0-23-generic avahi-daemon fails to start inside debian squeeze lxc container #25-Ubuntu SMP Tue Nov 12 09:22:33 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• cat /proc/self/cgroup
  12:pids:/user.slice/user-1000.slice/user@1000.service
  11:devices:/user.slice
  10:hugetlb:/
  9:freezer:/
  8:memory:/user.slice/user-1000.slice/user@1000.service
  7:rdma:/
  6:net_cls,net_prio:/
  5:cpuset:/
  4:blkio:/user.slice
  3:perf_event:/
  2:cpu,cpuacct:/user.slice
  1:name=systemd:/user.slice/user-1000.slice/user@1000.service/gnome-terminal-server.service
  0::/user.slice/user-1000.slice/user@1000.service/gnome-terminal-server.service

• cat /proc/1/mounts
  sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
  proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
  udev /dev devtmpfs rw,nosuid,relatime,size=3997644k,nr_inodes=999411,mode=755 0 0
  devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
  tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=805524k,mode=755 0 0
  /dev/sda9 / ext4 rw,relatime,errors=remount-ro 0 0
  securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
  tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
  tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
  tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
  cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
  cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
  pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
  efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
  bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
  cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
  cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
  cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
  cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
  cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
  cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0
  cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
  cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
  cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
  cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
  cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
  systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14823 0 0
  debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
  hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0
  mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
  fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
  configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
  /dev/loop0 /snap/gnome-characters/359 squashfs ro,nodev,relatime 0 0
  /dev/loop1 /snap/gnome-calculator/536 squashfs ro,nodev,relatime 0 0
  /dev/loop2 /snap/gnome-system-monitor/111 squashfs ro,nodev,relatime 0 0
  /dev/loop3 /snap/gnome-system-monitor/107 squashfs ro,nodev,relatime 0 0
  /dev/loop5 /snap/gnome-3-28-1804/91 squashfs ro,nodev,relatime 0 0
  /dev/loop4 /snap/gnome-3-28-1804/110 squashfs ro,nodev,relatime 0 0
  /dev/loop7 /snap/gnome-characters/367 squashfs ro,nodev,relatime 0 0
  /dev/loop6 /snap/core18/1223 squashfs ro,nodev,relatime 0 0
  /dev/loop8 /snap/gnome-logs/73 squashfs ro,nodev,relatime 0 0
  /dev/loop10 /snap/core18/1265 squashfs ro,nodev,relatime 0 0
  /dev/loop9 /snap/gtk-common-themes/1353 squashfs ro,nodev,relatime 0 0
  /dev/loop11 /snap/gnome-logs/81 squashfs ro,nodev,relatime 0 0
  /dev/loop12 /snap/core/7917 squashfs ro,nodev,relatime 0 0
  /dev/loop13 /snap/core/8039 squashfs ro,nodev,relatime 0 0
  /dev/loop15 /snap/gtk-common-themes/1313 squashfs ro,nodev,relatime 0 0
  /dev/loop14 /snap/gnome-calculator/544 squashfs ro,nodev,relatime 0 0
  /dev/sda1 /boot/efi vfat rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0
  binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
  tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=805520k,mode=700,uid=1000,gid=1000 0 0
  gvfsd-fuse /run/user/1000/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0
  /dev/fuse /run/user/1000/doc fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0

Issue description

When I run a LXC container and add a new device, it does not appear into LXC blkio cgroup group. It only shows my filesystem clock throttle.

Steps to reproduce

1. Start any container:
   $ lxc-start -n ubuntu
2. Attach a block device:
   $ lxc-device -n ubuntu -- add /dev/loopX
3. Check LXC cgroup blkio:
   $ cat /sys/fs/cgroup/blkio/lxc.payload/blkio.throttle.io_serviced
   8:0 Read 0
   8:0 Write 0
   8:0 Sync 0
   8:0 Async 0
   8:0 Discard 0
   8:0 Total 0
   Total 0

Information to attach

If we compare with docker, docker is adding devices into its cgroup blkio group.
$ docker run -ti --device=/dev/loop16:/dev/loop16 fedora:latest bash
$ cat /sys/fs/cgroup/blkio/docker/2db640bc7bb28a3b6bf83dd7b7c0f146b4433e3ed5dfa6899f8498f9a19c52c5/blkio.throttle.io_serviced
7:16 Read 34
7:16 Write 0
7:16 Sync 34
7:16 Async 0
7:16 Discard 0
7:16 Total 34
8:0 Read 417
8:0 Write 0
8:0 Sync 417
8:0 Async 0
8:0 Discard 0
8:0 Total 417
Total 451

[jcfaracco]

It seems that it is not a bug, but a feature missing.
It would be useful for lxc-top to fetch stats from extra devices.

[stgraber]

LXC doesn't add anything to the blkio controller unless you have it specified in your config though.
What's your config for this container?