lxc-start hangs on kernels >=5.17.0 when built with io-uring support

[living180]

Required information

• Distribution: Gentoo
• The output of
  • lxc-checkconfig:

    LXC version 4.0.12
    --- Namespaces ---
    Namespaces: enabled
    Utsname namespace: enabled
    Ipc namespace: enabled
    Pid namespace: enabled
    User namespace: enabled
    Network namespace: enabled
    
    --- Control groups ---
    Cgroups: enabled
    Cgroup namespace: enabled
    
    Cgroup v1 mount points: 
    /sys/fs/cgroup/systemd
    /sys/fs/cgroup/cpu,cpuacct
    /sys/fs/cgroup/devices
    /sys/fs/cgroup/net_cls,net_prio
    /sys/fs/cgroup/misc
    /sys/fs/cgroup/perf_event
    /sys/fs/cgroup/blkio
    /sys/fs/cgroup/memory
    /sys/fs/cgroup/freezer
    /sys/fs/cgroup/hugetlb
    /sys/fs/cgroup/pids
    /sys/fs/cgroup/cpuset
    
    Cgroup v2 mount points: 
    /sys/fs/cgroup/unified
    
    Cgroup v1 clone_children flag: enabled
    Cgroup device: enabled
    Cgroup sched: enabled
    Cgroup cpu account: enabled
    Cgroup memory controller: enabled
    Cgroup cpuset: enabled
    
    --- Misc ---
    Veth pair device: enabled, not loaded
    Macvlan: enabled, not loaded
    Vlan: enabled, not loaded
    Bridges: enabled, loaded
    Advanced netfilter: enabled, not loaded
    CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
    CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
    CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
    CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
    FUSE (for use with lxcfs): enabled, loaded
    
    --- Checkpoint/Restore ---
    checkpoint restore: enabled
    CONFIG_FHANDLE: enabled
    CONFIG_EVENTFD: enabled
    CONFIG_EPOLL: enabled
    CONFIG_UNIX_DIAG: enabled
    CONFIG_INET_DIAG: enabled
    CONFIG_PACKET_DIAG: enabled
    CONFIG_NETLINK_DIAG: enabled
    File capabilities: 
    
    Note : Before booting a new kernel, you can check its configuration
    usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig
    

  • uname -a:

    Linux elrond 5.17.7-gentoo-dist #1 SMP PREEMPT Fri May 13 08:16:02 EEST 2022 x86_64 AMD Ryzen 7 5700U with Radeon Graphics AuthenticAMD GNU/Linux
    

  • cat /proc/self/cgroup

    12:cpuset:/
    11:pids:/user.slice/user-1000.slice/session-1.scope
    10:hugetlb:/
    9:freezer:/
    8:memory:/user.slice/user-1000.slice/session-1.scope
    7:blkio:/user.slice
    6:perf_event:/
    5:misc:/
    4:net_cls,net_prio:/
    3:devices:/user.slice
    2:cpu,cpuacct:/user.slice
    1:name=systemd:/user.slice/user-1000.slice/session-1.scope
    0::/user.slice/user-1000.slice/session-1.scope
    

  • cat /proc/1/mounts

    proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
    sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
    devtmpfs /dev devtmpfs rw,nosuid,size=4096k,nr_inodes=1048576,mode=755,inode64 0 0
    securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
    tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0
    devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
    tmpfs /run tmpfs rw,nosuid,nodev,size=6454332k,nr_inodes=819200,mode=755,inode64 0 0
    tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755,inode64 0 0
    cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime 0 0
    cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
    pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
    efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
    bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
    cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /sys/fs/cgroup/misc cgroup rw,nosuid,nodev,noexec,relatime,misc 0 0
    cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
    cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
    cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
    cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
    cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
    /dev/disk/by-uuid/f45b231e-72c5-4228-b518-22d9e723a365 / ext4 rw,lazytime 0 0
    systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20582 0 0
    hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0
    mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
    debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
    tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
    fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
    configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
    tmpfs /tmp tmpfs rw,nosuid,nodev,noatime,inode64,huge=always 0 0
    /dev/nvme0n1p1 /boot vfat rw,nosuid,nodev,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro 0 0
    /dev/mapper/vg-var /var ext4 rw,lazytime,nodev 0 0
    /dev/mapper/vg-home /home ext4 rw,lazytime,nosuid,nodev 0 0
    tmpfs /var/tmp/portage tmpfs rw,nosuid,nodev,noatime,mode=755,uid=250,gid=250,inode64,huge=always 0 0
    binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
    lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
    tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3227164k,nr_inodes=806791,mode=700,uid=1000,gid=1000,inode64 0 0
    

Issue description

lxc-start hangs on kernels >=5.17.0 when built with io-uring support. This was caused by a combination of the kernel no longer downgrading tty/pty multishot poll requests to oneshot (kernel commit aa43477b0402) in conjunction with incorrect usage of multishot io-uring by LXC. Details of the problem are available in this email message and a POC fix is attached to this email message.

Steps to reproduce

The following commands are sufficient to reproduce the problem for me:

 sudo lxc-create --n lxc-test --template download --bdev dir --dir /var/lib/lxc/lxc-test/rootfs -- -d ubuntu -r bionic -a amd64
 sudo lxc-start -n lxc-test
 sudo lxc-stop -n lxc-test

The lxc-stop command never exits and the container continues running because lxc-start is hung in a read(2) call that will never return.

[brauner]

Thank you for the issue report. I've asked on the mailing list what the best way to fix this would be.

[n1tehawk]

This also seems to happen with older kernel versions, see e.g. #4081

I ran into similar issues with lxc 5.0.1 under various 5.15.x-gentoo kernels (gentoo-sources) on a virtual private server (KVM-based). When compiled with USE=io-uring, lxc-start and lxc-stop would occasionally work fine - but they'd fail more often than not, causing anything trying to access/monitor the containers to hang (like lxc-ls and lxc-monitor). The only (quick) way out was to kill -9 the corresponding lxc-start process.

Recompiled lxc without io-uring, and the troubles were gone.
Regards, NiteHawk

[n1tehawk]

see also #4256

[mihalicyn]

see also axboe/liburing#849

[mihalicyn]

It should be fixed by #4304 or (if it was a kernel bug) in recent Linux kernel versions.
Please check and feel free to reopen if it's still a problem.