Unable to start an unprivileged container on fresh install of Fedora 31

[toddhpoole]

Required information

• Distribution: Fedora
• Distribution version: 31
• The output of
  • lxc-start --version

[regularuser@testserver ~]$ lxc-start --version
3.0.4

• lxc-checkconfig

[regularuser@testserver ~]$ lxc-checkconfig
Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-5.3.14-300.fc31.x86_64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Warning: newuidmap is not setuid-root
Warning: newgidmap is not setuid-root
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points: 


Cgroup v2 mount points: 
/sys/fs/cgroup

Cgroup v1 systemd controller: missing
Cgroup v1 freezer controller: missing
Cgroup namespace: required
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, not loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: 

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

• uname -a

[regularuser@testserver ~]$ uname -a
Linux testserver 5.3.14-300.fc31.x86_64 #1 SMP Mon Dec 2 15:41:35 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• cat /proc/self/cgroup

[regularuser@testserver ~]$ cat /proc/self/cgroup
0::/user.slice/user-1000.slice/session-7.scope

• cat /proc/1/mounts

[regularuser@testserver ~]$ cat /proc/1/mounts
sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16337860k,nr_inodes=4084465,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0
devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0
cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate 0 0
pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0
efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/mapper/fedora_host-root / xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=25757 0 0
debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0
tmpfs /tmp tmpfs rw,seclabel,nosuid,nodev 0 0
/dev/sda2 /boot xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
/dev/sda1 /boot/efi vfat rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro 0 0
tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3270444k,mode=700 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3270444k,mode=700,uid=1000,gid=1000 0 0

Issue description

After a fresh install of Fedora 31 and LXC, and after creating the ~/.config/lxc directories and ~/.config/lxc/default.conf configuration file per the LXC Getting Started Guide, we're unable to start a newly-created container.

Steps to reproduce

1. Run lxc-create as an unprivileged user on a fresh install:

[regularuser@testserver ~]$ lxc-create -t download -n testfedora31_2 -- -d fedora -r 31 -a amd64
Setting up the GPG keyring
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created a Fedora 31 x86_64 (20191208_20:33) container.

2. Try to start the container:

[regularuser@testserver ~]$ lxc-start -n testfedora31_2 -l DEBUG -o ~/output.log
lxc-start: testfedora31_2: lxccontainer.c: wait_on_daemonized_start: 850 Received container state "STOPPING" instead of "RUNNING"
lxc-start: testfedora31_2: tools/lxc_start.c: main: 329 The container failed to start
lxc-start: testfedora31_2: tools/lxc_start.c: main: 332 To get more details, run the container in foreground mode
lxc-start: testfedora31_2: tools/lxc_start.c: main: 334 Additional information can be obtained by setting the --logfile and --logpriority options

Information to attach

• any relevant kernel output (dmesg)
  Not applicable. No kernel logs generated.
• container log (The file from running lxc-start -n <c> -l <log> -o DEBUG)
  Attached output.log.
• the containers configuration file
  Attached config.txt.

[emojifreak]

I observed the same phenomenon with Ubuntu 19.10 booted with systemd.unified_cgroup_hierarchy and

$ dpkg-query -W | grep lxc
liblxc-common	3.0.4-0ubuntu1
liblxc1	3.0.4-0ubuntu1
lxc-utils	3.0.4-0ubuntu1
lxcfs	3.0.4-2

When I started a container with systemd-run --user --scope -p "Delegate=yes" lxc-start -F then the error reported in this issue disappeared and I could start /bin/bash by lxc.init.cmd = /bin/bash in the config file.

@brauner @stgraber Could you consider to update the user instruction or the program code at the next bug fix release of 3.0 LTS series so that a non-root user following the instruction can start an unprivileged container?

[brauner]

On December 9, 2019 9:08:55 PM GMT+01:00, emojifreak ***@***.***> wrote: I observed the same phenomenon with Ubuntu 19.10 booted with `systemd.unified_cgroup_hierarchy` and ```shell-session $ dpkg-query -W | grep lxc liblxc-common 3.0.4-0ubuntu1 liblxc1 3.0.4-0ubuntu1 lxc-utils 3.0.4-0ubuntu1 lxcfs 3.0.4-2 ``` When I started a container with `systemd-run --user --scope -p "Delegate=yes" lxc-start` then the error reported in this issue disappeared and I could start `/bin/bash` by `lxc.init.cmd = /bin/bash` in the config file. @brauner @stgraber Could you consider to update the user instruction or the program code at the next bug fix release of 3.0 LTS series so that a non-root user following the instruction can start an unprivileged container?

Sure, but also, I've backported all the cgroup2 improvements from the master branch to the stable-3.0 branch. So on the next stable bugfix release the experience should be way better.

[emojifreak]

@brauner Thank you very much for your quick response! Are you going to tell users to use systemctl / systemd-run --user or to use another mechanism?

Unless user-writable cgroupv2 directory is provided, lxc completely fails to function even with the newest github master branch and user experience is probably the same. The LXC user instruction should allow a non-root user to at least start a container, even with no CGroup V2 controllers enabled.

[brauner]

On Mon, Dec 09, 2019 at 01:29:33PM -0800, emojifreak wrote: @brauner Thank you very much for your quick response! Are you going to tell users to use `systemctl / systemd-run --user` or to use another mechanism? I am just asking from my curiosity and I don't have an opinion about that.

I think I'd point that that LXC expects to be placed into a delegated cgroup and will assume complete ownership of it. Then go on to point out that systemctl and systemd-run --user are ways to achieve that and give an example. If you want you can also send a patch for this.

[emojifreak]

Thank you for your reply.

If you want you can also send a patch for this.

https://linuxcontainers.org/lxc/getting-started/ seems the most popular and the most authoritative instruction on how to prepare and start an unprivileged LXC container by a non-root user. Because googling lxc unprivileged gives that URL as the most relevant page. As it is top-ranked by Google, it probably has to contain up-to-date instruction on how to prepare unprivileged LXC container on host Linux with systemd.unified_cgroup_hierarchy.

On the other hand, the LXC github does not seem to have a definitive instruction on how a system administrator and non-root users should prepare and start an unprivileged LXC container used by non-root users. So, to which file did you mean by a patch? Is it https://github.com/lxc/lxc/blob/master/doc/pam_cgfs.sgml.in ?

[toddhpoole]

Hi @brauner, @emojifreak,

@brauner, good to hear you've already backported a few fixes to address this. What's your release candence like? It'd be a huge help if we could get that bugfix release before the holidays.

@emojifreak, https://linuxcontainers.org/lxc/getting-started/ is exactly the page that lead us astray. Given it's popularity (it's the top Google result, it's linked to from almost everywhere in StackOverflow, etc.), it was our understanding that this was the actual, official guide maintained by LXC. Its GitHub repo (https://github.com/lxc/linuxcontainers.org) seems to confirm that as it has many of the same contributors as this repo. @brauner, was our understanding correct?

Regardless, we'd also be happy to assisst with updating the guide so that it's no longer incorrect/doesn't mislead readers.

[emojifreak]

I am unsure if I am qualified to make a pull request to the document. But it seems that

https://github.com/lxc/linuxcontainers.org/blame/ef81da028bdeb1126a9eee466f2a6d4fd04c1fea/content/lxc/getting-started.md#L100-L102

Just before you create your first container, you probably should logout and login again,
or even reboot your machine to make sure that your user is placed in the right cgroups.
(This is only required if cgmanager wasn't installed on your machine prior to you installing LXC.)

• As cgmanager retired, cgmanager should be replaced by libpam-cgfs, while noting that libpam-cgfs is unnecessary as libpam-cgfs cannot help non-root users to use CGroup in LXC when Linux is booted with systemd.unified_cgroup_hierarchy #3198 for a host Linux with systemd.unified_cgroup_hierarchy (e.g. Fedora 31).

• lxc-create does not require a right cgroup, but lxc-start does.

https://github.com/lxc/linuxcontainers.org/blame/ef81da028bdeb1126a9eee466f2a6d4fd04c1fea/content/lxc/getting-started.md#L113-L115

A few seconds later your container will be created and you can start it with:

lxc-start -n my-container -d

For a host Linux with systemd.unified_cgroup_hierarchy (e.g. Fedora 31), systemd-run --user --scope -p "Delegate=yes" lxc-start -n my-container -d should also be suggested.

[toddhpoole]

What's your release candence like? It'd be a huge help if we could get that bugfix release before the holidays.

@brauner, pinging you again on this - do you plan to push out a release incorporating these bugfixes soon?

For a host Linux with systemd.unified_cgroup_hierarchy (e.g. Fedora 31), systemd-run --user --scope -p "Delegate=yes" lxc-start -n my-container -d should also be suggested.

@emojifreak, awesome, thanks for the suggested invocation method.

[brauner]

What's your release candence like? It'd be a huge help if we could get that bugfix release before the holidays.

@brauner, pinging you again on this - do you plan to push out a release incorporating these bugfixes soon?

Yeah, once we've stabilized for sure. Pre-holidays there's always a lot going on at the same time so my response times varies, sorry about that. @stgraber might have a date for the next LTS-bugfix release in mind.

[emojifreak]

I hope https://github.com/lxc/linuxcontainers.org/blob/master/content/lxc/getting-started.md will also be properly updated before the 3.0.5 release, otherwise Fedora 31 users continue to suffer from being unable to start an unprivileged container by non-root users, reading and following "Getting started" in the same way as @toddhpoole ...

[toddhpoole]

@brauner, understood - thanks for the added clarity. We're in a similar boat re: the pre-holiday hustle and are trying wrap up a few things before the holidays (this being one of them).

@stgraber, do you have any insight on the next bugfix release? Every project's release process is different, but we'd be happy to contribute man-hours if it'd help move the needle.

[toddhpoole]

Hi @stgraber,

Quick ping on the above. Any guidance on when the next LXC release will occur? 3.0.4 was published 6 months ago and there's been a lot of useful fixes and cgroup-releated changes merged since then.

In terms of timing, Fedora 32 expects to branch from Rawhide in a few weeks. Do you think you'll be able to release before early February?

[brauner]

We're about to release LXC 4.0 at the end of this week!

[toddhpoole]

Hi @brauner,

Glad to finally get some clarity here. 9 months is a long time for a project like LXC to go with no clear communication regarding a roadmap or release schedule. @stgraber, if either of these resources exist and I missed them, it might be worth ensuring they're visible on the project's website.

Two quick questions about your planned 4.0 release:

1. Has anyone actually tried using this release on distros with CGroups v2 enabled by default (basically, Fedora 31+ or any of its derivatives)? You'll recall that when 3.0 was pushed, the release notes championed full support for CGroups v2, but after so many people had so many issues running in CGroups v2-based distros, it seemed like the project's reputation took a pretty negative hit. (By way of example, it's not uncommon to see folks around the web and at conferences compare the project to "garbage", which is unfortunate).

2. Will this release come with updated/validated documentation on the official LXC Getting Started Guide published at https://linuxcontainers.org/lxc/getting-started/?

[elraymond]

the release notes championed full support for CGroups v2, but after so many people had so many issues running in CGroups v2-based distros

Well, I've just been playing around with things a little - while moving back to lxc from lxd again for my private stuff, thanks to snap - and found that there's two very different things to consider: cgroup v2 integration and systemd (for cgroup v2) integration. And the complaints I came across during testing mostly concerned the latter.

More precisely, if you happened to read through https://systemd.io/CGROUP_DELEGATION/ you'd understand why systemd units need to be "delegated" and (some) lxc commands be launched through systemd-run if you're an unprivileged user. That's just because "no procs in inner nodes" and systemd trying to lay its greedy, greasy claws onto the entire tree, grabbing as much as it can (how often they repeat how much was "off limits" to you is actually pretty amusing - you come under the impression that you're trying to steal their chocolate from them).

Anyway, with the Arch recommendations (https://wiki.archlinux.org/index.php/Cgroups#Switching_to_cgroups_v2) my containers seem to run quite smoothly under an unprivileged user, too, on Ubuntu Focal. So, again, it's not cgroup v2 that seems to be the problem.

Edit: Just for clarification, the relevant part is "systemctl edit user@1000.service" to enable delegation on the user's service. Then run lxc-start like "systemd-run --user --scope lxc-start" to put it into a transient scope beneath (!) your user service, where it has write access thanks to delegation. And, for example, attach like "systemd-run --user -t lxc-attach" to do the same, only this time with a transient service that retains a terminal for you to use.

Edit # 2: Fact is, when reading through the systemd document, you do recognize (some of) the lxc cgroup v2 integration proper that has been done. Like the split into "-monitor" and "-payload" trees, to comply with the v2 "procs only in leaves" paradigm. See point 1 under "Some Dos."

Edit # 3: To close, all this works under Bionic, too, except systemd 237 doesn't play along and forces you to do what Poettering&affiliates so vehemently argue against, mess with the user slice: systemd/systemd#3388 (comment)
Also, as I found out in the meantime, the systemd-run calls above still need to have an explicit "-p Delegate=yes" option set, or else not all controllers will be handed down. All tests done with lxc 4.0.2, by the way.

[bersace]

Hi,

Wrapping lxc-start in a systemd-run call looks like a big regression in term of user experience. I added an alias to my shell to workaround this. Is there a way for lxc to manage this ?

$ grep lxc-start ~/.bashrc 
alias lxc-start='systemd-run --user --remain-after-exit --property Delegate=yes lxc-start'

[brauner]

On Wed, Apr 14, 2021 at 01:14:35AM -0700, Étienne BERSAC wrote: Hi, Wrapping lxc-start in a systemd-run call looks like a big regression in term of user experience. I added an alias to my shell to workaround this. Is there a way for lxc to manage this ?

Yeah, it's certainly annoying. I'm not sure if we could either move this into the lxc-* binaries for now to make this work as a short-term solution and ultimately find a way to interact with systemd directly. The problem I have with this is that the APIs they currently provide are a bit too convoluted. I essentially would like to have a simple single sd-bus call: int gimme_a_delegated_cgroup_fd() and be done with it. That's what we need. No complicated connecting to the bus first and stuff like that. Christian

[fergu]

I'm just commenting, from more of a user viewpoint, to add that I absolutely agree that having to use systemd-run to start a container, while it works, is a big user experience regression - to the point that I first dismissed the suggestion to run a container this way as a hack/outdated when I saw it on https://wiki.debian.org/LXC/CGroupV2 and ended up falling down the google rabbit hole for a few days. It did work to start the container - I just assumed it was a way of getting it done, versus doing it right.

The amount of old/outdated info out there concerning lxc doesn't help either. Even Debian's own wiki page for LXC gives outdated info: https://wiki.debian.org/LXC#Unprivileged_container