cgroupsv2 issue with ~/.config/container/libpod.conf

[rhatdan]

We are going to have a problem in Fedora 31, where we are changing the default of the OCI runtime from runc to crun. We have to do this because runc is not ready for cgroupsV2 support. The problem is each rootless user and toolbox user, has a libpod.conf file that was automatically generated in their homedir, via the podman command. This conf file has hard coded default to use runc, so once you upgrade into Fedora31 your podman rootless containers will blow up.

Simply rm -f ~/.config/containers/libpod.conf will fix the problem. Podman will then regenerate the file with the current Fedora 31 defaults (basically crun as the oci runtime.).

Another option would be for the user to edit the

runtime = "runc"
line, to:
runtime = "crun"

Bottom line, is I believe we need to fix this in the podman command.

[rhatdan]

@mheon @baude FYI

One possible solution would be to add a new flag to the libpod.conf file
supports_cgroup_v2 = true

Then if the file exists and does not have this flag, we change the default runc to crun the first time we run podman, with a warning message on the screen.

Then add the flag.

When runc gets support for cgroupsv2 users can change their libpod.conf file and use it, and podman will leave it alone.

[mheon]

My initial impression is that we really need to replace runc with crun automagically at package install time - but that's not really doable on a per-user basis.

We could make podman run ignore the option, but that's a super bad precedent - we should not override user configs, even if we mostly wrote them ourselves.

[TomSweeneyRedHat]

Just a scope question. This is only a problem if you upgrade to F31 from a lower level right? If you do a new clean install, then libpod.conf gets generated appropriately right? Can/should we just document the heck out of it? I'm be worried about what happens in debian, ubuntu, etc if we did the supports_cgroup_v2 trick.

[mheon]

Yeah, this is basically only for upgrades, and even then only until runc manages to pick up support for V2 (at which time this goes back to working)

[rhatdan]

I think we could handle this with a check to see if machine is booted with cgroupsV2 support, has crun installed and does not have the flag.
If this situation happens, then add the flag and swap runc->crun.
If any other thing is untrue, then we don't modify ~/.config/containers/libpod.conf

I agree this is just a stop gap so that we don't blow up all users of Fedora 31. I think most people upgrade or at least share the previous used homedir.

[debarshiray]

Maybe Podman should not write out a libpod.conf with the defaults, and instead just use the defaults anyway? That would have simplified this migration a bit. Update the logic in the code to pick crun on Cgroups v2 systems by default, and in the absence of a libpod.conf it would just do the right thing.

[rhatdan]

I agree, we should not be copying the file. I think this was put in so that users would do the correct thing in the local config file. But it locks us in. I think it would be better to get the code to merge the system defaults with user overrides in the homedir. Then we could just initialize a empty libpod.conf in the users homedir. And only do overrides if the user modifies it.

But we still need to handle this upgrade.

[AdamWill]

if we can detect at runtime whether the systems has cgroupsv2 support, why modify the config file at all? Why not just print a warning that cgroupsv2 has been detected so we're using crun instead of runc, and do it? That's 'transient' behaviour which is much easier to modify later, and doesn't involve poking the user's home directory.

[mheon]

There is precedent for doing what @AdamWill suggests. If you modify Podman storage settings and they differ from the DB, we print angry log messages telling you that you've specified a configuration that would break Podman, and ignore what you specified to use the cached DB configuration. I really dislike the idea of overriding libpod.conf but it might be the neatest way to solve this.

Counterpoint: If I had a CGroupsV2 enabled runc how would I get Podman to use it, if we're explicitly overriding the config file setting?

[AdamWill]

the general idea would be once such a runc is available we revert the podman behaviour so it doesn't override the setting any more. if you're testing such a runc before it's widely available, i guess patch podman locally yourself :/

[mheon]

This should be fixed now.