Fedora 40 Changes: Podman v5

[jlebon]

The podman team is planning to rebase to v5 in Fedora 40: https://fedoraproject.org/wiki/Changes/Podman5.

This includes a few breaking changes.

• It drops support for CNI networking entirely. The default has changed to Netavark since v4.0 (during the rebase to f36), but existing containers cannot be converted so are still using CNI networking.
• It drops support for cgroups v1. We've long changed the default for new nodes to cgroups v2, but not upgrading nodes.

We'll need to communicate this and include links to steps on how to adapt existing nodes.

[jlebon]

We moved to cgroups v2 by default in f34 and to netavark in f36. So anyone who's reprovisioned since f36 shouldn't be affected by this.

For anyone else, they will need to either reprovision with newer bootimages, or:

• move their nodes to cgroups v2 if not already
• reset podman storage and recreate containers to get the new defaults (to be verified)

[dustymabe]

We discussed this in the community meeting today.

12:09:29*  dustymabe | !info as soon as the podman v5 change gets accepted for Fedora 40 we should add CLHM helpers to
                     | notifiy people of the incoming changes and also a coreos-status post with the details

[dustymabe]

This has now been accepted

[gursewak1997]

Also, Podman v5 doesn't ship podman-plugins rpm anymore which is expected considering the drop of support for CNI networking entirely. Also, containernetworking-plugins was dropped as a dependency of podman but we explicitly pulled that in for upgrading nodes that were using CNI networking. We wouldn't need that anymore.
So, we will have to remove them from our manifest.
Relevant commits:

• containers/podman@ac3d570
• containers/podman@3a47342
• F36: tracker for package additions/removals #1128 (comment)

[dustymabe]

ok so we can drop podman-plugins and containernetworking-plugins from our manifest since they are in support of CNI networking which is no longer supported in Podman v5.

From:

core@apu2:~$ rpm -qi podman-plugins 
Name        : podman-plugins
Epoch       : 5
Version     : 4.8.3
Release     : 1.fc39
Architecture: x86_64
Install Date: Mon Feb  5 22:41:36 2024
Group       : Unspecified
Size        : 3701652
License     : Apache-2.0 AND BSD-2-Clause AND BSD-3-Clause AND ISC AND MIT AND MPL-2.0
Signature   : RSA/SHA256, Wed Jan  3 14:21:29 2024, Key ID 75cf5ac418b8e74c
Source RPM  : podman-4.8.3-1.fc39.src.rpm
Build Date  : Wed Jan  3 14:11:33 2024
Build Host  : buildvm-x86-20.iad2.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://podman.io/
Bug URL     : https://bugz.fedoraproject.org/podman
Summary     : Plugins for podman
Description :
This plugin sets up the use of dnsmasq on a given CNI network so
that Pods can resolve each other by name.  When configured,
the pod and its IP address are added to a network specific hosts file
that dnsmasq will read in.  Similarly, when a pod
is removed from the network, it will remove the entry from the hosts
file.  Each CNI network will have its own dnsmasq instance.

That explains why dnsmasq is there too on that line in the config. I would say we could remove it but... we do have docs that mention it being used for both podman and NetworkManager so we can't just remove it but we can update the docs to drop the podman reference and update the comment in the manifest to mention NetworkManager.

[gursewak1997]

We've incorporated CLHM helpers to inform individuals about upcoming changes. Additionally, we've removed the inclusion of containernetworking-plugins and podman-plugins.
If there's anything additional needed regarding the Podman v5 changes, please feel free to open this issue.

[dustymabe]

In the meeting #1629 (comment) we said we'd do a coreos-status post. I think that still needs to happen.

[travier]

https://discussion.fedoraproject.org/t/switching-from-cni-to-netavark-on-fedora-coreos-non-destructively/106594 > This sounds like a great question that we would need an answer to.

https://discussion.fedoraproject.org/t/switching-from-cni-to-netavark-on-fedora-coreos-non-destructively/106594/4

[travier]

I've just successfully converted a system that was using "ephemeral" containers (i.e. running using --rm and storing everything in volumes bind mounts, not podman volumes). It "still" needed conversion even though no volumes where used and the containers and networks were created fresh on boot (using quadlets).

Stopping all containers, running podman system reset --force and rebooting worked well.

[travier]

An option in podman system reset to remove all networks, containers but not volumes would be nice and would likely help for https://discussion.fedoraproject.org/t/switching-from-cni-to-netavark-on-fedora-coreos-non-destructively/106594 as it's likely the most common case.

[travier]

More Links:

• https://blog.podman.io/2024/03/podman-5-0-has-been-released/
• https://blog.podman.io/2024/03/podman-5-0-breaking-changes-in-detail/
• https://blog.podman.io/2024/03/migration-of-podman-4-to-podman-5-machines/

[dustymabe]

The fix for this went into next stream release 40.20240322.1.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into testing stream release 40.20240416.2.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 40.20240416.3.1.