Consider adding containernetworking-plugins explicitly to Silverblue F36+

[travier]

Describe the bug

Fedora 36 is moving to podman v4 and previous containers created with podman v3 which are thus still using the CNI networking might not work anymore.

This is the same issue as the one for Fedora CoreOS: coreos/fedora-coreos-config#1613

Thus we should probably consider shipping CNI networking by default for F36 at least.

This might break existing toolbxes too.

To Reproduce
Please describe the steps needed to reproduce the bug:

1. Create a podman container in Silverblue 35
2. Rebase to 36
3. Try using the network in the container

[tpopela]

Shouldn't this be reported against toolbx project? From my POV it would make sense to add that package as a requirement from the toolbox package so it continues to work in Fedora Workstation.

On the other hand I do have toolbxes that were created in SB35 and I'm already on SB36 and didn't noticed any network issues (and I'm a heavy toolbx user).

[travier]

This might impact only network listening applications in containers and thus toolbx would not be affected.

CC @dustymabe

[travier]

Toolbxes are not affected because they use host networking. So this is only for other podman containers.

[tpopela]

Then I would prefer not to do any changes in Silverblue - this should be handled on the podman side. Let me also ask Rishi.

[debarshiray]

Thanks for flagging this, @travier

It's true that Toolbx doesn't use a network namespace and uses the host's networking. So, I am leaning towards ignoring this problem for Silverblue. Unless, like CoreOS, we care strongly about keeping backwards compatibility with older generic Podman containers, and in that case, maybe containernetworking-plugins should be restored (ie., a partial revert of commit 06c601234feac4f5da7b6e1d9243) to the podman RPM?

[dustymabe]

In my opinion we do care about containers that aren't toolbox containers and preserving their functionality on upgrade. I've got 10+ containers on my system that aren't toolbox.

The podman devs dropped the containernetworking-plugins rpm on purpose but they weren't considering the fact that on OSTree we don't upgrade like traditional yum/dnf systems. Their intention was for any upgrading nodes to continue to keep containernetworking-plugins installed, which won't happen on OSTree by default.

Here's what we want to do:

• add containernetworking-plugins in our manifest list
  • supports CNI networks for upgrading systems
  • see F36: tracker for package additions/removals coreos/fedora-coreos-tracker#1128 (comment)
• add netavark in our manifest list
  • supports new Podman 4.0 networking
  • needs to be named because if we don't then only containernetworking-plugins will be installed
  • see F36: tracker for package additions/removals coreos/fedora-coreos-tracker#1128 (comment)
• add aardvark-dns in our manifest list
  • supports name resolution for container names in new networking
  • see F36: tracker for package additions/removals coreos/fedora-coreos-tracker#1128 (comment)

[debarshiray]

In my opinion we do care about containers that aren't toolbox
containers and preserving their functionality on upgrade. I've got
10+ containers on my system that aren't toolbox.

Okay!

The podman devs dropped the containernetworking-plugins
rpm on purpose but they weren't considering the fact that on
OSTree we don't upgrade like traditional yum/dnf systems. Their
intention was for any upgrading nodes to continue to keep
containernetworking-plugins installed, which won't happen on
OSTree by default.

Is it just about upgrading nodes? What if I have a separate $HOME partition on a Fedora Workstation where my rootless containers are, and I wipe the rest of OS partitions and do a clean installation of Fedora >= 36. It will also break my existing containers.

That's what I do every year. :)

Assuming that Podman wants to keep backwards compatibility, it seems best to restore the containernetworking-plugins dependency to the podman RPM? Otherwise we would be caught patching this up in all sorts of places. eg., Workstation's comps, the Silverblue and CoreOS manifests, etc., and then maybe in downstreams like RHEL in the future if the spec file gets copied around.

Is there a good reason not do that? After all, isn't CoreOS an important use-case for Podman -- the cloud, Kubernetes, OpenShift, etc.. So, if even CoreOS, with it's size restrictions, wants the extra dependency on containernetworking-plugins to keep old containers working, then is there any reason not to want it?

[dustymabe]

In my opinion we do care about containers that aren't toolbox
containers and preserving their functionality on upgrade. I've got
10+ containers on my system that aren't toolbox.

Okay!

The podman devs dropped the containernetworking-plugins
rpm on purpose but they weren't considering the fact that on
OSTree we don't upgrade like traditional yum/dnf systems. Their
intention was for any upgrading nodes to continue to keep
containernetworking-plugins installed, which won't happen on
OSTree by default.

Is it just about upgrading nodes? What if I have a separate $HOME partition on a Fedora Workstation where my rootless containers are, and I wipe the rest of OS partitions and do a clean installation of Fedora >= 36. It will also break my existing containers.

That's what I do every year. :)

:) - That's a valid use case, but probably much less common.

Assuming that Podman wants to keep backwards compatibility, it seems best to restore the containernetworking-plugins dependency to the podman RPM? Otherwise we would be caught patching this up in all sorts of places. eg., Workstation's comps, the Silverblue and CoreOS manifests, etc., and then maybe in downstreams like RHEL in the future if the spec file gets copied around.

Is there a good reason not do that? After all, isn't CoreOS an important use-case for Podman -- the cloud, Kubernetes, OpenShift, etc.. So, if even CoreOS, with it's size restrictions, wants the extra dependency on containernetworking-plugins to keep old containers working, then is there any reason not to want it?

I think it depends. I was originally arguing for keeping it as well, but there is also an argument for not having new installs keep extra packages that they don't need. If you're not upgrading (or migrating $HOME like you suggested above) then you don't need the old stack.

Unfortunately I think the "migrate $HOME" use case For Fedora Workstation is where the line gets drawn.

In FCOS we autoupdate nodes so we really need to try to keep things working. For SB I could foresee us telling people they need to wipe and re-provision their containers, but I'd prefer if we did some sort of communications campaign about it.

We'll probably try to do something similar for FCOS (i.e. messages that show up in the console letting people know they should migrate), so we might be able to collaborate on it.

[debarshiray]

Personally, I don't have a strong opinion about this particular issue about containernetworking-plugins. All my OCI containers are Toolbxes, and all but one of my computers are running Silverblue. :)

What I really care about is Podman being serious about backwards compatibility, and API and ABI stability because those are crucial for Toolbx. It used to be very rocky in the past, but has improved a lot these days. So, I am seeing this through the lens of Podman being serious about backwards compatibility, and that always comes with a price and needs to be considered at every step.

Assuming that Podman wants to keep backwards compatibility, it seems best to restore the containernetworking-plugins dependency to the podman RPM? Otherwise we would be caught patching this up in all sorts of places. eg., Workstation's comps, the Silverblue and CoreOS manifests, etc., and then maybe in downstreams like RHEL in the future if the spec file gets copied around.
Is there a good reason not do that? After all, isn't CoreOS an important use-case for Podman -- the cloud, Kubernetes, OpenShift, etc.. So, if even CoreOS, with it's size restrictions, wants the extra dependency on containernetworking-plugins to keep old containers working, then is there any reason not to want it?

I think it depends. I was originally arguing for keeping it as well, but there is also an argument for not having new installs keep extra packages that they don't need. If you're not upgrading (or migrating $HOME like you suggested above) then you don't need the old stack.

Would it hurt so much to keep it around for a couple of Fedora releases or so? We do keep things like Obsoletes and other bits and pieces of RPM metadata around for some cases.

Unfortunately I think the "migrate $HOME" use case For Fedora Workstation is where the line gets drawn.

Upgrading a Workstation has only started being reliable relatively recently, and even if it doesn't explode, it's common for it to start diverging from a clean installation. eg., plugins shipped separately through comps don't get pulled in, and there might be other cases.

Anyway, ... :)

In FCOS we autoupdate nodes so we really need to try to keep things working. For SB I could foresee us telling people they need to wipe and re-provision their containers, but I'd prefer if we did some sort of communications campaign about it.

We'll probably try to do something similar for FCOS (i.e. messages that show up in the console letting people know they should migrate), so we might be able to collaborate on it.

I wonder what the failure mode is.

One option could be that Podman 4.0 starts throwing a deprecation notice on encountering the old networking stack for pre-existing containers, give users a year or so to move away from it, and continue to pull in containernetworking-plugins during that deprecation period.

[dustymabe]

Personally, I don't have a strong opinion about this particular issue about containernetworking-plugins. All my OCI containers are Toolbxes, and all but one of my computers are running Silverblue. :)

What I really care about is Podman being serious about backwards compatibility, and API and ABI stability because those are crucial for Toolbx. It used to be very rocky in the past, but has improved a lot these days. So, I am seeing this through the lens of Podman being serious about backwards compatibility, and that always comes with a price and needs to be considered at every step.

+1 It's mostly up to the podman team on this one. I'm just the messenger.

Assuming that Podman wants to keep backwards compatibility, it seems best to restore the containernetworking-plugins dependency to the podman RPM? Otherwise we would be caught patching this up in all sorts of places. eg., Workstation's comps, the Silverblue and CoreOS manifests, etc., and then maybe in downstreams like RHEL in the future if the spec file gets copied around.
Is there a good reason not do that? After all, isn't CoreOS an important use-case for Podman -- the cloud, Kubernetes, OpenShift, etc.. So, if even CoreOS, with it's size restrictions, wants the extra dependency on containernetworking-plugins to keep old containers working, then is there any reason not to want it?

I think it depends. I was originally arguing for keeping it as well, but there is also an argument for not having new installs keep extra packages that they don't need. If you're not upgrading (or migrating $HOME like you suggested above) then you don't need the old stack.

Would it hurt so much to keep it around for a couple of Fedora releases or so? We do keep things like Obsoletes and other bits and pieces of RPM metadata around for some cases.

I don't think it would hurt, but I'm not the podman team.

Unfortunately I think the "migrate $HOME" use case For Fedora Workstation is where the line gets drawn.

Upgrading a Workstation has only started being reliable relatively recently, and even if it doesn't explode, it's common for it to start diverging from a clean installation. eg., plugins shipped separately through comps don't get pulled in, and there might be other cases.

Anyway, ... :)

In FCOS we autoupdate nodes so we really need to try to keep things working. For SB I could foresee us telling people they need to wipe and re-provision their containers, but I'd prefer if we did some sort of communications campaign about it.
We'll probably try to do something similar for FCOS (i.e. messages that show up in the console letting people know they should migrate), so we might be able to collaborate on it.

I wonder what the failure mode is.

One option could be that Podman 4.0 starts throwing a deprecation notice on encountering the old networking stack for pre-existing containers, give users a year or so to move away from it, and continue to pull in containernetworking-plugins during that deprecation period.

Yeah that's a nice option to let people know.

[travier]

I don't think this is something that we have to ask of the podman team to act upon here as it can not be expressed in terms of RPM dependencies as this is an issue specific to rpm-ostree based systems: dependent packages no longer explicitly requested are dropped.

In this case, both Fedora CoreOS and Fedora IoT have already decided to keep this package in the image for now.

I think we should keep it in Silverblue and Kinotie and do as @dustymabe said in #246 (comment). It does nothing for new installations that will use the new defaults and it gives upgrading users more time to move to the full podman v4 network stack.

We can then drop it once the podman team drops support for the CNI stack.

The package is about ~72MB which is OK considering it was already there in F35.

I'll make a PR to give this a try and also test an update with a container to see how things break in practice.

[travier]

If we decide to not do this, we should probably communicate somewhere in the release notes that Silverblue users will have to reset their container storage (which includes their toolboxes) after the F36 update to make their containers work again: https://podman.io/blogs/2022/02/04/network-usage.html#setting-up-podman-4-with-netavark-and-aardvark-dns-on-fedora

[travier]

PR: https://pagure.io/workstation-ostree-config/pull-request/273 (needs testing)

[travier]

Or we can mention that users affected by this can overlay the package and we don't do anything.

[tpopela]

Thank you @travier for the PR, but I actually think that what you're proposing in the last comment - asking the affected users (in documentation and in Fedora Discourse) to overlay the packages in case the containers stop to work for them.

[travier]

OK. Let's make sure this is part of the announcement for the release or somewhere easily discoverable.

[travier]

Keeping open until we have docs.

[har7an]

Hi,

I just stumbled over this one and opened an issue against podman (containers/podman#14189) before I found this. In case this is a part of the release notes I'm afraid I haven't found it... I ran podman system reset --force and now it works.

I guess the /etc/cni folder is obsolete now, right?

[travier]

I haven't written the docs for that one yet unfortunately, sorry. The CNI plugins are not deprecated yet but they are no longer the default. See: https://www.redhat.com/sysadmin/podman-new-network-stack

[kristofer-lp8f]

I ran into this missing dependency. It took some time to realize the issue and solution. As one end-user of podman/podman-compose I would vote for the containernetworking-plugins package to come with podman/podman-compose.

[FilBot3]

@har7an

Hi,

I just stumbled over this one and opened an issue against podman (containers/podman#14189) before I found this. In case this is a part of the release notes I'm afraid I haven't found it... I ran podman system reset --force and now it works.

I guess the /etc/cni folder is obsolete now, right?

I'm afraid to try this out as I've been using my Toolbox container quite heavily before finding this issue.

[FilBot3]

@har7an
Hi,
I just stumbled over this one and opened an issue against podman (containers/podman#14189) before I found this. In case this is a part of the release notes I'm afraid I haven't found it... I ran podman system reset --force and now it works.
I guess the /etc/cni folder is obsolete now, right?

I'm afraid to try this out as I've been using my Toolbox container quite heavily before finding this issue.

Well, I did it. Same errors. So, I need to layer on the containernetworking-plugins package to make these errors go away?

➜  Development
> podman system reset --force
WARN[0077] Error validating CNI config file /var/home/filbot/.config/cni/net.d/87-podman.conflist: [failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "portmap" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "firewall" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "tuning" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]]
WARN[0077] Error validating CNI config file /var/home/filbot/.config/cni/net.d/k0s-net.conflist: [failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "portmap" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "firewall" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "tuning" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]]
➜  Development
> podman network create k0s-net
WARN[0001] Error validating CNI config file /var/home/filbot/.config/cni/net.d/87-podman.conflist: [failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "portmap" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "firewall" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "tuning" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]]
WARN[0001] Error validating CNI config file /var/home/filbot/.config/cni/net.d/k0s-net.conflist: [failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "portmap" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "firewall" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "tuning" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]]
k0s-net

[travier]

Yes, this is likely the fix for you

[FilBot3]

Installing containernetworking-plugins did the trick for me and cleared up a lot of the errors.

sudo rpm-ostree install containernetworking-plugins --restart

[debarshiray]

Assuming that Podman wants to keep backwards compatibility, it seems best to restore the
containernetworking-plugins dependency to the podman RPM? Otherwise we would be caught
patching this up in all sorts of places. eg., Workstation's comps, the Silverblue and CoreOS
manifests, etc., and then maybe in downstreams like RHEL in the future if the spec file gets
copied around.

And, we have someone complaining about the podman RPM not pulling in containernetworking-plugins on Fedora 36:
https://bugzilla.redhat.com/show_bug.cgi?id=2124887

I still think that the dependency should be mentioned in the RPM, instead of trying to play whack-a-mole in other places.

[travier]

That's something to ask to the podman team. Edit: See Dusty's answer below.

[dustymabe]

That's something to ask to the podman team.

A lot of context in #246 (comment)

TL;DR if you upgrade a non-RPM-OSTree system you don't have this problem. In FCOS we named it explicitly in coreos/fedora-coreos-config#1613, which is what @travier suggested in #246 (comment) and was rejected in https://pagure.io/workstation-ostree-config/pull-request/273#comment-169238 and IMO is what we should have done here.

The podman devs aren't going to change anything at this point as we're a whole cycle in and they are looking forward.

[cgwalters]

Agree we should keep the package - for now.

[boydkelly]

I updated silverblue to 38 and now getting this. Is it correct then I have two choices?

1. layer containernetworking-plugins (this may disappear one day and I'll have to do 2 anyways)
2. podman system reset --force (rebuild all containers and move on)

[travier]

@boydkelly Yes, this is the summary.

[travier]

We have not written documentation for this issue yet but Fedora 36 is about to reach EOL so I'll close it.