Running with rootless podman doesn't work as documented #2872
Comments
This is not helpful …
rootless podman support is contributed by @AkihiroSuda, nominally kind is developed for rootful docker. Akihiro contributed some fedora based CI for rootless docker and podman, which does happen to be fedora based. This is the first we’ve encountered a rootless podman user that wasn’t on fedora, as you can tell. Most of our users use docker, which has mature support. Podman support is experimental (the tool should be printing a warning when you use podman) and is fundamentally limited by some of the significant differences where it is not drop in compatible, we have entirely separate code paths for podman behavior.
The kind process doesn’t touch cgroups. |
|
Host cgroups configuration is not something we currently plan to touch from the kind process, for example podman we’re invoking may actually be talking to a remote instance anyhow and it’s difficult to detect reliably (see discussions linked to #2233) |
I thought you only needed the version,
Huh, interesting, I thought podman would be more widespread 😅
Mh, then I guess I'll have to continue to run |
I am also using kind on Arch/Garuda, and we have encountered each other before @BenTheElder. In fact, I remember raising an issue about this, and giving up; #2684. Making this issue a duplicate of mine. @maciekmm is also on Arch and uses rootless, and he just made issue a few days ago. Podman is FOSS, and in the spirit of Please make
In fairness, I think he was trying to be funny. |
|
Actually, I interpreted Even if I had docker I would have still just inserted the version there 😅
What do you mean by that? I would say it's not coupled to anything DE related.
I will try this tomorrow 👍
Yeah, you can wrap the systemd-run --user --scope --property=Delegate=yes kind create clusterIf that's the recommended way to run |
We should clarify the template, this command provides necessary debug info.
This is not about ethics and podman receives a lot of attention. There are not simply "Arch, Fedora, RedHat, and Debian", there are infinite linux distros and configurations, new ones are created every day. We cannot support all of them equally (and RHEL requires a license ...). We have limited time and resources to run them locally and in CI. So far that means podman and rootless are tested primarily in CI on Fedora, we already have a large CI matrix for this small project. KIND is already supporting podman to a more than reasonable extent, at a relatively outsized cost. It is mostly docker compatible except when it isn't, which is fine, we've already developed separate implementations to support podman and set up CI ... However podman also makes breaking changes against it's own behavior. Off the top of my head: #2257, #2085 (comment), ... Docker has made exactly one small breaking change for the duration of this project (#2046) even though it has been supported for far longer. We support podman anyhow, even though the primary purpose of this project is to develop Kubernetes (see: https://kind.sigs.k8s.io/docs/contributing/project-scope/) and There are additional limitations to using podman (mainly around restart support) because podman simply does not handle these things due to difference in approach. Please remember that @aojea and I are already lending our free time or cutting into work time to support this and we could instead be improving / fixing Kubernetes (which relates to our actual current day jobs) or shipping a new KIND release at impact to far more users. I have recently gone way out of my way to prevent rootless from being broken in particular (#2846, kubernetes/enhancements#361 (comment) and the less visible work from myself and others meeting to find a ship and last minute fix to Kubernetes) because I do care about our users, but my time is bounded. I will review PRs to fix this, but it's simply not a priority for me to debug rootless podman x Arch ... Kubernetes only works fully on rootful and docker is a perfectly acceptable free and open source alternative to podman, I don't personally use Arch and I cannot run it on my employer-provided machines. |
|
Another consideration here: Projects like Kubernetes, podman, docker, runc, containerd, etc. also only run CI or develop for a limited set of environments, so we carry a higher cost to keep these things working together because the things we're integrating with are not developed or tested in these ways so as in kubernetes/enhancements#361 (comment) we have to turn around and proactively convince them to support these things and fix them to enable to support in KIND. I'll go a step further and say I'm willing to write docs changes or code patches to fix these environments myself if we receive sufficient information about how to fix them, but we're not going to stretch our CI matrix even further or locally develop on additional environments. It's already a lot. |
BenTheElder
added
area/provider/podman
|
Uh, but if podman is the one needing Delegation, then shouldn't podman have the corresponding configuration and documentation on how to get it up and running (which can then be directly linked by kind). |
The documentation is already there: It's comprehensive, and I'd go as far as saying that these two combined form a complete guide on how to run kind on rootless podman/docker. On top of that |
|
#3032 will clarify the bug template re: docker info / podman info. |
Took me a while to find this issue, I followed the hint and did all the cgroups v2 checks along with adding the On my system (Ubuntu 22.04 LTS + Podman 4.3.1) that hint doesn't work $ systemd-run --scope --user kind create cluster
Running scope as unit: run-r39bac831e38c4a4fad72f425230b9030.scope
enabling experimental podman provider
ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
$ systemd-run --user --property=Delegate=yes kind create cluster
Running as unit: run-r7137d1a0db4f46a5a1c6d6fbcf7225eb.service
$ kind get clusters
enabling experimental podman provider
No kind clusters found.
$ systemd-run --scope --user kind get clusters
Running scope as unit: run-r923df78bf2bc4b2bae955892be078d3c.scope
enabling experimental podman provider
No kind clusters found.Is there a corresponding issue in podman's issues that links to this? Update: Upon further inspection, @cwrau's invocation fails exactly the same way, except the error gets dumped in |
You can run it with I jut wanted to try this again to check if it's working on my end, but I was getting different errors; I fixed that by adding Then I was getting the following error; Which I fixed by running After that it's working 😁 |
What happened: I tried running kind with rootless podman and followed the documentation. But this didn't work
What you expected to happen: That it would work
How to reproduce it (as minimally and precisely as possible): Install and configure podman for rootless, install kind. Use a terminal with systemd-scopes, like gnome-terminal. Use an OS that doesn't
Delegateeverything, like Arch Linux. (Seems to be done on Fedora, https://gitlab.gnome.org/GNOME/gnome-terminal/-/issues/7914#note_1523590)Anything else we need to know?: @benzea stated in this ticket, that tools that depend on cgroups like kind does, should wrap themselves in either a unit or switch to a different scope themselves. (https://gitlab.gnome.org/GNOME/gnome-terminal/-/issues/7914#note_1523646)
Environment:
kind version): 0.14.0kubectl version): 1.24.3docker info):command not found😉podman version): 4.1.1/etc/os-release): Arch LinuxThe text was updated successfully, but these errors were encountered: