"cgroup is not set: internal libpod error" when adding container to existing pod while rootless #10800
Comments
|
I'll take this one. Vague suspicion it is fixed in main with the patches from @cdoern to change cgroup parent for pods to the infra container's cgroup. |
|
@mheon, I can't seem to recreate this when running locally on my machine. output is as follows: |
|
You'll need to use |
this is running on my branch w/ the pod cgroup changes. Seems like it might be more of a miscommunication in the cgroup where the pod cgroup is set but like mentioned in irc, the other containers in the pod are in a different cgroup? not sure. |
|
OK, i've retried from start and it looks like system reboot changes something, not just pod create, start and stop. Right after create, and also after stop and start, it seems to work ok: It stops working after reboot: Don't know if it matters or not, but i'm logging into Alpine (which is running inside QEmu) system through SSH. Podman does not seem to create subgroup inside its cgroup: There is also BTW removing the pod also shows some error, but does in fact remove it: Let me know if i can provide some more info to help fix this problem :). |
|
No, I think we're set. I'm presently working on related changes to pod cgroups. I will tackle this once I am done with those. |
|
A friendly reminder that this issue had no activity for 30 days. |
|
+1 on this issue, our containers running rootless podman on Red Hat 8 stopped working once we updated from 3.0.2 to 3.2.3. Same error message as in the original port. The workaround mentioned above, adding |
|
Hello, I am using
after a machine restart. Output of |
|
Would it be possible to have a bit more info on what this error means/boils down to? I've seen it crop up before, and just had it appear now, but I think for an unrelated issue - but hard to diagnose as I don't really know what the internal issue is, without an intimate knowledge of cgroups. (@CardboardAgent I equally had an issue after a machine restart; my deployment is complex and via ansible and it seems that removing the pod and starting again is solving the issue. For whatever reason that specific container had been removed and needed recreating, but I could restart others no problem. I'm on Focal and Podman 3.3.0) |
the Even with that though, I am not able to reproduce locally. I think we can drop the check for an empty cgroup, not sure why we have it: Can anyone who is experiencing the issue try the following patch? $ git diff
diff --git a/libpod/runtime_ctr.go b/libpod/runtime_ctr.go
index 7d3891f6e..99fcc9053 100644
--- a/libpod/runtime_ctr.go
+++ b/libpod/runtime_ctr.go
@@ -343,9 +343,6 @@ func (r *Runtime) setupContainer(ctx context.Context, ctr *Container) (_ *Contai
if err != nil {
return nil, errors.Wrapf(err, "error retrieving pod %s cgroup", pod.ID())
}
- if podCgroup == "" {
- return nil, errors.Wrapf(define.ErrInternal, "pod %s cgroup is not set", pod.ID())
- }
canUseCgroup := !rootless.IsRootless() || isRootlessCgroupSet(podCgroup)
if canUseCgroup {
ctr.config.CgroupParent = podCgroup |
|
anyone had a chance to try the patch above? |
|
A friendly reminder that this issue had no activity for 30 days. |
|
I am assuming this is fixed, Closing. Reopen if I am mistaken. |
|
It's not fixed. |
|
A friendly reminder that this issue had no activity for 30 days. |
|
@giuseppe What is the state of this one? |
|
I am not able to reproduce locally, so I cannot verify my patch. I can open a PR with my patch, as I am not sure why we have that check in place anyway |
rootless containers do not use cgroups on cgroupv1 or if using cgroupfs, so drop the redundant check for an empty pod cgroup. Closes: containers#10800 Closes: https://bugzilla.redhat.com/show_bug.cgi?id=2028243 [NO NEW TESTS NEEDED] it requires rebooting and the rundir on a non tmpfs file system. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
rootless containers do not use cgroups on cgroupv1 or if using cgroupfs, so improve the check to account for such configuration. Closes: containers#10800 Closes: https://bugzilla.redhat.com/show_bug.cgi?id=2028243 [NO NEW TESTS NEEDED] it requires rebooting and the rundir on a non tmpfs file system. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
|
No, we still don't have support for setting resource limits, this just resolves a bug related to cgroups. We should really talk about this during planning and get it prioritized, though. |
|
Meet the same issue with wsl2 fedora without systemd , after restart wsl, |
the fix didn't get into a release yet. |
|
Ran into to this problem as well. Presumably after applying a number of updates that included podman. |
|
Hi everyone, I have the same problem. I already set the It works sometimes, but I get the same problem now. $ podman pod inspect alb
{
"Id": "21847f246e2322bd5603d1776530497e8d5eba12413f85681867468fc3262dbb",
"Name": "alb",
"Created": "2022-03-25T11:12:05.966783305+08:00",
"CreateCommand": [
"podman",
"pod",
"create",
"--cgroup-manager=cgroupfs",
"-p",
"3107:3306",
"-p",
"8080:8080",
"--name",
"alb"
],
"State": "Degraded",
"Hostname": "",
"CreateCgroup": true,
"CgroupParent": "/libpod_parent",
"CreateInfra": true,
"InfraContainerID": "40485cedc90045999f894413ecfa03005b651a5ed2475124be719d7a6689a4bf",
"InfraConfig": {
"PortBindings": {
"3306/tcp": [
{
"HostIp": "",
"HostPort": "3107"
}
],
"8080/tcp": [
{
"HostIp": "",
"HostPort": "8080"
}
]
},
"HostNetwork": true,
"StaticIP": "",
"StaticMAC": "",
"NoManageResolvConf": false,
"DNSServer": null,
"DNSSearch": null,
"DNSOption": null,
"NoManageHosts": false,
"HostAdd": null,
"Networks": null,
"NetworkOptions": null,
"pid_ns": "private",
"userns": "host"
},
"SharedNamespaces": [
"uts",
"ipc",
"net"
],
"NumContainers": 10,
"Containers": [...],
}$ podman run -d --name alb-nginx --cap-add CAP_NET_RAW --pod alb nginx
Error: pod 21847f246e2322bd5603d1776530497e8d5eba12413f85681867468fc3262dbb cgroup is not set: internal libpod error |
rootless containers do not use cgroups on cgroupv1 or if using cgroupfs, so improve the check to account for such configuration. Closes: containers#10800 Closes: https://bugzilla.redhat.com/show_bug.cgi?id=2028243 [NO NEW TESTS NEEDED] it requires rebooting and the rundir on a non tmpfs file system. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
rootless containers do not use cgroups on cgroupv1 or if using cgroupfs, so improve the check to account for such configuration. Closes: containers#10800 Closes: https://bugzilla.redhat.com/show_bug.cgi?id=2028243 [NO NEW TESTS NEEDED] it requires rebooting and the rundir on a non tmpfs file system. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
|
Hello I am getting this error on podman 3.4.7. $ rpm -q podman: $ podman pod inspect tomcat_db $ podman run --pod tomcat_db --name tomcat -d tomcat I restarted the podman service and even computer which did not help. |
|
This has been fixed upstream. Can you try a more recent Podman? |
In version 3.4.7 which I have, it already has been refactored somehow |
github-actions
bot
added
the
locked - please file new issue/PR
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
With podman v3.2.x, when running rootless on Alpine Linux (no systemd, cgroups switched to v2/"unified"), adding container to a existing pod that was stopped and started at least once, started showing errors like this:
Steps to reproduce the issue:
Create pod with some container.
Start the pod.
Stop the pod.
Try adding container (AFAIK does not matter if it's "temporary", i.e., with --rm, or not) to a pod:
podman run --pod testpod --rm docker.io/alpine sh -c "date"Describe the results you received:
Error information about cgroup:
Describe the results you expected:
No error, output returned and exit with status 0.
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
rpm -q podmanorapt list podman):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)
Yes
Additional environment details (AWS, VirtualBox, physical, etc.):
Podman is run on Alpine Linux within QEmu (KVM). There is no systemd there, only OpenRC. Cgroups switched from v1 to v2 (v2 only, not a "mix of both" type of thing).
The text was updated successfully, but these errors were encountered: