Cannot enter container – crun: setrlimit RLIMIT_NPROC: Operation not permitted: OCI permission denied
#1312
Comments
|
Asked ChatGPT and got some debugging tips (that it may be about cgroups v2): Still does not help… 🤔 |
|
Also have this here: |
Also found the old bug #500 of me as I noticed there was something different and found #1260. |
|
What was the /cc @giuseppe |
|
I'm hitting this same problem with the exact same crun-1.8.5-1.fc38 |
|
So I just discovered that one of my toolboxes is broken while the other is fine. I have two images: The fedora-toolbox-38 is working without issue while the bst2 toolbox based on |
|
could you show the output of What are your current limits if you run |
No clue why only one container has these special limits set. I don't remember doing anything differently when I created them. |
|
There was a podman change a while back in fedora that caused containers to have higher limits than the system allowed it is probably the root cause for this, sorry don’t have time to link the issue |
Did you mean containers/podman#17681 ? |
|
I am hitting the same problem |
|
I hit same problem today. Had to reboot due to system being wedged. Rebooted, and my Silverblue experience significantly soured. I have used it for less than one week. My rlimits are: |
|
Just did an update/reboot, hoping for a fix. nada. |
|
The problem is that One can readily recreate this as follows: |
|
I have "recovered" from this issue, by using this. Eventually someone will fix this and it will make it into my |
|
Okay found downstream issue at Fedora Silverblue, as it seems to affect many. It is fedora-silverblue/issue-tracker#460 It says this issue has been reported upstream to podman and merged in containers/podman#18714 / containers/podman#18721, but that PR is kinda old, why does it still fail? Ah beacuse of the release cycles, v4.6.0 of podman seems to fix the bug (search for ulimit) and that seems to have been released just some days ago. |
|
Same problem here. My rlimits: It is actually second time I'm having this issue. |
|
soru is still getting this issue, even with podman v4.6.1 |
|
based on this: containers/podman#19634 (comment) I created a script to migrate old/broken toolboxes to new ones. This error will keep persisting on older containers regardless of having an up to date podman version because the limits were previously set in the container spec by older versions of podman. Newer versions don't have this behaviour but the only way to fix this is to recreate the container with a newer podman version, just upgrading won't solve it. Anyway, this is a script that will export, reimport and recreate new toolboxes based on the ones you have currently on your system: the manual steps are as follows: The new containers will be named with a suffix |
A hint so people do not need to waste time on searching: You can then simply rename the container like this: $ podman rename ${containerName}-new ${containerName}
$ podman rename fedora-toolbox-38-new fedora-toolbox-38 |
|
Though it somehow does not work: $ podman ps -a --format json \
| jq -r '.[] | select(.Labels."com.github.containers.toolbox").Names[0]' \
| xargs -tI{} \
bash -c 'podman export {} -o {}.tar.gz && podman import {}.tar.gz {}-image && toolbox create --image localhost/{}-image {}-new'
bash -c 'podman export fedora-toolbox-38 -o fedora-toolbox-38.tar.gz && podman import fedora-toolbox-38.tar.gz fedora-toolbox-38-image && toolbox create --image localhost/fedora-toolbox-38-image fedora-toolbox-38-new'
Getting image source signatures
Copying blob 2b3919c89420 done
Copying config 788d5882a5 done
Writing manifest to image destination
sha256:788d5882a50575f328a8b85f3f92a2ad488b4ffc18949a4bdc20742ef2736bf2
Created container: fedora-toolbox-38-new
Enter with: toolbox enter fedora-toolbox-38-new
$ podman rm fedora-toolbox-38
fedora-toolbox-38
$ podman rename fedora-toolbox-38-new fedora-toolbox-38
$ toolbox enter
Error: failed to initialize container fedora-toolbox-38
$ podman --version
podman version 4.6.0
$ podman inspect fedora-toolbox-38 --format "{{.HostConfig.Ulimits}}"
[{RLIMIT_NOFILE 524288 524288} {RLIMIT_NPROC 60983 60983}]And the issue has supposedly been fixed in v4.6.0 of podman, so why is it happening? $ podman ps -a --format json \
| jq -r '.[] | select(.Labels."com.github.containers.toolbox").Names[0]' \
| xargs -tI{} \
bash -c 'podman export {} -o {}.tar.gz && podman import {}.tar.gz {}-image && toolbox create --image localhost/{}-image {}-new'
bash -c 'podman export fedora-toolbox-38 -o fedora-toolbox-38.tar.gz && podman import fedora-toolbox-38.tar.gz fedora-toolbox-38-image && toolbox create --image localhost/fedora-toolbox-38-image fedora-toolbox-38-new'
Getting image source signatures
Copying blob 8a5a751b18cf done
Copying config aabcd2283e done
Writing manifest to image destination
sha256:aabcd2283e30f618f8d72d2b5363bc0d467698eebd8e3f2031452c164765ba9b
Created container: fedora-toolbox-38-new
Enter with: toolbox enter fedora-toolbox-38-new
$ podman inspect fedora-toolbox-38-new --format "{{.HostConfig.Ulimits}}"
[]
$ toolbox enter fedora-toolbox-38-new
Error: invalid entry point PID of container fedora-toolbox-38-new
$ podman inspect fedora-toolbox-38-new --format "{{.HostConfig.Ulimits}}"
[{RLIMIT_NOFILE 524288 524288} {RLIMIT_NPROC 60983 60983}]Though note the error |
|
The error relates to the entrypoint of the resulting image not being correct (mm toolbox is supposed to set itself as entrypoint, it is possibly expecting entrypoint as /bin/bash but it is toolbox as this image was already prepped by toolbox) quite unfortunate - this worked in my limited testing - you could confirm my theory by inspecting the entrypoint of the image |
|
Yeah after cloning and running |
|
Duplicate of containers/podman#19634 |
Describe the bug
A clear and concise description of what the bug is. If possible, re-run the command(s) with
--log-level debugand put the output here.Steps how to reproduce the behaviour
toolbox create)toolbox rm fedora-toolbox-36etc.)toolbox enteranymore.Expected behaviour
Can enter and not suddently, after rebooting, fail to enter my container… :(
Actual behaviour
Output of
toolbox --version(v0.0.90+)Toolbox package info (
rpm -q toolbox)Output of
podman versione.g.,
Podman package info (
rpm -q podman)Info about your OS
Fedora Silverblue 38
Additional context
I mean, you had one job, and this failed… I mean, no offense, I am just sad…
When I run podman manually:
Related issues
Search only turned up #1297, but I use no VirtualBox.
Alos tried containers/podman#14284 (comment) (again altghough I donÄt use VirtualBox), and it kinda works, but is still throws erros (and is no solution of course):
edit: forget the command line above, it starts a fresh container based on the
…:38image, so it's not my existing container instance…I have no idea what config files these are…
Also found this file, which seems to be created at reboot time(?):
The text was updated successfully, but these errors were encountered: