podman rm -a shows "No such file or directory" #2900
Comments
openshift-ci-robot
added
the
kind/bug
|
I've been seeing this one occasionally. The container is definitely removed, so this seems to be a cleanup error - the container is gone, but some resources might not be fully freed. It's an ENOENT, definitely. The curious thing is that we're not actually printing where the error happened. All of these cleanup errors should prefix what went wrong (the "No such file or directory" part) with the component that failed to clean up after itself - storage, networking, etc. Oh, wait... they only do that if there's more than one error... Alright, first step here is to get a patch in to improve debug information on cleanup error. |
|
I'm going to try and deliberately trigger this, see if i can get proper debug output now. I suspect it's related to freeing the lock. |
|
I've been seeing similar, where I'd assumed it was some sort of corruption, and we've been working around it by either deleting the podman DB and reinstalling, or by using |
|
The one thing I can almost guarantee is that the DB is in good shape, and
shouldn't require manual cleaning or deletion (though I'd be interested to
hear if you found otherwise exploring your database after seeing this). I'm
fairly certain this is an error from trying to clean up various container
resources, but it should be a nonfatal one (I'm fairly certain the
container is already gone from the database by the time it occurs). I've
merged a PR upstream to make the cause of the failures more clear, but
haven't been able to reproduce since said patch landed.
…On Wed, Apr 17, 2019, 04:40 Steven Hardy ***@***.***> wrote:
I've been seeing similar, where podman pod exists somepod returns zero,
then immediately after podman pod rm somepod fails with Error: no such
file or directory
I'd assumed it was some sort of corruption, and we've been working around
it by either deleting the podman DB and reinstalling, or by using go get
github.com/br0xen/boltbrowser then using that to manually clean up.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#2900 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHYHCBkVPUpzqwsqUfqD48GvwVQ-nRR3ks5vht4RgaJpZM4cpiQG>
.
|
|
I am blocked by the same issue. When this new version with improved debug output will be released? |
|
It is very strange. |
|
1.3 should be releasing on Monday, and hopefully available in the repos later this week. It will include this patch. For reference, after some further debugging, I'm fairly certain this is coming out of the lock allocation code, and it's not actually a "no such file or directory" - just a somewhat questionable reuse of the error code to indicate that a lock has already been released (the error code is my fault - I promise it made some degree of sense at the time). The lock error itself is harmless, though it may indicate that a reboot occurred without Podman detecting it, which can lead to somewhat undesirable effects (for example, container states being desynced). |
|
This command removes one container and shows the error. |
|
@mheon the error is preventing |
|
That's a separate bug in |
|
I just went and deleted all the references to the unremovable container with boltbrowser: |
|
@celebdor An unremovable container is very concerning - if it happens again, would you mind opening an issue here? Containers that manage to wedge themselves to the point that they can't be removed would be a top-priority bug for us. The original issue here ( |
|
Think I have a clue on what the original |
|
Confirmed - I've identified the root cause of the issue. Patching now. |
After a reboot, when we refresh Podman's state, we retrieved the lock from the fresh SHM instance, but we did not mark it as allocated to prevent it being handed out to other containers and pods. Provide a method for marking locks as in-use, and use it when we refresh Podman state after a reboot. Fixes containers#2900 Signed-off-by: Matthew Heon <matthew.heon@pm.me>
|
#3073 to fix |
After a reboot, when we refresh Podman's state, we retrieved the lock from the fresh SHM instance, but we did not mark it as allocated to prevent it being handed out to other containers and pods. Provide a method for marking locks as in-use, and use it when we refresh Podman state after a reboot. Fixes containers#2900 Signed-off-by: Matthew Heon <matthew.heon@pm.me>
After a reboot, when we refresh Podman's state, we retrieved the lock from the fresh SHM instance, but we did not mark it as allocated to prevent it being handed out to other containers and pods. Provide a method for marking locks as in-use, and use it when we refresh Podman state after a reboot. Fixes containers#2900 Signed-off-by: Matthew Heon <matthew.heon@pm.me>
|
@mheon just to clarify, the issues we're seeing are either pods or containers that cannot be removed (e.g the container/pod is not gone after rm runs, hence the DB hacking). An example which I just ran into now, immediately after a reboot: This is with podman-1.2-2.git3bd528e.el7.x86_64 so I'm wondering if this is likely to be fixed by the #3073 PR, or if there's another issue to flush out? Happy to build/test the latest locally but wanted to confirm if that's worthwhile effort before proceeding, thanks! |
github-actions
bot
added
the
locked - please file new issue/PR
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
When running podman rm -a, an error message: "No such file or directory is shown"
Steps to reproduce the issue:
Describe the results you received:
Describe the results you expected:
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:Output of
podman info --debug:Additional environment details (AWS, VirtualBox, physical, etc.):
The text was updated successfully, but these errors were encountered: