New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Podman hangs with deadlock on /var/lib/containers/storage/storage.lock #1146
Comments
Looks like a c/storage issue, probably around initialization. |
If you kill the podman inspect do things clear up? |
Sorry, I tore down the environment and worked around the issue by disabling concurrency in my script. I'll post again when I reproduce the lock. By the way, I've noticed I'm running quite an old version of podman. I've seen there are rpms for 0.6.4 around (I'm using 0.6.1 from centos-extras). Is version 0.7.x already packaged for centos? |
@lsm5 is a good source for an answer on that |
We release a podman every week 7==July 3 == third week. |
@lsm5 We really need to update the README on how to get the latest packaged version for each distro. |
I'm going to leave this open for other people to report potential locking issues, but it's hard to track these down unless we have more to go on. |
Reproduced fairly quickly (once - haven't tried again) with the attached script. Looks similar, but this time the deadlock is in
Login access available on request until, say, 4pm EDT. After that I'll kill these and start hammering again to see how reliable that reproducer is. (Speaking of which: much of the script is mumbo-jumbo incantations. It may be possible to reduce it some more; I've just been playing somewhat haphazardly). |
Looks like the same locks as last time |
@edsantiago managed to reproduce with an strace. Seems to be a race between one of the calls in the image library's |
The only call to c/storage in |
One side seems to be the overlay driver's |
I'm also having this bug when deploying ~30 containers on a busy VM. |
I've uploaded an sosreport if that something which can help: https://files.macchi.pro:8443/sosreport/ |
I have a self-contained script[1] which shows buildah hitting what I believe is the same lock problem. For this script, the build blocks usually on the FROM directive, sometimes on the COPY directive. While blocked, calls to "podman images" also blocks indefinitely. Killing the buildah processes unblocks other calls. You can tweak the concurrency variable to cause more or less stress. When I set concurrency=4 I see different race issues. For example sometimes an image doesn't get built, and the following is logged:
|
containers/storage#213 should fix |
I ran my buildah torture script with concurrency=16 (!) and can no longer reproduce my lockup. |
Awesome. |
This is only temporary until the cobra PR containers#1146 is merged. Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
This is only temporary until the cobra PR containers#1146 is merged. Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
This is only temporary until the cobra PR containers#1146 is merged. Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
This is only temporary until the cobra PR containers#1146 is merged. Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
This is only temporary until the cobra PR containers#1146 is merged. Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
This is only temporary until the cobra PR containers#1146 is merged. Signed-off-by: Paul Holzinger <paul.holzinger@web.de>
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
Description
Got podman in a locked state:
strace
shows it's waiting on the lock at "/var/lib/containers/storage/storage.lock":Steps to reproduce the issue:
Launch several containers at the same time
Wait for podman to hang
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version
:Output of
podman info
:I can't provide the output of
podman info
, it hangs.The text was updated successfully, but these errors were encountered: