Fail on unmount/rmdir in overlay.Driver.Put #1607
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I have filed containers/podman#18634 to trigger a Podman test run at least, to see if there is something very obvious I’m missing. |
mtrmac
added a commit
to mtrmac/libpod
that referenced
this pull request
May 19, 2023
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
mtrmac
added a commit
to mtrmac/libpod
that referenced
this pull request
May 19, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 19, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
mtrmac
force-pushed
the
overlay-put-failures
branch
from
36c85d0
to
19fdd6e
Compare
mtrmac
added a commit
to mtrmac/libpod
that referenced
this pull request
May 19, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 19, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 19, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 19, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 20, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 20, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 20, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 20, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 21, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 22, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 22, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 22, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 22, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 22, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
instead of only logging a debug-level failure. We are seeing layers being mounted while our metadata claims that they are unmounted, and this seems to be a way how that could happen without leaving a visible trace. It's unclear to me why this silently succeeds; this behavior is very long-standing. Signed-off-by: Miloslav Trmač <mitr@redhat.com>
mtrmac
force-pushed
the
overlay-put-failures
branch
from
19fdd6e
to
da0df59
Compare
edsantiago
pushed a commit
to edsantiago/libpod
that referenced
this pull request
May 22, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
mtrmac
added a commit
to mtrmac/libpod
that referenced
this pull request
May 22, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
mtrmac
added a commit
to mtrmac/libpod
that referenced
this pull request
May 22, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
|
@giuseppe PTAL |
giuseppe
approved these changes
May 23, 2023
I don’t know that either, but I am seeing EINVAL on unmount in containers/podman#18634 . (At this point I have no idea whether that’s a long-standing bug to be fixed, a failure that should be ignored, or a short-term fallout of some unrelated bug.) |
mtrmac
added a commit
to mtrmac/libpod
that referenced
this pull request
May 23, 2023
Nerf the bloat check, this PR is testing-only Signed-off-by: Miloslav Trmač <mitr@redhat.com>
rhatdan
changed the title
|
LGTM |
Luap99
added a commit
to Luap99/libpod
that referenced
this pull request
Jun 9, 2023
A c/storage PR[1] chnage the behavior to correctly report umount errors. This is causing problem in the updgrade tests. The problem is that a cotnainer is mounted inside another container and then unmounted on the host. Therefore both operations happen in different mount namespaces. this is expcted but we want to share the mounts between them. This is the default but c/stroage make the root private by default thus the mounts were not shared. To fix this use the `skip_mount_home` storage option so the mount is kept shared. [1] containers/storage#1607 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99
added a commit
to Luap99/libpod
that referenced
this pull request
Jun 9, 2023
A c/storage PR[1] chnage the behavior to correctly report umount errors. This is causing problem in the updgrade tests. The problem is that a cotnainer is mounted inside another container and then unmounted on the host. Therefore both operations happen in different mount namespaces. this is expcted but we want to share the mounts between them. This is the default but c/stroage make the root private by default thus the mounts were not shared. To fix this use the `skip_mount_home` storage option so the mount is kept shared. [1] containers/storage#1607 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99
added a commit
to Luap99/libpod
that referenced
this pull request
Jun 12, 2023
A c/storage PR[1] chnage the behavior to correctly report umount errors. This is causing problem in the updgrade tests. The problem is that a cotnainer is mounted inside another container and then unmounted on the host. Therefore both operations happen in different mount namespaces. this is expcted but we want to share the mounts between them. This is the default but c/stroage make the root private by default thus the mounts were not shared. To fix this use the `skip_mount_home` storage option so the mount is kept shared. [1] containers/storage#1607 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
cgiradkar
pushed a commit
to cgiradkar/podman
that referenced
this pull request
Jul 12, 2023
A c/storage PR[1] chnage the behavior to correctly report umount errors. This is causing problem in the updgrade tests. The problem is that a cotnainer is mounted inside another container and then unmounted on the host. Therefore both operations happen in different mount namespaces. this is expcted but we want to share the mounts between them. This is the default but c/stroage make the root private by default thus the mounts were not shared. To fix this use the `skip_mount_home` storage option so the mount is kept shared. [1] containers/storage#1607 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
cgiradkar
pushed a commit
to cgiradkar/podman
that referenced
this pull request
Jul 13, 2023
A c/storage PR[1] chnage the behavior to correctly report umount errors. This is causing problem in the updgrade tests. The problem is that a cotnainer is mounted inside another container and then unmounted on the host. Therefore both operations happen in different mount namespaces. this is expcted but we want to share the mounts between them. This is the default but c/stroage make the root private by default thus the mounts were not shared. To fix this use the `skip_mount_home` storage option so the mount is kept shared. [1] containers/storage#1607 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
cgiradkar
pushed a commit
to cgiradkar/podman
that referenced
this pull request
Jul 17, 2023
A c/storage PR[1] chnage the behavior to correctly report umount errors. This is causing problem in the updgrade tests. The problem is that a cotnainer is mounted inside another container and then unmounted on the host. Therefore both operations happen in different mount namespaces. this is expcted but we want to share the mounts between them. This is the default but c/stroage make the root private by default thus the mounts were not shared. To fix this use the `skip_mount_home` storage option so the mount is kept shared. [1] containers/storage#1607 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
cgiradkar
pushed a commit
to cgiradkar/podman
that referenced
this pull request
Jul 17, 2023
A c/storage PR[1] chnage the behavior to correctly report umount errors. This is causing problem in the updgrade tests. The problem is that a cotnainer is mounted inside another container and then unmounted on the host. Therefore both operations happen in different mount namespaces. this is expcted but we want to share the mounts between them. This is the default but c/stroage make the root private by default thus the mounts were not shared. To fix this use the `skip_mount_home` storage option so the mount is kept shared. [1] containers/storage#1607 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
flouthoc
added a commit
to flouthoc/storage
that referenced
this pull request
Aug 14, 2023
It seems there are cases where underlying layers can be already unmounted in such case the layer directory is not removed and put fails early with `EINVAL` causing dangling directory for these layers. Following case may be a hidden underlying issue with `unmounting` of `c/stoage` which needs more investigation however prior discussion shows that this is a long standing issue in `storage` and which was never caught till containers#1607 Signed-off-by: Aditya R <arajan@redhat.com>
flouthoc
added a commit
to flouthoc/storage
that referenced
this pull request
Aug 14, 2023
It seems there are cases where underlying layers can be already unmounted in such case the layer directory is not removed and put fails early with `EINVAL` causing dangling directory for these layers. Following case may be a hidden underlying issue with `unmounting` of `c/stoage` which needs more investigation however prior discussion shows that this is a long standing issue in `storage` and which was never caught till containers#1607 Signed-off-by: Aditya R <arajan@redhat.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
… instead of only logging a debug-level failure.
We are seeing layers being mounted while our metadata claims that they are unmounted, and this seems to be a way how that could happen without leaving a visible trace.
It's unclear to me why this silently succeeds; this behavior is very long-standing.
Motivated by containers/podman#17216 , where we seem to be seeing mounted layers with zero count in
mount points.json.Debuglogs to be elevated to at least aWarning, possibly outrightError. @nalind @giuseppe PTAL.