Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman images: panic #7444

Closed
edsantiago opened this issue Aug 25, 2020 · 3 comments
Closed

podman images: panic #7444

edsantiago opened this issue Aug 25, 2020 · 3 comments
Assignees
@vrothberg
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@edsantiago
Copy link
Collaborator

I don't suppose this one will be easily reproduced. I was infinite-looping a podman build / podman rmi for purposes of reproducing #7148. I ^Ced it (in the middle of build), ran podman images, and boom:

STEP 4: RUN /bin/echo hello
hello
^C
# ../bin/podman images
ERRO[0000] error reading image "11a9efdcc082f29e92296c1fe71d91d96bc198e7681d8a23955a1b93303fc079" as image: error locating item named "manifest" for image with ID "11a9efdcc082f29e92296c1fe71d91d96bc198e7681d8a23955a1b93303fc079": file does not exist
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe0 pc=0x12a96c2]

goroutine 1 [running]:
github.com/containers/podman/v2/libpod/image.areParentAndChild(0xc000189d40, 0x0, 0xc00038f350)
        libpod/image/image.go:1249 +0x32
github.com/containers/podman/v2/libpod/image.(*layerTree).children.func1(0xc0000bc480, 0x268e36cbaa7147e7, 0xc000189d40, 0x0)
        libpod/image/layer_tree.go:127 +0xae
github.com/containers/podman/v2/libpod/image.(*layerTree).children.func2(0xc000184340, 0x1ddfa00, 0x0, 0x0)
        libpod/image/layer_tree.go:135 +0xbd
github.com/containers/podman/v2/libpod/image.(*layerTree).children(0xc00049cb80, 0x1ddfa00, 0xc00038f350, 0xc0004fa900, 0x0, 0x0, 0x0, 0x0, 0x1da0900, 0xc000201000)
        libpod/image/layer_tree.go:153 +0x253
github.com/containers/podman/v2/libpod/image.(*Runtime).IntermediateFilter.func1(0xc0004fa900, 0x0)
        libpod/image/filters.go:43 +0x61
github.com/containers/podman/v2/libpod/image.FilterImages(0xc000446240, 0x4, 0x4, 0xc00051fa28, 0x1, 0x1, 0xc00000e7e0, 0x0, 0x0)
        libpod/image/filters.go:145 +0x77
github.com/containers/podman/v2/pkg/domain/infra/abi.(*ImageEngine).List(0xc000010e48, 0x1ddfa00, 0xc00038f350, 0xc00034da00, 0x2b24230, 0x0, 0x0, 0xc0000225a0, 0xc00051fc78, 0xc160c7, ...)
        pkg/domain/infra/abi/images_list.go:21 +0xb3b
github.com/containers/podman/v2/cmd/podman/images.images(0x2a6c9a0, 0x2b24230, 0x0, 0x0, 0x0, 0x0)
        cmd/podman/images/list.go:97 +0x110
github.com/spf13/cobra.(*Command).execute(0x2a6c9a0, 0xc00000e090, 0x0, 0x0, 0x2a6c9a0, 0xc00000e090)
        vendor/github.com/spf13/cobra/command.go:838 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0x2a7b0a0, 0xc00003e0b8, 0x186d800, 0x2b24230)
        vendor/github.com/spf13/cobra/command.go:943 +0x317
github.com/spf13/cobra.(*Command).Execute(...)
        vendor/github.com/spf13/cobra/command.go:883
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        vendor/github.com/spf13/cobra/command.go:876
main.Execute()
        cmd/podman/root.go:90 +0xec
main.main()
        cmd/podman/main.go:77 +0x18c

master @ 8fdc116 on f32 but podman-2.0.4-1.fc32.x86_64 barfs similarly:

# /usr/bin/podman images
Error: error reading image "11a9efdcc082f29e92296c1fe71d91d96bc198e7681d8a23955a1b93303fc079" as image: error locating item named "manifest" for image with ID "11a9efdcc082f29e92296c1fe71d91d96bc198e7681d8a23955a1b93303fc079": file does not exist

buildah runs fine:

# /path/to/hand-built/buildah images
REPOSITORY                     TAG      IMAGE ID       CREATED          SIZE
<none>                         <none>   11a9efdcc082   6 minutes ago    4.68 MB
<none>                         <none>   1f33a0733727   27 minutes ago   4.68 MB
quay.io/libpod/alpine_labels   latest   4fab981df737   22 months ago    4.69 MB

# .../buildah containers
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
c1fa5978538e     *     4fab981df737 quay.io/libpod/alpine_labels:... alpine_labels-working-container
e86ade9fecf7     *     3a53a5e5b703                                  3a53a5e5b70354647cba4d33d77d148a40332f350eacf7f27ce285d2968a7c5a-working-container
0b2b6691f7fd     *     1f33a0733727                                  1f33a07337275b9461ecabf2dc5ec8b4833f41a0fb31c5c441f3369f2515a9f8-working-container
82af2bad9767     *     4fab981df737 quay.io/libpod/alpine_labels:... alpine_labels-working-container-1
7c7e9beeb8ee     *     3a53a5e5b703                                  3a53a5e5b70354647cba4d33d77d148a40332f350eacf7f27ce285d2968a7c5a-working-container-1
b7a55b883c52     *     1f33a0733727                                  1f33a07337275b9461ecabf2dc5ec8b4833f41a0fb31c5c441f3369f2515a9f8-working-container-1

I will leave this system untouched for the next hour or so (until 14:30 EDT) in case anyone wants to log in and poke. Please LMK. Otherwise, I hope the stack trace is sufficient to debug.
baude added a commit to baude/podman that referenced this issue Aug 25, 2020
@baude
fix panic when checking len on nil object
f6ab67e 
issue containers#7444 describes a problem where an image does not have a manifest file and cannot be processed by our library correctly.  the origin of the panic is because we are checking the len of a nil object's attribute.  this is a temporary fix to protect from the panic in the future.  the origin of the problem is more interesting and requires more work when the code author returns from pto.

Signed-off-by: Brent Baude <bbaude@redhat.com>
@baude baude mentioned this issue Aug 25, 2020
Merged
@baude
Copy link
Member

baude commented Aug 25, 2020

@vrothberg you will have to chase this down and determine how you want to deal with it. i have put a simple nil check to stop the panic. it would probably pay to perform better error handling in libpod/image/filters.go:49 when you get an error on :43

baude added a commit to baude/podman that referenced this issue Aug 28, 2020
@baude
fix panic when checking len on nil object
a6f8586 
issue containers#7444 describes a problem where an image does not have a manifest file and cannot be processed by our library correctly.  the origin of the panic is because we are checking the len of a nil object's attribute.  this is a temporary fix to protect from the panic in the future.  the origin of the problem is more interesting and requires more work when the code author returns from pto.

Signed-off-by: Brent Baude <bbaude@redhat.com>
vrothberg added a commit to vrothberg/libpod that referenced this issue Sep 7, 2020
@vrothberg
make image parent check more robust
599370b 
Follow up on issue containers#7444 and make the parent checks more robust.
We can end up with an incoherent storage when, for instance, a
build has been killed.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
@vrothberg vrothberg mentioned this issue Sep 7, 2020
Merged
@vrothberg
Copy link
Member

@vrothberg you will have to chase this down and determine how you want to deal with it. i have put a simple nil check to stop the panic. it would probably pay to perform better error handling in libpod/image/filters.go:49 when you get an error on :43

I opened #7554 to follow up on the issue. I think that the nil checks are sufficient to address the issue. As the build has been killed, we may very well end up with some missing data / an incomplete image.

vrothberg added a commit to vrothberg/libpod that referenced this issue Sep 7, 2020
@vrothberg
make image parent check more robust
238abf6 
Follow up on issue containers#7444 and make the parent checks more robust.
We can end up with an incoherent storage when, for instance, a
build has been killed.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
@vrothberg
Copy link
Member

Closing as it's fixed in master. Will backport to 2.0 now.

vrothberg pushed a commit to vrothberg/libpod that referenced this issue Sep 8, 2020
@baude @vrothberg
make image parent check more robust
8f99530 
Follow up on issue containers#7444 and make the parent checks more robust.
We can end up with an incoherent storage when, for instance, a
build has been killed.

Backport of commit a6f8586 and commit 238abf6.  Squashed for easier
tracking and referencing.

Fixes: containers#7444
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1876576

Signed-off-by: Brent Baude <bbaude@redhat.com>
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
@vrothberg vrothberg mentioned this issue Sep 8, 2020
Merged
edsantiago pushed a commit to edsantiago/libpod that referenced this issue Sep 14, 2020
@baude @edsantiago
fix panic when checking len on nil object
e744079 
issue containers#7444 describes a problem where an image does not have a manifest file and cannot be processed by our library correctly.  the origin of the panic is because we are checking the len of a nil object's attribute.  this is a temporary fix to protect from the panic in the future.  the origin of the problem is more interesting and requires more work when the code author returns from pto.

Signed-off-by: Brent Baude <bbaude@redhat.com>
edsantiago pushed a commit to edsantiago/libpod that referenced this issue Sep 14, 2020
@vrothberg @edsantiago
make image parent check more robust
bfaed88 
Follow up on issue containers#7444 and make the parent checks more robust.
We can end up with an incoherent storage when, for instance, a
build has been killed.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
@edsantiago edsantiago mentioned this issue Oct 26, 2020
Closed
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@vrothberg vrothberg

Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@edsantiago @baude @vrothberg