Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Interupting "podman run" while the image is pulled, leads to the error "Error: checking platform of image ... inspecting image: layer not known" the next time "podman run" is run #15853

Closed
eriksjolund opened this issue Sep 18, 2022 · 2 comments · Fixed by containers/common#1160
Closed

Interupting "podman run" while the image is pulled, leads to the error "Error: checking platform of image ... inspecting image: layer not known" the next time "podman run" is run #15853

eriksjolund opened this issue Sep 18, 2022 · 2 comments · Fixed by containers/common#1160
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@eriksjolund
Copy link
Contributor

eriksjolund commented Sep 18, 2022 •
edited

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Interupting podman run --rm -ti docker.io/library/nginx /bin/true with SIGINT while the image is being pulled, leads to the error

Error: checking platform of image 0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983: inspecting image: layer not known

the next time podman run --rm -ti docker.io/library/nginx /bin/true is run.

I've seen the error happening on the following systems:

operating system podman arch
fedora-coreos-38.20220917.91.0-qemu.aarch64.qcow2 with qemu 4.2.1 aarch64 (macbook pro)
fedora-coreos-37.20220910.1.0-qemu.aarch64.qcow2 with qemu 4.2.0 aarch64 (macbook pro)
fedora 36 4.2.0 x86_64

For Podman 4.2.0 (fedora-coreos-37.20220910.1.0-qemu.aarch64.qcow2) I once got another error:

Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:8460b172ee88c1cae796be6b03de46d6582e476cedc01afe5738684478de82d7": creating layer with ID "b146b9995a6331c26c39e066b8c4e2248c4dbe159ba84012ed4a25182986024e": Stat 
/var/home/testuser4/.local/share/containers/storage/overlay/1e4e5f955f2ee5f2dede918c140478399420295df93f62283201a9e4ef6c1ecd: no such file or directory

Steps to reproduce the issue:

  1. Boot up a rawhide Fedora CoreOS qcow2 image fedora-coreos-38.20220917.91.0-qemu.aarch64.qcow2
    and create a new user.
[core@localhost ~]$ rpm -q podman
podman-4.2.1-2.fc38.aarch64
[core@localhost ~]$ cat /etc/fedora-release 
Fedora release 38 (Rawhide)
[core@localhost ~]$ sudo -i
[root@localhost ~]# useradd testuser1
[root@localhost ~]# machinectl shell testuser1@
Connected to the local host. Press ^] three times within 1s to exit session.
  1. Run /usr/bin/timeout -s INT 10s podman --log-level=debug run --rm -ti docker.io/library/nginx /bin/true (the 10 second timeout was chosen so that the SIGINT is sent to Podman while Podman is pulling the image)
[testuser1@localhost ~]$ /usr/bin/timeout -s INT 10s podman --log-level=debug run --rm -ti docker.io/library/nginx /bin/true 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run --rm -ti docker.io/library/nginx /bin/true) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/testuser1/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] systemd-logind: Unknown object '/'.          
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/testuser1/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1001/containers     
DEBU[0000] Using static dir /var/home/testuser1/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1001/libpod/tmp      
DEBU[0000] Using volume path /var/home/testuser1/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
DEBU[0000] systemd-logind: Unknown object '/'.          
DEBU[0000] Invalid systemd user session for current user 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run --rm -ti docker.io/library/nginx /bin/true) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/testuser1/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] systemd-logind: Unknown object '/'.          
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/testuser1/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1001/containers     
DEBU[0000] Using static dir /var/home/testuser1/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1001/libpod/tmp      
DEBU[0000] Using volume path /var/home/testuser1/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: test mount with multiple lowers succeeded 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] overlay: test mount indicated that metacopy is not being used 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
DEBU[0000] Successfully loaded 1 networks               
DEBU[0000] Initialized SHM lock manager at path /libpod_rootless_lock_1001 
DEBU[0000] Podman detected system restart - performing state refresh 
INFO[0000] Setting parallel job count to 4              
DEBU[0000] Pulling image docker.io/library/nginx (policy: missing) 
DEBU[0000] Looking up image "docker.io/library/nginx" in local containers storage 
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux  [] } 
DEBU[0000] Trying "docker.io/library/nginx:latest" ...  
DEBU[0000] Trying "docker.io/library/nginx:latest" ...  
DEBU[0000] Trying "docker.io/library/nginx" ...         
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux  [] } 
DEBU[0000] Attempting to pull candidate docker.io/library/nginx:latest for docker.io/library/nginx 
DEBU[0000] parsed reference into "[overlay@/var/home/testuser1/.local/share/containers/storage+/run/user/1001/containers]docker.io/library/nginx:latest" 
Trying to pull docker.io/library/nginx:latest...
DEBU[0000] Copying source image //nginx:latest to destination image [overlay@/var/home/testuser1/.local/share/containers/storage+/run/user/1001/containers]docker.io/library/nginx:latest 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "docker.io/library/nginx:latest" 
DEBU[0000] No credentials matching docker.io/library/nginx found in /run/user/1001/containers/auth.json 
DEBU[0000] No credentials matching docker.io/library/nginx found in /var/home/testuser1/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io/library/nginx found in /var/home/testuser1/.docker/config.json 
DEBU[0000] No credentials matching docker.io/library/nginx found in /var/home/testuser1/.dockercfg 
DEBU[0000] No credentials for docker.io/library/nginx found 
DEBU[0000]  No signature storage configuration found for docker.io/library/nginx:latest, using built-in default file:///var/home/testuser1/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Fnginx%3Apull&service=registry.docker.io 
DEBU[0001] GET https://registry-1.docker.io/v2/library/nginx/manifests/latest 
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" 
DEBU[0001] Using blob info cache at /var/home/testuser1/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0001] Source is a manifest list; copying (only) instance sha256:7bf51d0871af4c4f688515562aac4ed424e3841ef6e33caef11c17232f958f78 for current system 
DEBU[0001] GET https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7bf51d0871af4c4f688515562aac4ed424e3841ef6e33caef11c17232f958f78 
DEBU[0002] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" 
DEBU[0002] IsRunningImageAllowed for image docker:docker.io/library/nginx:latest 
DEBU[0002]  Using default policy section                
DEBU[0002]  Requirement 0: allowed                      
DEBU[0002] Overall: allowed                             
DEBU[0002] Downloading /v2/library/nginx/blobs/sha256:0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983 
DEBU[0002] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983 
Getting image source signatures
DEBU[0003] Reading /var/home/testuser1/.local/share/containers/sigstore/library/nginx@sha256=7bf51d0871af4c4f688515562aac4ed424e3841ef6e33caef11c17232f958f78/signature-1 
DEBU[0003] Not looking for sigstore attachments: disabled by configuration 
DEBU[0003] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0003] ... will first try using the original manifest unmodified 
DEBU[0003] Checking if we can reuse blob sha256:8460b172ee88c1cae796be6b03de46d6582e476cedc01afe5738684478de82d7: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Checking if we can reuse blob sha256:3d898485473e3507374cea2e09f019c2ff5728f0911aa36c70b7a7235e9bc8ac: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Checking if we can reuse blob sha256:43509f6ae4b39a1fb2082ff97c7f493fff2c11cd9794539e758831e0247563c0: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0003] Checking if we can reuse blob sha256:a73e5a7988e9f087715d8e5378600a1579e7773ec7afc16275a34f11f4b038d0: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0003] Checking if we can reuse blob sha256:3f45c0a5377fb0022f94f1dd07b4167b39a6e80d603702100dcea8ec624511fb: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Checking if we can reuse blob sha256:f1cdcf23708a52dd51337c421723108b592b8d2c81d3354037f81d49b2c47ee2: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0003] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0003] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0003] Downloading /v2/library/nginx/blobs/sha256:43509f6ae4b39a1fb2082ff97c7f493fff2c11cd9794539e758831e0247563c0 
DEBU[0003] Downloading /v2/library/nginx/blobs/sha256:3d898485473e3507374cea2e09f019c2ff5728f0911aa36c70b7a7235e9bc8ac 
DEBU[0003] Downloading /v2/library/nginx/blobs/sha256:a73e5a7988e9f087715d8e5378600a1579e7773ec7afc16275a34f11f4b038d0 
DEBU[0003] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:43509f6ae4b39a1fb2082ff97c7f493fff2c11cd9794539e758831e0247563c0 
DEBU[0003] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:3d898485473e3507374cea2e09f019c2ff5728f0911aa36c70b7a7235e9bc8ac 
DEBU[0003] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0003] Downloading /v2/library/nginx/blobs/sha256:8460b172ee88c1cae796be6b03de46d6582e476cedc01afe5738684478de82d7 
DEBU[0003] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:8460b172ee88c1cae796be6b03de46d6582e476cedc01afe5738684478de82d7 
DEBU[0003] Downloading /v2/library/nginx/blobs/sha256:3f45c0a5377fb0022f94f1dd07b4167b39a6e80d603702100dcea8ec624511fb 
DEBU[0003] Downloading /v2/library/nginx/blobs/sha256:f1cdcf23708a52dd51337c421723108b592b8d2c81d3354037f81d49b2c47ee2 
DEBU[0003] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:3f45c0a5377fb0022f94f1dd07b4167b39a6e80d603702100dcea8ec624511fb 
DEBU[0003] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:f1cdcf23708a52dd51337c421723108b592b8d2c81d3354037f81d49b2c47ee2 
DEBU[0003] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:a73e5a7988e9f087715d8e5378600a1579e7773ec7afc16275a34f11f4b038d0 
Copying blob 43509f6ae4b3 [--------------------------------------] 0.0b / 665.0b
Copying blob a73e5a7988e9 [--------------------------------------] 0.0b / 895.0b
Copying blob 8460b172ee88 [--------------------------------------] 0.0b / 1.4KiB
Copying blob 3d898485473e [--------------------------------------] 0.0b / 28.7MiB
Copying blob f1cdcf23708a [--------------------------------------] 0.0b / 599.0b
Copying blob 3f45c0a5377f [--------------------------------------] 0.0b / 24.1MiB
DEBU[0003] Detected compression format gzip             
DEBU[0003] Using original blob without modification     
DEBU[0003] Detected compression format gzip             
DEBU[0003] Using original blob without modification     
DEBU[0003] Detected compression format gzip             
DEBU[0003] Using original blob without modification     
Copying blob 43509f6ae4b3 done  
Copying blob 43509f6ae4b3 done  
Copying blob 43509f6ae4b3 done  
Copying blob a73e5a7988e9 done  
Copying blob 8460b172ee88 done  
Copying blob 3d898485473e done  
Copying blob f1cdcf23708a done  
Copying blob 3f45c0a5377f done  
INFO[0009] Received shutdown signal "interrupt", terminating!  PID=1817
INFO[0009] Received shutdown signal "interrupt", terminating!  PID=1830
INFO[0009] Invoking shutdown handler "libpod"            PID=1830
INFO[0009] Invoking shutdown handler "libpod"            PID=1817
[testuser1@localhost ~]$
  1. Check the exit value
[testuser1@localhost ~]$ echo $?
124
  1. Run podman --log-level=debug run --rm -ti docker.io/library/nginx /bin/true.
[testuser1@localhost ~]$ podman --log-level=debug run --rm -ti docker.io/library/nginx /bin/true
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run --rm -ti docker.io/library/nginx /bin/true) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/testuser1/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] systemd-logind: Unknown object '/'.          
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/testuser1/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1001/containers     
DEBU[0000] Using static dir /var/home/testuser1/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1001/libpod/tmp      
DEBU[0000] Using volume path /var/home/testuser1/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 4              
DEBU[0000] Pulling image docker.io/library/nginx (policy: missing) 
DEBU[0000] Looking up image "docker.io/library/nginx" in local containers storage 
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux  [] } 
DEBU[0000] Trying "docker.io/library/nginx:latest" ...  
DEBU[0000] Trying "docker.io/library/nginx:latest" ...  
DEBU[0000] Trying "docker.io/library/nginx" ...         
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux  [] } 
DEBU[0000] Attempting to pull candidate docker.io/library/nginx:latest for docker.io/library/nginx 
DEBU[0000] parsed reference into "[overlay@/var/home/testuser1/.local/share/containers/storage+/run/user/1001/containers]docker.io/library/nginx:latest" 
Trying to pull docker.io/library/nginx:latest...
DEBU[0000] Copying source image //nginx:latest to destination image [overlay@/var/home/testuser1/.local/share/containers/storage+/run/user/1001/containers]docker.io/library/nginx:latest 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "docker.io/library/nginx:latest" 
DEBU[0000] No credentials matching docker.io/library/nginx found in /run/user/1001/containers/auth.json 
DEBU[0000] No credentials matching docker.io/library/nginx found in /var/home/testuser1/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io/library/nginx found in /var/home/testuser1/.docker/config.json 
DEBU[0000] No credentials matching docker.io/library/nginx found in /var/home/testuser1/.dockercfg 
DEBU[0000] No credentials for docker.io/library/nginx found 
DEBU[0000]  No signature storage configuration found for docker.io/library/nginx:latest, using built-in default file:///var/home/testuser1/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Fnginx%3Apull&service=registry.docker.io 
DEBU[0001] GET https://registry-1.docker.io/v2/library/nginx/manifests/latest 
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" 
DEBU[0001] Using blob info cache at /var/home/testuser1/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0001] Source is a manifest list; copying (only) instance sha256:7bf51d0871af4c4f688515562aac4ed424e3841ef6e33caef11c17232f958f78 for current system 
DEBU[0001] GET https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7bf51d0871af4c4f688515562aac4ed424e3841ef6e33caef11c17232f958f78 
DEBU[0002] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" 
DEBU[0002] IsRunningImageAllowed for image docker:docker.io/library/nginx:latest 
DEBU[0002]  Using default policy section                
DEBU[0002]  Requirement 0: allowed                      
DEBU[0002] Overall: allowed                             
DEBU[0002] Downloading /v2/library/nginx/blobs/sha256:0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983 
DEBU[0002] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983 
Getting image source signatures
DEBU[0002] Reading /var/home/testuser1/.local/share/containers/sigstore/library/nginx@sha256=7bf51d0871af4c4f688515562aac4ed424e3841ef6e33caef11c17232f958f78/signature-1 
DEBU[0002] Not looking for sigstore attachments: disabled by configuration 
DEBU[0002] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0002] ... will first try using the original manifest unmodified 
DEBU[0002] Checking if we can reuse blob sha256:8460b172ee88c1cae796be6b03de46d6582e476cedc01afe5738684478de82d7: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0002] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0002] Checking if we can reuse blob sha256:43509f6ae4b39a1fb2082ff97c7f493fff2c11cd9794539e758831e0247563c0: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0002] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0002] Checking if we can reuse blob sha256:a73e5a7988e9f087715d8e5378600a1579e7773ec7afc16275a34f11f4b038d0: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0002] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0002] Checking if we can reuse blob sha256:3d898485473e3507374cea2e09f019c2ff5728f0911aa36c70b7a7235e9bc8ac: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0002] Checking if we can reuse blob sha256:3f45c0a5377fb0022f94f1dd07b4167b39a6e80d603702100dcea8ec624511fb: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0002] Checking if we can reuse blob sha256:f1cdcf23708a52dd51337c421723108b592b8d2c81d3354037f81d49b2c47ee2: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0002] Downloading /v2/library/nginx/blobs/sha256:8460b172ee88c1cae796be6b03de46d6582e476cedc01afe5738684478de82d7 
DEBU[0002] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0002] Downloading /v2/library/nginx/blobs/sha256:f1cdcf23708a52dd51337c421723108b592b8d2c81d3354037f81d49b2c47ee2 
DEBU[0002] Downloading /v2/library/nginx/blobs/sha256:43509f6ae4b39a1fb2082ff97c7f493fff2c11cd9794539e758831e0247563c0 
DEBU[0002] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:f1cdcf23708a52dd51337c421723108b592b8d2c81d3354037f81d49b2c47ee2 
DEBU[0002] Downloading /v2/library/nginx/blobs/sha256:a73e5a7988e9f087715d8e5378600a1579e7773ec7afc16275a34f11f4b038d0 
DEBU[0002] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:a73e5a7988e9f087715d8e5378600a1579e7773ec7afc16275a34f11f4b038d0 
DEBU[0002] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0002] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0002] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:8460b172ee88c1cae796be6b03de46d6582e476cedc01afe5738684478de82d7 
DEBU[0002] Downloading /v2/library/nginx/blobs/sha256:3d898485473e3507374cea2e09f019c2ff5728f0911aa36c70b7a7235e9bc8ac 
DEBU[0002] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:3d898485473e3507374cea2e09f019c2ff5728f0911aa36c70b7a7235e9bc8ac 
DEBU[0002] Downloading /v2/library/nginx/blobs/sha256:3f45c0a5377fb0022f94f1dd07b4167b39a6e80d603702100dcea8ec624511fb 
DEBU[0002] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:43509f6ae4b39a1fb2082ff97c7f493fff2c11cd9794539e758831e0247563c0 
DEBU[0002] GET https://registry-1.docker.io/v2/library/nginx/blobs/sha256:3f45c0a5377fb0022f94f1dd07b4167b39a6e80d603702100dcea8ec624511fb 
Copying blob 8460b172ee88 [--------------------------------------] 0.0b / 1.4KiB
Copying blob 43509f6ae4b3 [--------------------------------------] 0.0b / 665.0b
Copying blob 8460b172ee88 [--------------------------------------] 0.0b / 1.4KiB
Copying blob 43509f6ae4b3 [--------------------------------------] 0.0b / 665.0b
Copying blob a73e5a7988e9 [--------------------------------------] 0.0b / 895.0b
Copying blob f1cdcf23708a done  
Copying blob 3d898485473e [--------------------------------------] 0.0b / 28.7MiB
Copying blob 3f45c0a5377f [--------------------------------------] 0.0b / 24.1MiB
DEBU[0003] Detected compression format gzip             
DEBU[0003] Using original blob without modification     
Copying blob 8460b172ee88 [--------------------------------------] 0.0b / 1.4KiB
Copying blob 43509f6ae4b3 done  
Copying blob 8460b172ee88 done  
Copying blob 43509f6ae4b3 done  
Copying blob a73e5a7988e9 done  
Copying blob 8460b172ee88 done  
Copying blob 8460b172ee88 done  
Copying blob 8460b172ee88 done  
Copying blob 8460b172ee88 done  
Copying blob 8460b172ee88 done  
Copying blob 43509f6ae4b3 done  
Copying blob a73e5a7988e9 done  
Copying blob f1cdcf23708a done  
Copying blob 3d898485473e done  
Copying blob 3f45c0a5377f done  
DEBU[0011] No compression detected                      
DEBU[0011] Compression change for blob sha256:0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983 ("application/vnd.docker.container.image.v1+json") not supported 
DEBU[0011] Using original blob without modification     
Copying config 0c404972e1 done  
Writing manifest to image destination
Storing signatures
DEBU[0011] setting image creation date to 2022-09-13 14:31:48.297458941 +0000 UTC 
WARN[0011] Found incomplete layer "ecb31a0505792ef54b0b6df8a1845e0260878fd2a736a4894e94df779d2cbb5a", deleting it 
DEBU[0011] created new image ID "0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983" 
DEBU[0011] saved image metadata "{\"signatures-sizes\":{\"sha256:7bf51d0871af4c4f688515562aac4ed424e3841ef6e33caef11c17232f958f78\":[]}}" 
DEBU[0011] added name "docker.io/library/nginx:latest" to image "0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983" 
DEBU[0011] Pulled candidate docker.io/library/nginx:latest successfully 
DEBU[0011] Looking up image "0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983" in local containers storage 
DEBU[0011] Trying "0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983" ... 
DEBU[0011] parsed reference into "[overlay@/var/home/testuser1/.local/share/containers/storage+/run/user/1001/containers]@0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983" 
DEBU[0011] Found image "0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983" as "0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983" in local containers storage 
DEBU[0011] Found image "0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983" as "0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983" in local containers storage ([overlay@/var/home/testuser1/.local/share/containers/storage+/run/user/1001/containers]@0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983) 
Error: checking platform of image 0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983: inspecting image: layer not known
[testuser1@localhost ~]$
  1. Run echo $?
[testuser1@localhost ~]$ echo $?
125
  1. Run podman images
[testuser1@localhost ~]$ podman images
Error: error retrieving label for image "0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983": you may need to remove the image to resolve the error: layer not known
[testuser1@localhost ~]$

Describe the results you received:

At step 4, i.e., the second time podman run ... is run, I see the error message

Error: checking platform of image 0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983: inspecting image: layer not known

The exit value is 125.

Describe the results you expected:

I expected not seeing the error message and I expected the exit value to be 0.

(Although the first podman run ... command was interupted, I thought the second podman run ...command would be able to run successfully)

Output of podman version:

Client:       Podman Engine
Version:      4.2.1
API Version:  4.2.1
Go Version:   go1.19.1
Built:        Wed Sep  7 19:57:08 2022
OS/Arch:      linux/arm64

Output of podman info:

host:
  arch: arm64
  buildahVersion: 1.27.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.2-3.fc37.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.2, commit: '
  cpuUtilization:
    idlePercent: 97.3
    systemPercent: 1.17
    userPercent: 1.53
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.0.0-0.rc5.37.fc38.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1397706752
  memTotal: 2051166208
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.6-2.fc38.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.6
      commit: 18cf2efbb8feb2b2f20e316520e0fd0b6c41ef4d
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-5.fc37.aarch64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 29m 27.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/testuser1/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/testuser1/.local/share/containers/storage
  graphRootAllocated: 10188992512
  graphRootUsed: 1916112896
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1001/containers
  volumePath: /var/home/testuser1/.local/share/containers/storage/volumes
version:
  APIVersion: 4.2.1
  Built: 1662580628
  BuiltTime: Wed Sep  7 19:57:08 2022
  GitCommit: ""
  GoVersion: go1.19.1
  Os: linux
  OsArch: linux/arm64
  Version: 4.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.2.1-2.fc38.aarch64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Although the bug reproducer above was done with Podman
4.2.1, here are some tests I did with Podman 4.2.0:

Tests on fedora-coreos-37.20220910.1.0-qemu.aarch64.qcow2

This Fedora CoreOS VM has Podman 4.2.0.

Test 1 (fedora-coreos-37.20220910.1.0-qemu.aarch64.qcow2)

Here I got a new error message:

Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:8460b172ee88c1cae796be6b03de46d6582e476cedc01afe5738684478de82d7": creating layer with ID "b146b9995a6331c26c39e066b8c4e2248c4dbe159ba84012ed4a25182986024e": Stat 
/var/home/testuser4/.local/share/containers/storage/overlay/1e4e5f955f2ee5f2dede918c140478399420295df93f62283201a9e4ef6c1ecd: no such file or directory

See the details:

(SIGINT was sent by pressing Ctrl-C)

[testuser4@localhost ~]$ podman run --rm -ti docker.io/library/nginx -- jq --help
Trying to pull docker.io/library/nginx:latest...
Getting image source signatures
Copying blob 8460b172ee88 done
Copying blob 3d898485473e done
Copying blob 3f45c0a5377f done
Copying blob f1cdcf23708a done
Copying blob a73e5a7988e9 done
Copying blob 43509f6ae4b3 done
^C[testuser4@localhost ~]$ ^C
[testuser4@localhost ~]$ ^C
[testuser4@localhost ~]$ podman run --rm -ti docker.io/library/nginx -- /usr/bin/jq --help
Trying to pull docker.io/library/nginx:latest...
Getting image source signatures
Copying blob 3d898485473e skipped: already exists
Copying blob 3f45c0a5377f skipped: already exists
Copying blob f1cdcf23708a skipped: already exists
Copying blob a73e5a7988e9 skipped: already exists
Copying blob 8460b172ee88 done
Copying blob 43509f6ae4b3 done
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:8460b172ee88c1cae796be6b03de46d6582e476cedc01afe5738684478de82d7": creating layer with ID "b146b9995a6331c26c39e066b8c4e2248c4dbe159ba84012ed4a25182986024e": Stat /var/home/testuser4/.local/share/containers/storage/overlay/1e4e5f955f2ee5f2dede918c140478399420295df93f62283201a9e4ef6c1ecd: no such file or directory
[testuser4@localhost ~]$
[testuser4@localhost ~]$ podman images
REPOSITORY  TAG         IMAGE ID    CREATED     SIZE
WARN[0000] Found incomplete layer "1e4e5f955f2ee5f2dede918c140478399420295df93f62283201a9e4ef6c1ecd", deleting it
[testuser4@localhost ~]$ podman images
REPOSITORY  TAG         IMAGE ID    CREATED     SIZE
[testuser4@localhost ~]$

Test 2 (fedora-coreos-37.20220910.1.0-qemu.aarch64.qcow2)

This time I got the same error as with Fedora CoreOS rawhide:

Error: checking platform of image 0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983: inspecting image: layer not known

See the details:

[testuser5@localhost ~]$ /usr/bin/timeout -s INT 10s podman run --rm -ti docker.io/library/nginx /bin/true ; podman run --rm -ti docker.io/library/nginx /bin/true
Trying to pull docker.io/library/nginx:latest...
Getting image source signatures
Copying blob 8460b172ee88 done
Copying blob 3d898485473e done
Copying blob 3f45c0a5377f done
Copying blob f1cdcf23708a done
Copying blob a73e5a7988e9 done
Copying blob 43509f6ae4b3 done
Trying to pull docker.io/library/nginx:latest...
Getting image source signatures
Copying blob 8460b172ee88 done
Copying blob 3d898485473e done
Copying blob 3f45c0a5377f done
Copying blob f1cdcf23708a done
Copying blob a73e5a7988e9 done
Copying blob 43509f6ae4b3 done
Copying config 0c404972e1 done
Writing manifest to image destination
Storing signatures
WARN[0011] Found incomplete layer "ecb31a0505792ef54b0b6df8a1845e0260878fd2a736a4894e94df779d2cbb5a", deleting it
Error: checking platform of image 0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983: inspecting image: layer not known
[testuser5@localhost ~]$ echo $?
125
[testuser5@localhost ~]$ podman image rm docker.io/library/nginx
WARN[0000] Failed to determine if an image is a parent: layer not known, ignoring the error
WARN[0000] Failed to determine parent of image: layer not known, ignoring the error
Untagged: docker.io/library/nginx:latest
Deleted: 0c404972e13056a866875f2bf8a981a911dc17071f505b9dc72cdf08e0d40983
[testuser5@localhost ~]$ podman images
REPOSITORY  TAG         IMAGE ID    CREATED     SIZE
[testuser5@localhost ~]$
@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Sep 18, 2022
@eriksjolund eriksjolund changed the title Interupting "podman run" while pulling the image leads to the error "inspecting image: layer not known" the next time "podman run" is run Interupting "podman run" while pulling the image leads to the error "Error: checking platform of image ... inspecting image: layer not known" the next time "podman run" is run Sep 18, 2022
@eriksjolund eriksjolund changed the title Interupting "podman run" while pulling the image leads to the error "Error: checking platform of image ... inspecting image: layer not known" the next time "podman run" is run Interupting "podman run" while the image is pulled, leads to the error "Error: checking platform of image ... inspecting image: layer not known" the next time "podman run" is run Sep 18, 2022
@mheon
Copy link
Member

mheon commented Sep 19, 2022

@vrothberg PTAL

@vrothberg
Copy link
Member

Thanks for reaching out, @eriksjolund! I'll take a look.

vrothberg added a commit to vrothberg/common that referenced this issue Sep 20, 2022
@vrothberg
libimage: platform: check if image is corrupted
00e1dc5 
When checking the platform of an image, take into account that it may be
corrupted.

Partially-fixes: containers/podman/issues/15853
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
@vrothberg vrothberg mentioned this issue Sep 20, 2022
Merged
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 15, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

libimage: platform: check if image is corrupted vrothberg/common
3 participants
@eriksjolund @mheon @vrothberg