misleading error during manifest list push

[ipanova]

(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ podman manifest push --all   report-bug2  pulp3-source-fedora34.fluffy.example.com/foo2:report-bug2 --format oci
Getting image list signatures
Copying 0 of 0 images in list
Writing manifest list to image destination
Error: Uploading manifest list failed, attempted the following formats: application/vnd.oci.image.index.v1+json(uploading manifest report-bug2 to pulp3-source-fedora34.fluffy.example.com/foo2: name unknown: Repository not found.)

Should be - invalid manifest

[lubosmj]

The error is raised when pushing an empty manifest list to the registry. According to the official documentation, manifest lists with the OCI index format may contain zero manifests (https://github.com/opencontainers/image-spec/blob/main/image-index.md#image-index-property-descriptions).

Podman sets the content header to application/vnd.oci.image.manifest.v1+json with an empty manifest list of the OCI index type. It correctly sets the header to application/vnd.docker.distribution.manifest.list.v2+json for v2s2 manifest lists.

In Pulp, we consider application/vnd.oci.image.manifest.v1+json to be a normal manifest and proceed to add that manifest to a repository.

[lubosmj]

I am not seeing the problem with podman v4.

Pulp Registry:

(pulp) [vagrant@pulp3-source-fedora35 ~]$ podman manifest push --all   report-bug  localhost:24817/foo:report-bug --tls-verify=false --format oci --log-level=debug
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman manifest push --all report-bug localhost:24817/foo:report-bug --tls-verify=false --format oci --log-level=debug) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/vagrant/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] systemd-logind: Unknown object '/'.          
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/vagrant/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/vagrant/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/vagrant/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 13             
DEBU[0000] Looking up image "report-bug" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Trying "localhost/report-bug:latest" ...     
DEBU[0000] parsed reference into "[overlay@/home/vagrant/.local/share/containers/storage+/run/user/1000/containers]@6183b9f7fb8243739d7125200822c25c494c405cd013d6c52e917d97a95ad052" 
DEBU[0000] Found image "report-bug" as "localhost/report-bug:latest" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] parsed reference into "[overlay@/home/vagrant/.local/share/containers/storage+/run/user/1000/containers]@6183b9f7fb8243739d7125200822c25c494c405cd013d6c52e917d97a95ad052" 
DEBU[0000] Found credentials for localhost:24817/foo in credential helper containers-auth.json in file /run/user/1000/containers/auth.json 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/containers/sigstore  
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/localhost:24817 
DEBU[0000] Using blob info cache at /home/vagrant/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] Source is a manifest list; copying all instances 
Getting image list signatures
DEBU[0000] Manifest list has MIME type application/vnd.docker.distribution.manifest.list.v2+json, ordered candidate list [application/vnd.oci.image.index.v1+json] 
DEBU[0000] ... will convert to application/vnd.oci.image.index.v1+json first, and then try [] 
Copying 0 of 0 images in list
Writing manifest list to image destination
DEBU[0000] Trying to use manifest list type application/vnd.oci.image.index.v1+json… 
DEBU[0000] Manifest list has been updated               
DEBU[0000] GET https://localhost:24817/v2/              
DEBU[0010] Ping https://localhost:24817/v2/ err Get "https://localhost:24817/v2/": net/http: TLS handshake timeout (&url.Error{Op:"Get", URL:"https://localhost:24817/v2/", Err:http.tlsHandshakeTimeoutError{}}) 
DEBU[0010] GET http://localhost:24817/v2/               
DEBU[0010] Ping http://localhost:24817/v2/ status 401   
DEBU[0010] GET https://pulp3-source-fedora35.localhost.example.com/token/?account=admin&scope=repository%3Afoo%3Apull%2Cpush&service=localhost%3A24817 
DEBU[0010] PUT http://localhost:24817/v2/foo/manifests/report-bug 
Storing list signatures
DEBU[0017] Called push.PersistentPostRunE(podman manifest push --all report-bug localhost:24817/foo:report-bug --tls-verify=false --format oci --log-level=debug) 
(pulp) [vagrant@pulp3-source-fedora35 ~]$ podman -v
podman version 4.1.1

Dockerhub Registry:

(pulp) [vagrant@pulp3-source-fedora35 ~]$ podman manifest push --all   report-bug  docker.io/lubosmj/foo:report-bug --tls-verify=false --format oci --log-level=debug
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman manifest push --all report-bug docker.io/lubosmj/foo:report-bug --tls-verify=false --format oci --log-level=debug) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/vagrant/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] systemd-logind: Unknown object '/'.          
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/vagrant/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/vagrant/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/vagrant/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 13             
DEBU[0000] Looking up image "report-bug" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Trying "localhost/report-bug:latest" ...     
DEBU[0000] parsed reference into "[overlay@/home/vagrant/.local/share/containers/storage+/run/user/1000/containers]@6183b9f7fb8243739d7125200822c25c494c405cd013d6c52e917d97a95ad052" 
DEBU[0000] Found image "report-bug" as "localhost/report-bug:latest" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] parsed reference into "[overlay@/home/vagrant/.local/share/containers/storage+/run/user/1000/containers]@6183b9f7fb8243739d7125200822c25c494c405cd013d6c52e917d97a95ad052" 
DEBU[0000] Found credentials for docker.io/lubosmj/foo in credential helper containers-auth.json in file /run/user/1000/containers/auth.json 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/containers/sigstore  
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] Using blob info cache at /home/vagrant/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] Source is a manifest list; copying all instances 
Getting image list signatures
DEBU[0000] Manifest list has MIME type application/vnd.docker.distribution.manifest.list.v2+json, ordered candidate list [application/vnd.oci.image.index.v1+json] 
DEBU[0000] ... will convert to application/vnd.oci.image.index.v1+json first, and then try [] 
Copying 0 of 0 images in list
Writing manifest list to image destination
DEBU[0000] Trying to use manifest list type application/vnd.oci.image.index.v1+json… 
DEBU[0000] Manifest list has been updated               
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://auth.docker.io/token?account=lubosmj&scope=repository%3Alubosmj%2Ffoo%3Apull%2Cpush&service=registry.docker.io 
DEBU[0000] PUT https://registry-1.docker.io/v2/lubosmj/foo/manifests/report-bug 
Storing list signatures
DEBU[0001] Called push.PersistentPostRunE(podman manifest push --all report-bug docker.io/lubosmj/foo:report-bug --tls-verify=false --format oci --log-level=debug) 
(pulp) [vagrant@pulp3-source-fedora35 ~]$ podman -v
podman version 4.1.1

[ipanova]

user is free to use whether podman 3 or 4 , we should properly raise correct error. Yes the media type is not correct which is being sent by podman3, in that case we should still raise manifest invalid and not repo not found error.