Listing images via Docker-compatible REST API returns 500 on Fedora 34

[m-kn]

Is this a BUG REPORT or FEATURE REQUEST?

/kind bug

Description

I'm using testcontainers to start a localstack container to run some integration tests which involve some AWS services. afaik testcontainers doesn't intend to support podman, so a fallback to the Docker-compatible API is my only option (short of using docker). The podman + testcontainer setup used to work with (iirc) up to podman version 3.1.x, but with newer versions of podman I get 500s, when testcontainer tries to do some setup stuff.

Steps to reproduce the issue:

1. start the Docker-compatible API

   podman system service --time=0
   

2. try to list images by executing

   curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.32/images/json
   

   or

   curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.40/images/json
   

   (i.e. https://docs.docker.com/engine/api/v1.40/#operation/ImageList)
   The API version doesn't seem to matter, I get the same errors regardless, but the testcontainers version (1.16.0) I used to debug this ended up using 1.32.

Describe the results you received:
The request ends up with a 500

{
    "cause": "no image found in manifest list for architecture amd64, variant \"\", OS linux",
    "message": "Failed transform image summaries: failed to obtain summary for image 3a158f2977d8c194f16569c80f821c0829efd6067df29bbfaa5c02d4a6956684: choosing image instance: no image found in manifest list for architecture amd64, variant \"\", OS linux",
    "response": 500
}

service log doesn't contain any additional information

...
DEBU[0067] exporting opaque data as blob "sha256:1fd8e1b0bb7ef6cc63e0094e2fb5a35259bdce7041ef5e7e4c99694ded1041d7" 
DEBU[0067] exporting opaque data as blob "sha256:1fd8e1b0bb7ef6cc63e0094e2fb5a35259bdce7041ef5e7e4c99694ded1041d7" 
DEBU[0067] exporting opaque data as blob "sha256:1fd8e1b0bb7ef6cc63e0094e2fb5a35259bdce7041ef5e7e4c99694ded1041d7" 
DEBU[0067] exporting opaque data as blob "sha256:1fd8e1b0bb7ef6cc63e0094e2fb5a35259bdce7041ef5e7e4c99694ded1041d7" 
DEBU[0067] Inspecting image fc1c3f6f641b1eb5a8efe2f18a522d1e3ef07844a12d4aaf2a78f45ff7f350ad 
DEBU[0067] parsed reference into "[overlay@/home/mkn/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@fc1c3f6f641b1eb5a8efe2f18a522d1e3ef07844a12d4aaf2a78f45ff7f350ad" 
DEBU[0067] exporting opaque data as blob "sha256:fc1c3f6f641b1eb5a8efe2f18a522d1e3ef07844a12d4aaf2a78f45ff7f350ad" 
DEBU[0067] exporting opaque data as blob "sha256:fc1c3f6f641b1eb5a8efe2f18a522d1e3ef07844a12d4aaf2a78f45ff7f350ad" 
DEBU[0067] exporting opaque data as blob "sha256:fc1c3f6f641b1eb5a8efe2f18a522d1e3ef07844a12d4aaf2a78f45ff7f350ad" 
DEBU[0067] exporting opaque data as blob "sha256:fc1c3f6f641b1eb5a8efe2f18a522d1e3ef07844a12d4aaf2a78f45ff7f350ad" 
DEBU[0067] Inspecting image 3a158f2977d8c194f16569c80f821c0829efd6067df29bbfaa5c02d4a6956684 
DEBU[0067] parsed reference into "[overlay@/home/mkn/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@3a158f2977d8c194f16569c80f821c0829efd6067df29bbfaa5c02d4a6956684" 
INFO[0067] Request Failed(Internal Server Error): Failed transform image summaries: failed to obtain summary for image 3a158f2977d8c194f16569c80f821c0829efd6067df29bbfaa5c02d4a6956684: choosing image instance: no image found in manifest list for architecture amd64, variant "", OS linux 
@ - - [30/Nov/2021:14:56:43 +0100] "GET /v1.40/images/json HTTP/1.1" 500 357 "" "curl/7.76.1"

the variant \"\" / variant "" looks kinda wrong, but I have no clue what it's supposed to be, how I set it or why it would even matter.

Describe the results you expected:
List of images in JSON, e.g.

[
    ...
    },
    {
        "Containers": -1,
        "Created": 1630084785,
        "Id": "sha256:14119a10abf4669e8cdbdff324a9f9605d99697215a0d21c360fe8dfa8471bab",
        "Labels": null,
        "ParentId": "",
        "RepoDigests": [
            "alpine@sha256:e1c082e3d3c45cccac829840a25941e679c25d438cc8412c2fa221cf1a824e6a"
        ],
        "RepoTags": [
            "alpine:3.14"
        ],
        "SharedSize": -1,
        "Size": 5595292,
        "VirtualSize": 5595292
    },
    {
    ...
]

Additional information you deem important (e.g. issue happens only occasionally):
Happens every time I try to run a test that uses testcontainers and because of that breaks my local test setup. 😢

Output of podman version:

Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.8
Built:        Sun Nov 14 01:16:48 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 12
  distribution:
    distribution: fedora
    variant: workstation
    version: "34"
  eventLogger: journald
  hostname: lin-mkn
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.5-100.fc34.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 222711808
  memTotal: 16390782976
  ociRuntime:
    name: crun
    package: crun-1.3-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.3
      commit: 8e5757a4e68590326dafe8a8b1b4a584b10a1370
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 16832520192
  swapTotal: 16903036928
  uptime: 4h 8m 34.72s (Approximately 0.17 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/mkn/.config/containers/storage.conf
  containerStore:
    number: 104
    paused: 0
    running: 0
    stopped: 104
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.4
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.4
        using FUSE kernel interface version 7.31
  graphRoot: /home/mkn/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 84
  runRoot: /run/user/1000/containers
  volumePath: /home/mkn/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 1636849008
  BuiltTime: Sun Nov 14 01:16:48 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

Last metadata expiration check: 0:21:23 ago on Tue 30 Nov 2021 12:56:56 CET.
Installed Packages
podman.x86_64                                                                                                                                             3:3.4.2-1.fc34                                                                                                                                              @updates

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[vrothberg]

Thank you for taking the time to open the issue, @m-kn.

I can reproduce the bug which happens when there is a local manifest with no entry matching the current system. I'll wrap up a fix.

[m-kn]

Thx for the quick fix. I'll check it out as soon as it hits fedoras updates-testing repo.