auto-update leaves variant empty, fails to pull image

[holgerpieta]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug

Description
Running docker.io/syncthing/syncthing on Raspberry Pi works like a charm, but when I try to do podman auto-update, it fails:

Error: 1 error occurred:
        * error auto-updating container "83f2e77d0ee018361ad4998f4f5ba4f54abdcb033746261810f6f5941cacf05f": image check for "docker.io/syncthing/syncthing:latest" failed: Error choosing image instance: no image found in manifest list for architecture arm, variant "", OS linux

Looks to me like podman correctly sets variant to v7 "everywhere" (at least everywhere I tried) except for auto-update, where it leaves it blank.
The official Syncthing image appears to be one of the "official" images that use arch=arm and variant=v7 instead of arch=arm32v7 like most other images do.

Steps to reproduce the issue:

1. On 32 bit Raspberry Pi OS, start container with image docker.io/syncthing/syncthing:latest
2. Create a service with podman generate systemd --new
3. Start newly created service.
4. Run podman auto-update

Describe the results you received:
auto-update fails to find and pull the correct image.
Container is not updated.

Describe the results you expected:
Correct image identified and pulled.
Container updated, if new image available.

Additional information you deem important (e.g. issue happens only occasionally):
Not sure if that makes any difference, but I'm running rootless.

Output of podman version:

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.15.2
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/arm

Output of podman info --debug:

host:
  arch: arm
  buildahVersion: 1.19.4
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.26, commit: '
  cpus: 4
  distribution:
    distribution: raspbian
    version: "10"
  eventLogger: journald
  hostname: server01
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.11-v7l+
  linkmode: dynamic
  memFree: 2923245568
  memTotal: 4025630720
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.18.1-7931a-dirty
      commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.3.3
  swapFree: 104853504
  swapTotal: 104853504
  uptime: 1h 11m 53.51s (Approximately 0.04 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/pi/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.4.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.4.1
        using FUSE kernel interface version 7.27
  graphRoot: /home/pi/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/pi/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/arm
  Version: 3.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown 100:3.0.1-2 armhf

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?
Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical machine, Raspberry Pi 4B 4GB, Raspberry Pi OS / Raspbian 10 Buster
podman installed from the Kubic repositories as described in installation guides.

[mheon]

@vrothberg PTAL

[Saroufim]

I can confirm the same issue on Fedora IoT on raspberry pi 3B+

[vrothberg]

Thanks for reaching out!

Matching the described symptoms to the code, it makes sense since we're not (yet) taking the variant into account but only the architecture (see https://github.com/containers/podman/blob/master/pkg/autoupdate/autoupdate.go#L265).

[vrothberg]

Looking into it but I am not yet sure whether we can actually access the Variant of an image once pulled. AFAICS, that's only part of the index/list but not part of an individual image.

[holgerpieta]

Also the container doesn't know it, at least I cannot find anything in podman inspect. Also podman info doesn't help. But given that podman pull will pull the right image, there must be some guessing function that selects the right image. I did a search for variant in the repo but couldn't find anything useful. So it must be hidden somewhere else.
If we could find it, maybe we can use it. Otherwise there might be the inelegant workaround of just calling pull directly, letting it do the heavy lifting. Given that podman auto-update can only be called on the machine where the container is running (I think, because it needs to call systemd), pulling the image matching the machine should be fine. And then just compare the image ID returned to decide if we need to restart.

[rhatdan]

There is a Variant field, but it is not filled. We are not displaying it in podman info, but containers/image does seem to have a field in types/

// ImageInspectInfo is a set of metadata describing Docker images, primarily their manifest and configuration.
// The Tag field is a legacy field which is here just for the Docker v2s1 manifest. It won't be supported
// for other manifest types.
type ImageInspectInfo struct {
	Tag           string
	Created       *time.Time
	DockerVersion string
	Labels        map[string]string
	Architecture  string
	Variant       string
	Os            string
	Layers        []string
	Env           []string
}

But this always seems to have "" in the field.

@mtrmac WDYT?

[Saroufim]

I've tried adding --platform=is/arch/variant to the systemd service files for the containers that are affected by auto-update but it didn't have an effect. The only way to get around this is to find an image that supports only arm (such as linuxserver/jellyfin:latest-armv32)

[vrothberg]

As mentioned above, the variant is only part of the manifest list/image index but not part of an individual image. In other words, we lose that information once an individual image is pulled.

@mtrmac pointed me to opencontainers/image-spec#809 where the variant along with other platform information is proposed to be added to the config.

But even if that gets into the OCI spec, the problem remains for Docker images. To support variants in auto-update, we must do it manually. We could, for instance, store the create/run --variant=xxx in the container config along with --os/--architecture and add a (*Container) Platform()... to libpod.

[rhatdan]

Why would you want a specific ARCH/Variant anyways? If I was running a container on a host, I would figure it is the default? Is this a case where a user is not running the default image on a host?

[Saroufim]

I've only ran default images on the host so far. The issue is that the default arch and variant aren't being automatically detected nor are they respecting the flags

[rhatdan]

So we are specifying the wrong fields? Or if they are not specified at all, then auto-update pulls the wrong arch?

[Saroufim]

No, auto-update simply doesn't find the right images and concludes that they don't exist. The containers don't get updated.

[vrothberg]

I concur. Since auto-update doesn't have the variant, it may select another image in the manifest list/image index.

To make it work, we must give users a means to make the variant explicit (see #9637 (comment)).

[holgerpieta]

Maybe I'm missing something important, but podman pull is able to pull the right image without any further hints from the user. This indicates to me that there is some variant guessing function somewhere which does a decent job. So it should (I think) be possible to:

1. Import and use this function in podman auto-update. I could not find it, but it must be somewhere.
2. Alternatively, find out what's the difference between how auto-update calls the actual image pull function compared to how pull does it and then do the same. I tried to figure that out, too, but I think I got lost in the call tree somewhere and I'm not sure my results are any good.
3. Alternatively, just execute podman pull as a separate process, let it figure out the variant and pull the right image, then check the image ID returned by the process and compare it to the image ID associated to the container. If they differ, restart the container.

Anyone can explain to me why all of above options are actually not an option?

[rhatdan]

Yes the guessing code is in containers/image, that is why I am not sure this should not happen automatically for the specific platform you are running on.

[vrothberg]

That indeed changes the picture. I was under the impression that the image was pulled with --variant=....

The code should be exactly the same but I will have another look to check why that might be.

[vrothberg]

$ podman pull --arch=arm docker.io/syncthing/syncthing:latest
Trying to pull docker.io/syncthing/syncthing:latest...
  no image found in manifest list for architecture arm, variant "", OS linux
Error: Error choosing an image from manifest list docker://syncthing/syncthing:latest: no image found in manifest list for architecture arm, variant "", OS linux

Doesn't look like podman pull would be able to pull it

[vrothberg]

Docker works docker pull --platform linux/arm docker.io/syncthing/syncthing:latest, so it seems we need to massage the code in containers/image a bit.

[vrothberg]

I opened containers/image#1176 which will fix the issue.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[holgerpieta]

I currently cannot test if 3.1 fixes the problem, because I'm waiting for a package I can install in my Raspi. Building from source would require a lot of preparation (and learning) on my side, so I would like to avoid that if possible.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Saroufim]

From my end the problem is fixed

[holgerpieta]

I haven't been able to try, because there seems to be nothing beyond 3.0.1 on any Raspberry Pi OS compatible repository.
It looks like 3.2 is in Ubuntu 21.10 Impish, but that's arm64 and so doesn't work on armhf from Raspberry Pi OS.
I'll probably going to switch that Pi from Raspberry Pi OS to Ubuntu sometimes in the near future, but I have to find the time to do that.

@Saroufim : How did you try it? Build your own podman?

But anyway, as it looks like @Saroufim confirmed that it is fixed, we can probably close the issue.

[Saroufim]

I'm running Fedora IoT 34 with podman 3.2. It was fixed back when the 3.1.2 update was pushed

[vrothberg]

From my end the problem is fixed

Thanks for checking!