Can't upgrade core machine

[m-emelchenkov]

Issue Description

podman machine ssh 'sudo rpm-ostree upgrade --check' failed with error

error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: reading manifest 5.0 in quay.io/containers/podman-machine-os: unauthorized: access to the requested resource is not authorized

Steps to reproduce the issue

Steps to reproduce the issue

1. init machine with podman machine init, macOS 14.4.1 w/ podman 5.0.2 from HomeBrew.
2. run podman machine ssh 'sudo rpm-ostree upgrade --check'.

Describe the results you received

error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: reading manifest 5.0 in quay.io/containers/podman-machine-os: unauthorized: access to the requested resource is not authorized

Describe the results you expected

It should output list of available updates.

podman info output

host:
  arch: arm64
  buildahVersion: 1.35.4
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.87
    systemPercent: 0.08
    userPercent: 0.05
  cpus: 6
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2045
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.8.8-300.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 3632762880
  memTotal: 4085985280
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240426.gd03c4e2-1.fc40.aarch64
    version: |
      pasta 0^20240426.gd03c4e2-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 17m 59.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 68114427904
  graphRootUsed: 5851197440
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.0.3
  Built: 1715299200
  BuiltTime: Fri May 10 03:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/arm64
  Version: 5.0.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[Luap99]

With what version did you create the podman machine VM, the URL was only temporary and not part of 5.0 AFAIK. The proper address is quay.io/podman/machine-os. And I don't see the old one used in any 5.0.x releases

[m-emelchenkov]

@Luap99 I created VM with Podman 5.0.2 with no configs altered.

[Luap99]

@baude PTAL is there something in the machine that hard codes the wrong image path?

[tnk4on]

This bug hits users with v5.0 machines who cannot use Rosetta when updating to Podman CLI v5.1 or later.
A simple workaround is to recreate the Podman machine.
Hopefully in the long run this issue will be resolved.

[m-emelchenkov]

@tnk4on Am I understand right that it is fixed in 5.1?

[tnk4on]

@m-emelchenkov No, this relates to machine-os.
Other workarounds use podman machine os apply. This works.
However, I am not sure if this is the update method the Podman team wants.

% podman machine init test --now
% podman machine os apply quay.io/podman/machine-os:5.1 --restart test
Pulling manifest: ostree-unverified-registry:quay.io/podman/machine-os:5.1
Importing: ostree-unverified-registry:quay.io/podman/machine-os:5.1 (digest: sha256:ee1f8b9842df2191e2605d4b3c5fecad2c557a9ed70e07acc2f5ffb1d7a20d97)
ostree chunk layers needed: 51 (713.6 MB)
custom layers needed: 17 (135.3 MB)
Staging deployment...done
Upgraded:
  aardvark-dns 1.10.0-1.fc40 -> 102:1.10.0-1.20240506173313423293.main.51.g069ab45.fc40
  containers-common 5:0.58.0-2.fc40 -> 102:0.58.0-1.20240513130008279470.main.169.g477496cf.fc40
  containers-common-extra 5:0.58.0-2.fc40 -> 102:0.58.0-1.20240513130008279470.main.169.g477496cf.fc40
  crun 1.14.4-1.fc40 -> 102:1.14.4-1.20240424212458225367.main.39.gd075e53.fc40
  crun-wasm 1.14.4-1.fc40 -> 102:1.14.4-1.20240424212458225367.main.39.gd075e53.fc40
  libgomp 14.0.1-0.15.fc40 -> 14.1.1-1.fc40
  netavark 1.10.3-3.fc40 -> 102:1.10.1-1.20240513124445753694.main.112.gd982b8b.fc40
  podman 5:5.0.3-1.fc40 -> 102:5.1.0~dev-1.20240513183216697996.main.477.c9808e7ed.fc40
Changes queued for next boot. Run "systemctl reboot" to start a reboot
Machine "test" restarted successfully

% podman machine ssh test sudo rpm-ostree status
State: idle
Deployments:
● ostree-unverified-image:docker://quay.io/podman/machine-os:5.1
                   Digest: sha256:ee1f8b9842df2191e2605d4b3c5fecad2c557a9ed70e07acc2f5ffb1d7a20d97
                  Version: 40.20240504.2.0 (2024-05-13T19:01:35Z)

  ostree-remote-image:fedora:docker://quay.io/containers/podman-machine-os:5.0
                   Digest: sha256:cf6b3b16eebfba0093a06e7884f5d6db25a460d645eded1b6e067458da7fdef6
                  Version: 40.20240504.2.0 (2024-05-10T19:15:09Z)
% podman machine ssh test sudo rpm-ostree upgrade --check
Note: --check and --preview may be unreliable.  See https://github.com/coreos/rpm-ostree/issues/1579
No updates available.

[tnk4on]

The code for the cause is hard coded below
https://github.com/dustymabe/build-podman-machine-os-disks/blob/b23251436930b628afea4b5429c5621aa4eee0fb/build-podman-machine-os-disks.sh#L115

[Luap99]

The code for the cause is hard coded below https://github.com/dustymabe/build-podman-machine-os-disks/blob/b23251436930b628afea4b5429c5621aa4eee0fb/build-podman-machine-os-disks.sh#L115

This should be fixed to point to the actual image @baude