Can't upgrade core machine #22678

m-emelchenkov · 2024-05-13T08:19:49Z

Issue Description

podman machine ssh 'sudo rpm-ostree upgrade --check' failed with error

error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: reading manifest 5.0 in quay.io/containers/podman-machine-os: unauthorized: access to the requested resource is not authorized

Steps to reproduce the issue

init machine with podman machine init, macOS 14.4.1 w/ podman 5.0.2 from HomeBrew.
run podman machine ssh 'sudo rpm-ostree upgrade --check'.

Describe the results you received

error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: reading manifest 5.0 in quay.io/containers/podman-machine-os: unauthorized: access to the requested resource is not authorized

Describe the results you expected

It should output list of available updates.

podman info output

host:
  arch: arm64
  buildahVersion: 1.35.4
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.87
    systemPercent: 0.08
    userPercent: 0.05
  cpus: 6
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2045
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.8.8-300.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 3632762880
  memTotal: 4085985280
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240426.gd03c4e2-1.fc40.aarch64
    version: |
      pasta 0^20240426.gd03c4e2-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 17m 59.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 68114427904
  graphRootUsed: 5851197440
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.0.3
  Built: 1715299200
  BuiltTime: Fri May 10 03:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/arm64
  Version: 5.0.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

The text was updated successfully, but these errors were encountered:

Luap99 · 2024-05-13T10:22:39Z

With what version did you create the podman machine VM, the URL was only temporary and not part of 5.0 AFAIK. The proper address is quay.io/podman/machine-os. And I don't see the old one used in any 5.0.x releases

m-emelchenkov · 2024-05-13T10:25:14Z

@Luap99 I created VM with Podman 5.0.2 with no configs altered.

Luap99 · 2024-05-13T10:29:39Z

@baude PTAL is there something in the machine that hard codes the wrong image path?

tnk4on · 2024-05-18T05:04:56Z

This bug hits users with v5.0 machines who cannot use Rosetta when updating to Podman CLI v5.1 or later.
A simple workaround is to recreate the Podman machine.
Hopefully in the long run this issue will be resolved.

m-emelchenkov · 2024-05-18T06:31:23Z

@tnk4on Am I understand right that it is fixed in 5.1?

tnk4on · 2024-05-20T06:13:32Z

@m-emelchenkov No, this relates to machine-os.
Other workarounds use podman machine os apply. This works.
However, I am not sure if this is the update method the Podman team wants.

% podman machine init test --now
% podman machine os apply quay.io/podman/machine-os:5.1 --restart test
Pulling manifest: ostree-unverified-registry:quay.io/podman/machine-os:5.1
Importing: ostree-unverified-registry:quay.io/podman/machine-os:5.1 (digest: sha256:ee1f8b9842df2191e2605d4b3c5fecad2c557a9ed70e07acc2f5ffb1d7a20d97)
ostree chunk layers needed: 51 (713.6 MB)
custom layers needed: 17 (135.3 MB)
Staging deployment...done
Upgraded:
  aardvark-dns 1.10.0-1.fc40 -> 102:1.10.0-1.20240506173313423293.main.51.g069ab45.fc40
  containers-common 5:0.58.0-2.fc40 -> 102:0.58.0-1.20240513130008279470.main.169.g477496cf.fc40
  containers-common-extra 5:0.58.0-2.fc40 -> 102:0.58.0-1.20240513130008279470.main.169.g477496cf.fc40
  crun 1.14.4-1.fc40 -> 102:1.14.4-1.20240424212458225367.main.39.gd075e53.fc40
  crun-wasm 1.14.4-1.fc40 -> 102:1.14.4-1.20240424212458225367.main.39.gd075e53.fc40
  libgomp 14.0.1-0.15.fc40 -> 14.1.1-1.fc40
  netavark 1.10.3-3.fc40 -> 102:1.10.1-1.20240513124445753694.main.112.gd982b8b.fc40
  podman 5:5.0.3-1.fc40 -> 102:5.1.0~dev-1.20240513183216697996.main.477.c9808e7ed.fc40
Changes queued for next boot. Run "systemctl reboot" to start a reboot
Machine "test" restarted successfully

% podman machine ssh test sudo rpm-ostree status
State: idle
Deployments:
● ostree-unverified-image:docker://quay.io/podman/machine-os:5.1
                   Digest: sha256:ee1f8b9842df2191e2605d4b3c5fecad2c557a9ed70e07acc2f5ffb1d7a20d97
                  Version: 40.20240504.2.0 (2024-05-13T19:01:35Z)

  ostree-remote-image:fedora:docker://quay.io/containers/podman-machine-os:5.0
                   Digest: sha256:cf6b3b16eebfba0093a06e7884f5d6db25a460d645eded1b6e067458da7fdef6
                  Version: 40.20240504.2.0 (2024-05-10T19:15:09Z)
% podman machine ssh test sudo rpm-ostree upgrade --check
Note: --check and --preview may be unreliable.  See https://github.com/coreos/rpm-ostree/issues/1579
No updates available.

tnk4on · 2024-05-20T06:19:39Z

The code for the cause is hard coded below
https://github.com/dustymabe/build-podman-machine-os-disks/blob/b23251436930b628afea4b5429c5621aa4eee0fb/build-podman-machine-os-disks.sh#L115

Luap99 · 2024-05-21T14:39:03Z

The code for the cause is hard coded below https://github.com/dustymabe/build-podman-machine-os-disks/blob/b23251436930b628afea4b5429c5621aa4eee0fb/build-podman-machine-os-disks.sh#L115

This should be fixed to point to the actual image @baude

github-actions · 2024-06-21T00:07:26Z

A friendly reminder that this issue had no activity for 30 days.

afbjorklund · 2024-07-14T08:13:33Z

When trying to upgrade 5.1.2, it also still points to 5.0 (and not to 5.1, as installed)

Looking up Podman Machine image at quay.io/podman/machine-os:5.1 to create VM

core@localhost:~$ sudo rpm-ostree upgrade
Pulling manifest: ostree-remote-image:fedora:docker://quay.io/containers/podman-machine-os:5.0
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: reading manifest 5.0 in quay.io/containers/podman-machine-os: unauthorized: access to the requested resource is not authorized

sidkshatriya · 2024-08-15T04:28:04Z

podman machine is used by a lot of a people I would suppose. I am still running into the "Failed to invoke skopeo" error (see above) when trying to do sudo rpm-ostree upgrade for many weeks now; even when I've tried reinstalling new machines.

Fix is probably not a difficult one -- what is preventing resolution of this issue ?

afbjorklund · 2024-08-15T05:48:11Z

The new machine is not (directly) based on CoreOS anymore, it is now using the "Podman Machine OS" distributed using OCI

So one would run podman machine os apply, with a new image, in order to update it. This is undocumented, and new in v5.

sidkshatriya · 2024-08-15T06:33:10Z

@afbjorklund Thanks -- shouldnt podman come baked in with a default machine like before that "just works" though ?

More over when doing this:

...
Pulling manifest: ostree-unverified-registry:quay.io/podman/machine-os:5.1
Importing: ostree-unverified-registry:quay.io/podman/machine-os:5.1 (digest: 
...

This statement "ostree-unverified-registry" by using the word "unverified" feels a bit unsecure though it is probably OK i guess ?? Never saw the word unverified before in the context of podman machine/ostree

afbjorklund · 2024-08-15T06:44:12Z

The automatic updates have been disabled (also for CoreOS), since Podman version 4.8. (commit 94818f5)

It is now up to the user, to keep their "testing" distribution updated... Or recreate their machines, occasionally

afbjorklund · 2024-08-15T07:08:21Z

This statement "ostree-unverified-registry" by using the word "unverified" feels a bit unsecure though it is probably OK i guess ??

This is a separate issue, as far as I know it is supposed to be signed and verified

EDIT: I mean eventually, it should... Probably isn't now?

sidkshatriya · 2024-08-15T07:12:50Z

The automatic updates have been disabled (also for CoreOS), since Podman version 4.8. (commit 94818f5)

Thanks again for your reply @afbjorklund ! The issue is not automatic updates (which used to happen via "zincati") but manual updates via sudo rpm-ostree upgrade. That seems broken too !

When I do:

podman machine init -m 8192 --cpus 4

I get the Podman Machine OS as expected:

Looking up Podman Machine image at quay.io/podman/machine-os:5.2 to create VM
Getting image source signatures
Copying blob 44a1ba262848 done   | 
Copying config 44136fa355 done   | 
Writing manifest to image destination
44a1ba262848f3649685c1e2fc9907f9141c8fa7b451692687a4fe490e48daff
Extracting compressed file: podman-machine-default-arm64.raw: done  
Machine init complete
To start your machine run:
...

But when I try to upgrade it manually like so:

[core@localhost ~]$ sudo rpm-ostree upgrade --check
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: reading manifest 5.0 in quay.io/containers/podman-machine-os: unauthorized: access to the requested resource is not authorized
[core@localhost ~]$ sudo rpm-ostree upgrade        
Pulling manifest: ostree-remote-image:fedora:docker://quay.io/containers/podman-machine-os:5.0
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: reading manifest 5.0 in quay.io/containers/podman-machine-os: unauthorized: access to the requested resource is not authorized
[core@localhost ~]$

I get this skopeo error.

afbjorklund · 2024-08-15T07:15:01Z

Right, the documentation needs to be updated (to use os apply)

Luap99 · 2024-08-15T08:15:42Z

@baude PTAL again, can you please fix the image name in the images?

sidkshatriya · 2024-10-16T15:22:35Z

I believe this is a fairly important issue because podman core machines are no longer updated as they used to be. People are still greeted by an error when doing sudo rpm-ostree upgrade --check many months later. This method should be disabled if it is no longer allowed.

The workaround is to podman machine os apply <new-image> but no documentation seems to advise that this is the new way going forward. There is no note anywhere on the podman homepage on how to maintain or upgrade your machines. Also when doing this podman uses the phrase "unverified registry" which makes this approach seem hackish/not recommended. See here

Am I missing something ? Have people moved away from podman machine as a way of doing things on, say, macOS ? Are podman machines in general deprecated ? Given the active discussion on this ticket I would guess this is not the case.

Is there a blocker to having this ticket being resolved ?

Sorry for all these questions but I'm surprised by the still open status of this ticket. I would guess a lot of people would be using podman on macOS at least !

Luap99 · 2024-10-16T15:29:56Z

@baude PTAL and fix this

sidkshatriya · 2024-10-18T20:45:15Z

@Luap99 -- Is there someone else that can be pinged ?

baude · 2024-10-29T18:14:40Z

As was pointed out, podman os apply <new-image> is the upgrade mechanism but would also agree we are in bad place for this. In no place (that I am aware of) was it recommended to use rpm-ostree to upgrade an image. Perhaps there is an old blog or something. moreover, there are versioning issues in doing this. that said, the canonical place for images is quay.io/podman/machine-os. But frankly, we do not promise to published updated OS until a new version of podman comes out.

This can and should get better in the future.

sidkshatriya · 2024-10-29T19:46:13Z

Can anybody point to a podman resource on the internet which confirms the canonical URL for the machine OS image ? It's good to have this documented somewhere in official podman materials. I couldn't find this myself. It just increases confidence when downloading the image.

The (main) fedora project itself has great documentation on Fedora Desktop/Sever images download, image shasums etc. A short note for podman saying what the official distribution link for podman machine os images would be helpful.

baude · 2024-10-29T22:28:12Z

Can anybody point to a podman resource on the internet which confirms the canonical URL for the machine OS image ? It's good to have this documented somewhere in official podman materials. I couldn't find this myself. It just increases confidence when downloading the image.

It is in the source code ->

podman/pkg/machine/ocipull/ociartifact.go

Line 29 in b2e6d53

artifactImageName = "machine-os"

and I also provided it in the reply above. As I mentioned, this is a weakness for us right now. We have not been able to address this like we'd like to; hang tight, future changes coming.

The (main) fedora project itself has great documentation on Fedora Desktop/Sever images download, image shasums etc. A short note for podman saying what the official distribution link for podman machine os images would be helpful.

If you follow to the canonical sources as I indicated, you can see all the sha sums you want. But I think we have a use case issue here. This is all supposed to automatic and not require user intervention. Updates are a feature we are working on. This stuff is meant to be appliance-like and on top of that --- potentially ephemeral in nature.

If you would like to complain about this, please come to one of our community meetings and you can let me have it there. I'm actually glad you are passionate about it. Of course, contributions are also welcome too!

Luap99 · 2024-11-01T11:17:25Z

Our docs still say to use podman machine ssh 'sudo rpm-ostree upgrade --check' so we recommend that
https://docs.podman.io/en/latest/markdown/podman-machine-init.1.html

Otherwise I agree we are still building out or automation in https://github.com/containers/podman-machine-os at the moment, once that is all in place we can properly automate most of this and work on the upgrade flow in the client.
We could fix the URL short term but it would not really make sense as we hard code the x.y version so checking for an update from within will not keep updating for long only the .z version. As mentioned we working on a better flow but there have been some other priorities lately so this has taken longer that we would have hoped.

afbjorklund · 2024-11-01T14:13:01Z

I think the new update mechanism could use some more (philosophical) documentation as well. Now it says:

"By default, the VM distribution is Fedora CoreOS <Testing> except for WSL which is based on a custom Fedora image. While Fedora CoreOS upgrades come out every 14 days, the automatic update mechanism Zincata <sic> is disabled by Podman machine"

"Fedora CoreOS upgrades come out every 14 days and are detected and installed automatically."

So the OS itself is automatically updated, but the updates have been disabled? Could need a little "why" there.

baude · 2024-11-13T19:19:17Z

this should be fixed in 5.3 ... more to come but the manual commands work.

sidkshatriya · 2024-11-14T20:15:43Z

@baude I did a cursory look at the release notes -- nothing caught my eye regarding the problems discussed in this github issue -- can you share the improvements in 5.3 regarding upgrading the core machine if possible ?

baude · 2024-11-14T20:32:53Z

the core of the problem is the ostree reference embedded into podman machine's OS. This is nothing podman related per'se. If you run the rpm-ostree commands as was posted in the original issue writeup, they should now work

m-emelchenkov added the kind/bug Categorizes issue or PR as related to a bug. label May 13, 2024

github-actions bot added the remote Problem is in podman-remote label May 13, 2024

jakecorrenti added the machine label May 13, 2024

github-actions bot added the stale-issue label Jun 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't upgrade core machine #22678

Can't upgrade core machine #22678

m-emelchenkov commented May 13, 2024 •

edited

Loading

Luap99 commented May 13, 2024

m-emelchenkov commented May 13, 2024

Luap99 commented May 13, 2024

tnk4on commented May 18, 2024

m-emelchenkov commented May 18, 2024

tnk4on commented May 20, 2024 •

edited

Loading

tnk4on commented May 20, 2024

Luap99 commented May 21, 2024

github-actions bot commented Jun 21, 2024

afbjorklund commented Jul 14, 2024 •

edited

Loading

sidkshatriya commented Aug 15, 2024

afbjorklund commented Aug 15, 2024 •

edited

Loading

sidkshatriya commented Aug 15, 2024

afbjorklund commented Aug 15, 2024

afbjorklund commented Aug 15, 2024 •

edited

Loading

sidkshatriya commented Aug 15, 2024

afbjorklund commented Aug 15, 2024

Luap99 commented Aug 15, 2024

sidkshatriya commented Oct 16, 2024

Luap99 commented Oct 16, 2024

sidkshatriya commented Oct 18, 2024

baude commented Oct 29, 2024

sidkshatriya commented Oct 29, 2024

baude commented Oct 29, 2024

Luap99 commented Nov 1, 2024

afbjorklund commented Nov 1, 2024

baude commented Nov 13, 2024

sidkshatriya commented Nov 14, 2024

baude commented Nov 14, 2024

Can't upgrade core machine #22678

Can't upgrade core machine #22678

Comments

m-emelchenkov commented May 13, 2024 • edited Loading

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

Luap99 commented May 13, 2024

m-emelchenkov commented May 13, 2024

Luap99 commented May 13, 2024

tnk4on commented May 18, 2024

m-emelchenkov commented May 18, 2024

tnk4on commented May 20, 2024 • edited Loading

tnk4on commented May 20, 2024

Luap99 commented May 21, 2024

github-actions bot commented Jun 21, 2024

afbjorklund commented Jul 14, 2024 • edited Loading

sidkshatriya commented Aug 15, 2024

afbjorklund commented Aug 15, 2024 • edited Loading

sidkshatriya commented Aug 15, 2024

afbjorklund commented Aug 15, 2024

afbjorklund commented Aug 15, 2024 • edited Loading

sidkshatriya commented Aug 15, 2024

afbjorklund commented Aug 15, 2024

Luap99 commented Aug 15, 2024

sidkshatriya commented Oct 16, 2024

Luap99 commented Oct 16, 2024

sidkshatriya commented Oct 18, 2024

baude commented Oct 29, 2024

sidkshatriya commented Oct 29, 2024

baude commented Oct 29, 2024

Luap99 commented Nov 1, 2024

afbjorklund commented Nov 1, 2024

baude commented Nov 13, 2024

sidkshatriya commented Nov 14, 2024

baude commented Nov 14, 2024

m-emelchenkov commented May 13, 2024 •

edited

Loading

tnk4on commented May 20, 2024 •

edited

Loading

afbjorklund commented Jul 14, 2024 •

edited

Loading

afbjorklund commented Aug 15, 2024 •

edited

Loading

afbjorklund commented Aug 15, 2024 •

edited

Loading