[5.0.0 osx] Unable to start the podman machine with osx Monterey

[manofthepeace]

Issue Description

My podman updated to 5.0.0 from 4.9.3 via brew. With that update I can no longer create the podman machine.

the problem seems to be the following; crc-org/vfkit#37 related to efi.

Unfortunately the macbook pro is an older model that is now unsupported by apple, so I cannot have it upgraded to ventura or sonoma.

Steps to reproduce the issue

Steps to reproduce the issue

1. Have a monterey os
2. run podman machine init
3. run podman machine start

Describe the results you received

Error shown by podman; Error: vfkit exited unexpectedly with exit code 1
Error while running vfkit manually: Error: unsupported macOS version

Describe the results you expected

I would have expected the machine to start. Now my questions is more, is there a way to run the podman machine without efi? Not sure if there are scripts I can tweak or options to set about that so the machine can be created/started without efi, to have a working podman/podman-desktop env.

podman info output

OS: darwin/amd64
provider: applehv
version: 5.0.0

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Darwin Kernel Version 21.6.0: Mon Feb 19 20:24:34 PST 2024

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[ashley-cui]

Unfortunately I think the best way forward is to move back to Podman 4.9, by uninstalling it through brew and re-installing it via the pkginstaller shipped on our release page: https://github.com/containers/podman/releases/tag/v4.9.3

[manofthepeace]

Thanks for the reply. Do you know if 4.9.X will be supported for a while within podman-desktop or this setup is actually sort of deprecated. Is there a possibility to fix that in podman, is efi really required? Would be nice if it could detect older macs and use a non efi version in the future.

Thanks again.

[ashley-cui]

I think that's a question to ask on podman-desktop for desktop support.

@baude @cfergeau Do you know about our EFI requirement?

[cfergeau]

The only alternative to using EFI would be to extract the kernel/inintrd/kernel cmdline from the VM image, and provide these to vfkit. Using EFI is simpler as vfkit/apple virtualization framework can then boot the disk image directly.
EFI support is only available in macOS 13 or newer, but I'm afraid a fallback LinuxBootloader would be quite some work.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[urkle]

@ashley-cui how does one exactly get podman 4.9.x working again? as my podman machine auto-upgraded yesterday and switched to podman 5 whcih means I cannot do anything as I can't start a podman 5 machine and I cannot connect the 4.9 client to my qemu machine.

This whole upgrade to podman 5 has not been handled well.

[ashley-cui]

@urkle Podman v4.9 was an experiment in podman machine, and since we were relatively inexperienced in that area, we needed to re-vamp our machine stack almost completely for 5.0. 5.0 to fixes a lot of these potholes, but the gap between 4.9 and 5.0 was big enough that implementing direct upgrades was not possible. Unfortunately, it may have caused a rough transition between 4.9 and 5.0, but, we hope that our fixes in Podman 5.0 and beyond makes our UX much better. Our blog goes into a bit more detail about this.

If you can, we'd recommend you upgrade to Podman v5.0. However, in the case that you're unable to do so, using the following commands to init a new 4.9 machine should fix your issue. Let me know if this gets things working again! And of course, feel free to file any more issues you come across.

x86:

podman machine init --image-path https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20240407.3.0/x86_64/fedora-coreos-39.20240407.3.0-qemu.x86_64.qcow2.xz

arm:

podman machine init --image-path=https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20240407.3.0/aarch64/fedora-coreos-39.20240407.3.0-qemu.aarch64.qcow2.xz

[urkle]

@ashley-cui thanks for the detailed information.

I had tried to do a full upgrade to 5.0 but ran into the issue described here and was unable to start the machine.

after much fiddling I managed to get the 4.9.4 installed and used --image-path stable to get the latest 39.x machine created. Then I further disable zincati ( I believe that is all that is needed to prevent auto updates). Further I had to do a lot of "fun" creative work to extract out my volumes from the old image to import into the new image since there is no direct way to export/import volumes in podman machine.

I had to

1. ssh into the vm (before upgrading to podman 5)
2. run podman volume export on each volume (in the vm)
3. use podman machine ssh to "cat" the tar to a local file.
   podman machine ssh cat myvolume_name.tar > myvolume_name.tar
4. then after the failed 5 upgrade and getting the 4.9 vm build back up I ended up using a busybox container to copy the volumes back in. (probably should have just done that to export them as well, but was hoping the export/import would have been nicer.. )

Is there not a possibility of adding back the QEMU backend as an alternative for older macOS?

[ashley-cui]

@urkle Hmm interesting, I believe we disabled zincati already with this commit, that's strange that it's still updating. I can take a look at that.

We moved to appleHV as it resolved a lot of issues wrt dependencies, speed, and some other factors. We currently plan on only supporting appleHV for our future.

[urkle]

@ashley-cui it could be my original podman machine was created before podman 4.8.0 and that is why it had updated itself?

[ashley-cui]

@urkle That would be it!

[hritik5102]

Problem

I was doing Podman machine configuration setup for M1, and when I ran the podman machine start command, It failed with

Starting machine "podman-machine-default"
Error: vfkit exited unexpectedly with exit code 0

Addition information:

• I installed the latest podman version 5.0.2 using brew install podman
• I installed CoreOS version 37.20230401.3.0 instead of 38.20230430.2.1 because the Postgres container crashed during the initial schematic setup on version 38.20230430.2.1, causing the database to be in an invalid state.
• Here are the steps that I've followed.
• Download the older version OS image file (link)

curl -O https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/37.20230401.3.0/aarch64/fedora-coreos-37.20230401.3.0-qemu.aarch64.qcow2.xz

• Initialise the podman machine

podman machine init --disk-size 50 --memory 6144 --cpus 2 --image-path /var/podman_home/fedora-coreos-37.20230401.3.0-qemu.aarch64.qcow2.xz

• Start the machine - podman machine start
• Check the screenshot attached below for more information

Things that i've tried:

• I followed the steps mentioned here and removed the podman machine rm podman-machine-default and reinstall the coreos and did the podman machine configuration again from the start, but it didn't work out.
• Followed this step as well, but it didn't work out: https://podman-desktop.io/docs/troubleshooting/troubleshooting-podman-on-macos#recovering-from-a-failed-start

Note

I'm not sure whether this is relevant or not, I've checked no podman-machine-default file is created inside ~/.ssh folder.

@ashley-cui Could you please help me here? Thank you.

[hritik5102]

JFYI : The above issue got resolved.

Solution:

• Installed the latest version of podman ( viz 5.0.3 ), Not sure, whether this is actually contributed to the fix.
• Installed the latest CoreOS ( i.e fedora image ) version. Instead of using 37.20230401.3.0 version, Didn't face the postgres issue which we faced earlier.
• Run the podman machine init command without image path, as it will installed the latest one.

podman machine init --disk-size 50 --memory 6144 --cpus 2

[glaydston]

@hritik5102 I've had the same problem here and aftering to search I'm found this article:

• Migration of Podman 4 to Podman 5 machines

In the case of Macs running Podman 4, it was based on QEMU virtualization. Podman 5 uses the Apple Hypervisor on Macs and support for QEMU was deprecated. Also, depending on where you have obtained your Podman 5 binaries, you will want to make sure you have vfkit installed as it replaces QEMU.

Although the problem persisted, following the steps outlined in this article the dreaded vfkit exit 1 and removing the brew install to use the Installer Launcher resolved the issue for me.

[hritik5102]

Thanks, @glaydston for sharing the article 🙌🏻

[lesinigo]

I had just gone through the whole "podman on macOS Monterey" thing myself and while I do understand that podman 4.x is "the old one" and issues are often closed as "it's now unsupported", I also see that 4.9.5 came out last week meaning it's not totally dead.

AFAIK two/three simple changes would greatly improve the experience for anyone still stuck on older Apple OSes:

1. in the downloads section of the website / release descriptions of GitHub, make it clear that 5.x requires macOS >= 13 (or whatever it actually is) - solves people ending up in the situation of the issue we're discussing here
2. let the default machine image for podman 4.9.x be FCOS 39 or whatever the latest compatible image is (solves podman v4 is fetching podman v5 and it's not compliant on macOS #22517)
3. I'm not a Podman Desktop user so not sure about it but I'd double check that it is not proposing / trying updates to podman 5.x on known-incompatible macOS versions

Meanwhile, instructions to manually solve the situation are already available but a little scattered here and there so I'll recap it for anyone that's stumbling on this problem (worked for me on macOS 12.x Monterey):

• ensure you have no podman machine lying around (podman machine stop; podman machine rm -f - note that this will remove any image, volume, and so on)
• download and install the latest 4.x podman CLI from the GitHub releases page (it's 4.9.5 at the time of this writing) - this will overwrite any previous podman CLI installation so you shouldn't need to manually clean it up beforehand
• every time you need to init or re-init podman machine, force it to an older FCOS39 image (eg. podman machine init --image-path https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20240407.3.0/x86_64/fedora-coreos-39.20240407.3.0-qemu.x86_64.qcow2.xz --now). You can check if they publish any newer FCOS39 image in the Fedora CoreOS Release Notes
• I'm not using Podman Desktop so can't check it but I guess you just need to fix the podman cli / podman machine side and Desktop will probably work ok