podman machine start stuck "Waiting for VM" indefinitely with qemu synchronous exception MacOS M3

[peter-crist]

Issue Description

With a fresh install via homebrew, I cannot start a machine and it hangs indefinitely at Waiting for VM. Running with debug shows that qemu hits a Synchronous Exeception at 0x00000000BC308000.

Steps to reproduce the issue

Steps to reproduce the issue

1. brew install podman
2. podman machine init
3. podman machine start

Describe the results you received

VM fails to start.

Describe the results you expected

VM starts.

podman info output

Version: 4.7.2
OS: macOS 14.2
Chip: Apple M3 Pro
Architecture: arm64

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

[danielroseman]

I had this issue; updating qemu from 8.1.3_1 to 8.1.3_2 via brew upgrade qemu fixed it.

[peter-crist]

Updating to qemu 8.1.3_2 resulted in a different exception and hang:

ASSERT [ArmCpuDxe] /home/kraxel/projects/qemu/doms/edk2/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(333): ((BOOLEAN)(0==1))

[jwhostet]

Reading through some other issues I've found this workaround: podman machine init -m 3072

podman 4.7.2
qemu 8.1.3_2
MacOs 14.1.1 - M3 Max

[peter-crist]

Reading through some other issues I've found this workaround: podman machine init -m 3072

Unfortunately no such luck on my end.

[mchataigner]

podman machine init -m 3072 solved the issue for me

podman 4.8.8
qemu 8.1.3_2
MacOs 14.1.1 - M3 pro

[jaydorsey]

Reading through some other issues I've found this workaround: podman machine init -m 3072

Unfortunately no such luck on my end.

I didn't have any luck doing this initially either; then I absolutely nuked everything:

1. Killed all the podman related processes using Activity Monitor
2. Ran podman machine rm to delete the machine
3. Uninstalled podman, podman desktop, and qemu

Re-ran it all from scratch today, starting with podman only and then running podman desktop afterwards:

1. Reinstall podman & qemu
2. Run the command from jwhostet, then podman machine start and it worked
3. Open up podman desktop, everything looks good

podman 4.8.0
qemu 8.1.3_2
macos 14.1.1 M3 Max

I have a max, you have a pro, but one other person with a pro reported success so double-checking everything might be worth a shot? I tried similar steps before but skipped the podman machine rm step before and I deleted podman desktop by just dragging it to the trash before so that might have been why it failed the first time

[robahtou]

@jaydorsey I have the same specs and followed your instructions. I am still unable to get the VM to run:
Waiting for VM ...

EDIT:
I installed brew install podman-desktop and enabled podman engine this way and it is working now 🤷🏽‍♂️

[peter-crist]

@jaydorsey I have the same specs and followed your instructions. I am still unable to get the VM to run: Waiting for VM ...

EDIT: I installed brew install podman-desktop and enabled podman engine this way and it is working now 🤷🏽‍♂️

This worked for me as well!
Additionally, I did the following before installing podman-desktop:

• brew uninstall podman
• brew uninstall qemu
• Removed any lingering podman files I could find:
  • Removed ~/.ssh/podman-machine-default and ~/.ssh/podman-machine-default.pub.
  • Removed ~/.local/share/containers and ~/.config/containers

[jaydorsey]

One thing of maybe note: It worked great yesterday. I let my machine go to sleep, logged back on today, and the whole thing was locked up again list it was. Podman said it wasn't running, but it actually was, I can't delete images or machines, everything was essentially responsive. I tried force-quitting everything and restarting with no luck

I need to try a restart/reboot of my computer later to see if that fixes it but it didn't seem to work after I let my computer sleep (or perhaps some other unrelated reason, like a crash)

[peter-crist]

And now I'm seeming to have problems with volumes and my VM is somehow back in a "Currently starting" state. Can't stop, or remove it. 👎

[osamamunir23]

I have tried all steps as above, but still stuck on Waiting for VM ...

if i log out the errors, thats what i get back:

podman --log-level=debug machine start
INFO[0000] podman filtering at log level debug
DEBU[0000] Using Podman machine with qemu virtualization provider
Starting machine "podman-machine-default"
DEBU[0000] {true 1500 map[forward-dest:[/run/user/503/podman/podman.sock] forward-identity:[/Users/USERNAME/.ssh/podman-machine-default] forward-sock:[/Users/USERNAME/.local/share/containers/podman/machine/qemu/podman.sock] forward-user:[core]] [] map[listen-qemu:unix:///var/folders/v9/n5xqlp4j441csny4xvmgyr6c0000gq/T/podman/qmp_podman-machine-default.sock] /var/folders/v9/n5xqlp4j441csny4xvmgyr6c0000gq/T/podman/podman-machine-default_proxy.pid 49455}
DEBU[0000] qemu cmd: [/opt/homebrew/bin/qemu-system-aarch64 -accel hvf -accel tcg -cpu host -M virt,highmem=on -drive file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -drive file=/Users/USERNAME/.local/share/containers/podman/machine/qemu/podman-machine-default_ovmf_vars.fd,if=pflash,format=raw -m 3072 -smp 5 -fw_cfg name=opt/com.coreos/config,file=/Users/USERNAME/.config/containers/podman/machine/qemu/podman-machine-default.ign -qmp unix:/var/folders/v9/n5xqlp4j441csny4xvmgyr6c0000gq/T/podman/qmp_podman-machine-default.sock,server=on,wait=off -netdev socket,id=vlan,fd=3 -device virtio-net-pci,netdev=vlan,mac=5a:94:ef:e4:0c:ee -device virtio-serial -chardev socket,path=/var/folders/v9/n5xqlp4j441csny4xvmgyr6c0000gq/T/podman/podman-machine-default_ready.sock,server=on,wait=off,id=apodman-machine-default_ready -device virtserialport,chardev=apodman-machine-default_ready,name=org.fedoraproject.port.0 -pidfile /var/folders/v9/n5xqlp4j441csny4xvmgyr6c0000gq/T/podman/podman-machine-default_vm.pid -virtfs local,path=/Users,mount_tag=vol0,security_model=none -virtfs local,path=/private,mount_tag=vol1,security_model=none -virtfs local,path=/var/folders,mount_tag=vol2,security_model=none -drive if=virtio,file=/Users/USERNAME/.local/share/containers/podman/machine/qemu/podman-machine-default_fedora-coreos-39.20231119.2.0-qemu.aarch64.qcow2]
Waiting for VM ...

[jordyguntur]

Also stuck on Waiting for VM. Has anyone found a work-around without using podman-desktop?

[relistan]

M3 Pro
macOS 14.1
QEMU 8.1.3_2
podman 4.7.2

I have intermittently been able to get it to work between sessions of removing everything, re-installing, deleting and recreating machines. Once it decides to work, it is fine until the mac goes to sleep while the QEMU machine is running.

On recovery from sleep, the podman machine shows as running, but it's not accessible. From then on the issues remains until some series of uninstalling and reinstalling, deleting and recreating machines finally works. I have not yet figured it out.

The specific problem seems to be with -netdev socket,id=vlan,fd=3 if replacing fd=3 with listen=:1234 the machine starts, but of course does not work with podman.

EDIT: Steps that seem to make it work again:

1. podman machine rm
2. brew uninstall podman
3. brew uninstall qemu
4. Reboot
5. brew install podman (qemu installs with it)
6. Create machine from podman desktop but not let it start it
7. podman machine start

[pantaovay]

I have tried all the methods, but all do not work.

[herblet]

Is this not a result of this issue tracked on the qemu project, which is the result of a bug even further upstream?

https://gitlab.com/qemu-project/qemu/-/issues/1990

[m8nmueller]

I agree with @herblet. Also see the fix employed by Lima: lima-vm/lima#2061

What worked for me:

• Download https://github.com/lima-vm/edk2-patched.tmp/releases/download/edk2-stable202311%2Blima.0/edk2-aarch64-code.fd.gz from https://github.com/lima-vm/edk2-patched.tmp/releases

• Init podman machine

• Update podman machine config json using
  sed -i 's@file=.*edk2-aarch64-code.fd@file=/path/to/downloaded/edk2-aarch64-code.fd@g' /path/to/qemu-config/podman-machine-default.json

  where the qemu-config directory is reported by MachineConfigDir in the output of podman machine info.

• Start machine

[nzhuk98]

What worked for me:

• Download https://github.com/lima-vm/edk2-patched.tmp/releases/download/edk2-stable202311%2Blima.0/edk2-aarch64-code.fd.gz from https://github.com/lima-vm/edk2-patched.tmp/releases

These links don't work anymore.

I am on M3 Max. Changing memory on init didn't work.

Uninstalling everything, and installing podman-desktop worked.

podman machine rm
brew uninstall podman qemu
rm -rf ~/.config/containers
rm -rf ~/.local/share/containers
brew install podman-desktop

Start podman-desktop, init and start podman machine. I disabled "Machine with root privileges:", but it probably doesn't matter.

[nutshellfool]

What worked for me:

• Download https://github.com/lima-vm/edk2-patched.tmp/releases/download/edk2-stable202311%2Blima.0/edk2-aarch64-code.fd.gz from https://github.com/lima-vm/edk2-patched.tmp/releases

These links don't work anymore.

I am on M3 Max. Changing memory on init didn't work.

Uninstalling everything, and installing podman-desktop worked.

podman machine rm
brew uninstall podman qemu
rm -rf ~/.config/containers
rm -rf ~/.local/share/containers
brew install podman-desktop

Start podman-desktop, init and start podman machine. I disabled "Machine with root privileges:", but it probably doesn't matter.

This works for me.

[FullofQuarks]

What worked for me:

• Download https://github.com/lima-vm/edk2-patched.tmp/releases/download/edk2-stable202311%2Blima.0/edk2-aarch64-code.fd.gz from https://github.com/lima-vm/edk2-patched.tmp/releases

These links don't work anymore.

I am on M3 Max. Changing memory on init didn't work.

Uninstalling everything, and installing podman-desktop worked.

podman machine rm
brew uninstall podman qemu
rm -rf ~/.config/containers
rm -rf ~/.local/share/containers
brew install podman-desktop

Start podman-desktop, init and start podman machine. I disabled "Machine with root privileges:", but it probably doesn't matter.

Was having the same issue, but this worked for me. Not sure I like having to run an installer from within a cask, since this installation is controlled outside of brew

[denisborisov]

What worked for me:

• Download https://github.com/lima-vm/edk2-patched.tmp/releases/download/edk2-stable202311%2Blima.0/edk2-aarch64-code.fd.gz from https://github.com/lima-vm/edk2-patched.tmp/releases

These links don't work anymore.

I am on M3 Max. Changing memory on init didn't work.

Uninstalling everything, and installing podman-desktop worked.

podman machine rm
brew uninstall podman qemu
rm -rf ~/.config/containers
rm -rf ~/.local/share/containers
brew install podman-desktop

Start podman-desktop, init and start podman machine. I disabled "Machine with root privileges:", but it probably doesn't matter.

I have the same issue on my M1 and these instructions does not work for me. =(

[robahtou]

I've been having this issue on/off.

What got it working consistently for me is very similar to @FullofQuarks

If you already have podman-desktop installed make sure to delete it as well:

podman machine rm  
brew uninstall podman-desktop podman qemu  
rm -rf ~/.config/containers  
rm -rf ~/.local/share/containers
## <didn't do this>brew install podman-desktop

Instead of installing podman-desktop via brew I downloaded and installed the arm .dmg file (follow the instructions) and so far no issues (so far!!). Will report back if the issue resurfaces.

[rhatdan]

You could try this with the apple hypervisor, which we plan to move with to see if it works better. Qemu has proven to be not as stable as we like.

If you have a recent Podman, I believe you can do.

export CONTAINERS_MACHINE_PROVIDER=applehv
podman machine init

[robahtou]

@rhatdan Just tried your suggestion and now I'm getting this error:

➜  ~ podman machine init machine
Extracting compressed file: machine_fedora-coreos-39.20231204.2.1-applehv.aarch64.raw: done. 
Error: could not find "vfkit" in one of {[$BINDIR/../libexec/podman /usr/local/opt/podman/libexec/podman /opt/homebrew/opt/podman/libexec/podman /opt/homebrew/bin /usr/local/bin /usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman] {<nil>}}.  
To resolve this error, set the helper_binaries_dir key in the `[engine]` section of containers.conf to the directory containing your helper binaries.

Not sure what this means 😓

[denisborisov]

You could try this with the apple hypervisor, which we plan to move with to see if it works better. Qemu has proven to be not as stable as we like.

If you have a recent Podman, I believe you can do.

export CONTAINERS_MACHINE_PROVIDER=applehv
podman machine init

installed vfkit via this doc: https://github.com/crc-org/vfkit
but the result of podman machine start is Error: Get "http://localhost:8081/vm/state": EOF

[lockcrocodilegiraffe]

I've been having this issue on/off.

What got it working consistently for me is very similar to @FullofQuarks

If you already have podman-desktop installed make sure to delete it as well:

podman machine rm  
brew uninstall podman-desktop podman qemu  
rm -rf ~/.config/containers  
rm -rf ~/.local/share/containers
## <didn't do this>brew install podman-desktop

Instead of installing podman-desktop via brew I downloaded and installed the arm .dmg file (follow the instructions) and so far no issues (so far!!). Will report back if the issue resurfaces.

Sadly, this hasn't worked for me (Apple M2 Pro). I get "Error: Command execution failed with exit code 125"

[szafranek]

I worked around a recent instance of this error by following these instructions and downgrading qemu from 8.2.0 to 8.1.3:
#21088 (comment)

[dneto82]

I've been having this issue on/off.

What got it working consistently for me is very similar to @FullofQuarks

If you already have podman-desktop installed make sure to delete it as well:

podman machine rm  
brew uninstall podman-desktop podman qemu  
rm -rf ~/.config/containers  
rm -rf ~/.local/share/containers
## <didn't do this>brew install podman-desktop

Instead of installing podman-desktop via brew I downloaded and installed the arm .dmg file (follow the instructions) and so far no issues (so far!!). Will report back if the issue resurfaces.

Also works for me this way. Removed everything via brew and letting podman-desktop dmg setup everything back. =/

[lestex]

I'm experiencing this on M1 pro as well, a downgrade of qemu to 8.1.3 helped, so I'll stick to it for a time being.

[cbr7]

Same for me, just had the same problem on a M1 PRO after updating to podman 4.9.0. After several uninstalls, restarts and reinstalls I finally got it to work and the podman machine to start up correctly.

[andremayer]

I'm experiencing this on M1 pro as well, a downgrade of qemu to 8.1.3 helped, so I'll stick to it for a time being.

Hi @lestex Can you share how you downgraded qemu back to 8.1.3? I tried running this on my macOS but didn't work:

❯ brew install qemu@8.1.3
Warning: No available formula with the name "qemu@8.1.3".
==> Searching for similarly named formulae and casks...
Error: No formulae or casks found for qemu@8.1.3.

:(

[cfergeau]

For what it's worth, since this is an issue related to the qemu build available in brew, the discussion is better suited for this homebrew issue: Homebrew/homebrew-core#159971

[mobini0]

Removing everything, and installing dmg podman-desktop worked for me.
Here's the exact steps I took.

1. First remove any machines you have

podman machine rm --force

2. Then remove everything you have installed from homebrew (don't include podman-desktop if you haven't installed it)

brew uninstall podman-desktop podman qemu  

3. Then remove all podman caches:

rm -rf ~/.config/containers  
rm -rf ~/.local/share/containers
rm ~/.ssh/podman*

4. Then download podman desktop dmg from: https://podman-desktop.io/downloads

5. Then install podman dmg, It will prompt you that podman is not installed, follow the instruction and install it through podman-desktop. Then it will ask you to create a VM. Follow the instructions to create a VM. Wait for the machine to start.

Now you can go back to developing

[benz0li]

@macports users: Stay with QEMU v8.0.5; do not update to v8.2.0.

Cross reference: https://trac.macports.org/ticket/67731#comment:25

[andremayer]

@macports users: Stay with QEMU v8.0.5; do not update to v8.2.0.

Cross reference: https://trac.macports.org/ticket/67731#comment:25

Hi @benz0li how are you? Can you share the brew install instructions for this QEMU specific version v8.0.5? I got mine upgraded to latest 8.2.0 and now it is no longer working due this "Waiting for VM" issue. The command below will not work, correct?

brew install qemu@8.0.5

[benz0li]

Can you share the brew install instructions for this QEMU specific version v8.0.5?

@andremayer I am using MacPorts and not Homebrew.

[slemeur]

Along with the comment here: #20776 (comment) this is what I've done and worked on my side for M3.

If you are on M1/M2, the recommended path is to update to Podman 4.9.

1 - Remove everything podman and qemu to get a clean environment:

• remove eventual installation from podman/podman desktop installer from sudo rm -rf opt/podman
• remove brew installations
  • brew uninstall podman-desktop
  • brew uninstall podman
  • brew uninstall qemu
• cleanup of podman files
  • rm -rf ~/.ssh/podman-machine-default
  • rm -rf ~/.ssh/podman-machine-default.pub
  • rm -rf ~/.local/share/containers
  • rm -rf ~/.config/containers

2 - Reinstall using brew

• `brew install podman``

3 - Install patched EDK2

• Make sure you have bunzip2 (if not brew install bzip2)
• Make sure QEMU is installed at the following path /opt/homebrew/Cellar/qemu/8.2.0 (if you are using a different path, just adapt the command below)
• curl -sL https://github.com/AkihiroSuda/qemu/raw/704f7cad5105246822686f65765ab92045f71a3b/pc-bios/edk2-aarch64-code.fd.bz2 | bunzip2 > /opt/homebrew/Cellar/qemu/8.2.0/share/qemu/edk2-aarch64-code.fd

Now you should be able to do podman machine init and podman machine start

You can then reinstall Podman Desktop (either brew or Podman Desktop website)

The situation should get better with the upcoming release of QEMU 8.2.1

Note: If you are on M1/M2, you should be fine with the latest release of Podman 4.9. If not, please raise it!

[fuzzball81]

I am using a M1 with Podman 4.9 installed via brew. I am still seeing the issue. I have not tried to patch qemu like suggested as above. I did make sure to remove my old vm and re-ran podman machine init before running podman machine start

[benz0li]

@slemeur @fuzzball81 Please check HFS+/APFS compression of edk2-aarch64-code.fd. Most likely this is the problem.

See https://trac.macports.org/ticket/67731#comment:14

[fuzzball81]

@benz0li I am using homebrew but checked as you have recommended and got the following
afsctool -l /opt/homebrew/share/qemu/edk2-aarch64-code.fd /opt/homebrew/share/qemu/edk2-aarch64-code.fd is not HFS+/APFS compressed.

[lestex]

I'm experiencing this on M1 pro as well, a downgrade of qemu to 8.1.3 helped, so I'll stick to it for a time being.

Hi @lestex Can you share how you downgraded qemu back to 8.1.3? I tried running this on my macOS but didn't work:

❯ brew install qemu@8.1.3
Warning: No available formula with the name "qemu@8.1.3".
==> Searching for similarly named formulae and casks...
Error: No formulae or casks found for qemu@8.1.3.

:(

Like here #21088 (comment)

[adrian-moisa]

I had the same issue on Mac M1 Pro. "Waiting for VM". Got it working by reverting to qemu 8.1.3. It started using latest podman 4.9.0.

• podman machine init -m 3072 - tried it as suggested, did not help.
• I tried several earlier podman versions up to 4.8.0. No luck.
• podman machine start --log-level=debug - Retrieve more info about the failure. Thanks to this command I got out this error message from qemu: Synchronous Exception at 0x00000000BC562000 ASSERT [ArmCpuDxe] /home/kraxel/projects/qemu/roms/edk2/ArmPkg/Library/DefaultExceptionHandlerLib/Arch64/DefaultExceptionHandler.c(343): ((BOOLEAN)(0==1))
• Following hints from here: Some people suggest reverting to an older qemu - qemu ASSERT [ArmCpuDxe] DefaultExceptionHandler.c:333 on Mac M3, Synchronous Exception at 0x00000000BC562000
• I managed to get it running by reverting qemu to 8.1.3 qemu.rb. Reverting brew version instructions here: How To Install Older Versions of Homebrew Packages @andremayer

Edit:
Looks like the answer is buried in many answers. A plea to the original author @peter-crist . Please append the description to let people know that there's are 2 ways to fix: via the rollback of qemu or via applehv provider. I wasted 3 hours looking for the answer. Had it been in the first post it would have been only 30 mins lost. Cheers!

[fuzzball81]

I can confirm using the 8.1.3 version of qemu does allow the VM to start on a M1 on macOS 14.3 with podman 4.9.

[cfergeau]

QEMU 8.2.1 has been released and is available in brew. Hopefully this fixes this issue.

[robahtou]

Can confirm that the update to QEMU 8.2.1 fixed the issue!!
I made sure to start with a clean slate so removed all the programs and folders/files as has been stated above.

[benoitschipper]

Hey All,

Just did the following to make this work without downgrading (Thanks @cfergeau and @robahtou for the mention's):

Remove the Machine:

podman machine rm
...
Are you sure you want to continue? [y/N] y

# Press y to make sure to remove everything

Upgrade to the latest QEMU:

brew update && brew upgrade
...
==> Upgrading qemu
  8.2.0 -> 8.2.1

Re-Init your Podman Machine:

Extracting compressed file: podman-machine-default_fedora-coreos-39.20240112.2.0-qemu.aarch64.qcow2: done
Image resized.
Machine init complete
To start your machine run:

	podman machine start

Start your Podman Machine:

Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Users:/Users
Mounting volume... /private:/private
Mounting volume... /var/folders:/var/folders

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

        podman machine set --rootful

API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.

Machine "podman-machine-default" started successfully

This worked for me on M1, M2 and M3! 🥳

[dedenf]

just want to update my issue, after upgrading QEMU to 8.2.1, the problem seems go away, tried using applehv but somehow if i did not set the amount of disk, it will took all of the available space and crippling my system, so upgrading QEMU works 👍 .

now i can pull and operate as usual

  version:
  APIVersion: 4.8.3
  Built: 1704291040
  BuiltTime: Wed Jan  3 21:10:40 2024
  GitCommit: ""
  GoVersion: go1.21.5
  Os: linux
  OsArch: linux/arm64
  Version: 4.8.3

QEMU emulator version 8.2.1
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers

[cfergeau]

it will took all of the available space and crippling my system

I don't know what you saw exactly, but while it's true applehv will create a 100GB disk image, it is a sparse image, and it won't use 100GB on disk unless it's filled.
It's doing the equivalent of:

$ truncate -s 100G foo.img
$ du -sh foo.img
0

[dedenf]

@cfergeau so the system prompt me with warning that i had low free space after initiatingpodman machine init using applehv as its image, during the extraction it took more than 100GB, i had like 130GB+ free space at that time, but all of them taken by the extraction.

[cfergeau]

@cfergeau so the system prompt me with warning that i had low free space after initiatingpodman machine init using applehv as its image, during the extraction it took more than 100GB, i had like 130GB+ free space at that time, but all of them taken by the extraction.

Turns out podman did not create sparse files when extracting its disk images, but that's been solved in the main branch.

[thiner]

I followed @benoitschipper's guide, fixed the problem. One more step I had to do is that I uninstalled homebrew and then install it back. The reason is that I migrated my Mac system from previous Intel cpu Mac Book Pro to a new M3 Mac Book Pro. When I ran the podman machine start command, it prompted that my qemu is a x86 binary. I tried to reinstall qemu but that didn't work. I solved the problem by purge existing homebrew, and reinstall it.

[Luap99]

podman 5.0 no longer supports qemu on macos and uses the apple hypervisor by default.