Podman VM not starting on Arm mac #21096

userbradley · 2023-12-27T21:22:00Z

Issue Description

I was cleaning up and did a podman machine rm and then created a new machine, and now it wont start. It hangs with the below

When running podman machine start I get the below

➜ podman machine start --log-level=debug
INFO[0000] podman filtering at log level debug          
DEBU[0000] Using Podman machine with `qemu` virtualization provider 
Starting machine "podman-machine-default"
DEBU[0000] {true 1500 map[forward-dest:[/run/user/502/podman/podman.sock] forward-identity:[/Users/bradleystannard/.ssh/podman-machine-default] forward-sock:[/Users/bradleystannard/.local/share/containers/podman/machine/qemu/podman.sock] forward-user:[core]] [] map[listen-qemu:unix:///var/folders/8h/xxy2fqyn5v3f94msz5m2c0980000gp/T/podman/qmp_podman-machine-default.sock] /var/folders/8h/xxy2fqyn5v3f94msz5m2c0980000gp/T/podman/podman-machine-default_proxy.pid 52996} 
DEBU[0000] qemu cmd: [/opt/homebrew/bin/qemu-system-aarch64 -accel hvf -accel tcg -cpu host -M virt,highmem=on -drive file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -drive file=/Users/bradleystannard/.local/share/containers/podman/machine/qemu/podman-machine-default_ovmf_vars.fd,if=pflash,format=raw -m 2048 -smp 5 -fw_cfg name=opt/com.coreos/config,file=/Users/bradleystannard/.config/containers/podman/machine/qemu/podman-machine-default.ign -qmp unix:/var/folders/8h/xxy2fqyn5v3f94msz5m2c0980000gp/T/podman/qmp_podman-machine-default.sock,server=on,wait=off -netdev socket,id=vlan,fd=3 -device virtio-net-pci,netdev=vlan,mac=5a:94:ef:e4:0c:ee -device virtio-serial -chardev socket,path=/var/folders/8h/xxy2fqyn5v3f94msz5m2c0980000gp/T/podman/podman-machine-default_ready.sock,server=on,wait=off,id=apodman-machine-default_ready -device virtserialport,chardev=apodman-machine-default_ready,name=org.fedoraproject.port.0 -pidfile /var/folders/8h/xxy2fqyn5v3f94msz5m2c0980000gp/T/podman/podman-machine-default_vm.pid -virtfs local,path=/Users,mount_tag=vol0,security_model=none -virtfs local,path=/private,mount_tag=vol1,security_model=none -virtfs local,path=/var/folders,mount_tag=vol2,security_model=none -drive if=virtio,file=/Users/bradleystannard/.local/share/containers/podman/machine/qemu/podman-machine-default_fedora-coreos-39.20231204.2.1-qemu.aarch64.qcow2] 
Waiting for VM ...

And running with debug, the quemu VM hangs:

I've tried the below

#10824 (comment)

But this has not worked.

Steps to reproduce the issue

Delete Podman VM on ARM mac

Create new one

podman machine init
podman machine start

Describe the results you received

Podman machine hangs with image posted above

Describe the results you expected

Podman machine runs

podman info output

➜ brew info podman        
==> podman: stable 4.8.2 (bottled), HEAD
Tool for managing OCI containers and pods

➜ podman info                                
OS: darwin/arm64
provider: qemu
version: 4.8.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional information

Darwin 11586.local 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:18 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6000 arm64 arm

The text was updated successfully, but these errors were encountered:

userbradley · 2023-12-27T21:29:40Z

I have a feeling this is a QEMU issue, but I'm not 100% sure here

arixmkii · 2023-12-28T11:52:50Z

@userbradley you can find workarounds discussed in another issue #21088 (comment)

It is not a Podman issue per se, but probably it is worth to keep one of these issues open for others to have references to the workaround.

mqj0712 · 2023-12-30T15:42:08Z

I have the same issue, and also tried some methods discussed in #21088 (comment), none of them works, but i found a clue in qemu issue 1990, which point to a edk2 firmware: edk2-aarch64-code.fd.bz2, I download it extract the file edk2-aarch64-code.fd ,and then replace the one in /opt/homebrew/share/qemu, then remove the old podman machine, and run the "podman machine init" and "podman machine start", and the vm started as expected.

tebeco · 2024-01-10T07:31:39Z

TLDR: "i want something i can just run to patch it"

# pwd setup
mkdir ~/edk2-fix
cd edk2-fix

# download bz2
curl -LO https://gitlab.com/kraxel/qemu/-/raw/704f7cad5105246822686f65765ab92045f71a3b/pc-bios/edk2-aarch64-code.fd.bz2

# extract bz2
$ bzip2 -d edk2-aarch64-code.fd.bz2

# backup
$ cp /opt/homebrew/share/qemu/edk2-aarch64-code.fd ./edk2-aarch64-code.fd.old

# temporary manual patch of the edk2 fd file
$ cp ./edk2-aarch64-code.fd /opt/homebrew/share/qemu/edk2-aarch64-code.fd

# clean podman
$ podman machine stop
$ podman machine rm    # validate the prompt by Pressing 'y' + 'ENTER'

# reinit and and VM
$ podman machine init
$ podman machine start

michaelvanstraten · 2024-01-10T19:50:54Z

Could this be QEMU's DNS not working on macOS?

michaelvanstraten · 2024-01-10T19:51:59Z

Anyway, I am running in the same issue here, can you print QEMU debug information?

tebeco · 2024-01-10T20:42:02Z

Anyway, I am running in the same issue here, can you print QEMU debug information?

#21096 (comment)

that's how you Temp fix it until EDk2 is fixed
which QEMU will have to update in their next release
which podman will have to update at well

currently QEMU is not yet depending on the FIXED version of EDK2
read the very last comments of for more info:
https://gitlab.com/qemu-project/qemu/-/issues/1990

baigzeeshan · 2024-01-11T20:34:42Z

TLDR: "i want something i can just run to patch it"

# pwd setup
mkdir ~/edk2-fix
cd edk2-fix

# download bz2
curl -LO https://gitlab.com/kraxel/qemu/-/raw/704f7cad5105246822686f65765ab92045f71a3b/pc-bios/edk2-aarch64-code.fd.bz2

# extract bz2
$ bzip2 -d edk2-aarch64-code.fd.bz2

# backup
$ cp /opt/homebrew/share/qemu/edk2-aarch64-code.fd ./edk2-aarch64-code.fd.old

# temporary manual patch of the edk2 fd file
$ cp ./edk2-aarch64-code.fd /opt/homebrew/share/qemu/edk2-aarch64-code.fd

# clean podman
$ podman machine stop
$ podman machine rm    # validate the prompt by Pressing 'y' + 'ENTER'

# reinit and and VM
$ podman machine init
$ podman machine start

This fixed the issue on M3 MacBook Pro... Thanks

JamesKunstle · 2024-01-12T04:44:08Z

Having the same problem, including asynchronous exception in qemu when starting a machine with --log-level DEBUG.

➜  ~ brew info podman
==> podman: stable 4.8.3

➜  ~ podman info
OS: darwin/arm64
provider: qemu
version: 4.8.3

vhscom · 2024-01-12T22:01:33Z

TLDR: "i want something i can just run to patch it"

Nice hack. Tested and working on M1 with success after reinstalling podman had no effect.

carlca · 2024-01-17T10:23:46Z

The Nice Hack didn't work for me. podman machine init is still downloading the x86_64 stuff...
Extracting compressed file: podman-machine-default_fedora-coreos-39.20240104.2.0-qemu.x86_64.qcow2: done

arixmkii · 2024-01-17T10:52:20Z

podman machine init is still downloading the x86_64 stuff...

Could it be that you are using homebrew and that your homebrew was brought to your Apple Silicon mac from an older Intel machine via backup restore? Podman will download x86_64, when you are running app compiled for the Intel platform, so, it could be that you have launched Intel app of Podman via Rosetta2.

tebeco · 2024-01-17T12:14:44Z

i think it's because you need to clear the local cache from qemu which i might have forgot in the script above

else it's "downloading but not really" the qcow image from your already corrupted cache and you're cycling back again @carlca

carlca · 2024-01-17T16:55:07Z

Could you tell me how to "clear the qemu cache". I did search but didn't find anything that worked...

tebeco · 2024-01-17T21:36:57Z

iirc i found the file name on disk with the exact name in your error that it's extracting in a subfolder named "cache" so i deleted it
you also need to to pod an machine delete before

the error you have:

Extracting compressed file: podman-machine-default_fedora-coreos-39.20240104.2.0-qemu.x86_64.qcow2

i would try to find that file on disk.

i'm not on a computer anymore but i found it by looking into qemu folders

probably saw someone in all the link above mentioning it

but i'm sure it's in a cache folder with qcow2 extension

edit:
this might help

carlca · 2024-01-17T23:43:57Z

Thanks for the reply tebeco. I use find any file and found the very file you mentioned, and deleted it. I then did a podman machine rm followed by a podman init. It appears to be downloading the fedora-coreos-39.20240112.2.0-qemu.x86_64.qcow2.xz again. I've tried asking in the Podman Discord but there has been no help there, either!

carlca · 2024-01-17T23:44:04Z

Thanks for the reply tebeco. I use find any file and found the very file you mentioned, and deleted it. I then did a podman machine rm followed by a podman init. It appears to be downloading the fedora-coreos-39.20240112.2.0-qemu.x86_64.qcow2.xz again. I've tried asking in the Podman Discord but there has been no help there, either!

arixmkii · 2024-01-18T09:57:56Z

@carlca Could you provide output of which podman and file $(which podman) commands? I have strong belief that you have issues with your Podman installation.

Here is a sample from my machine, which indicates, that it's a homebrew installation for arm64 arch

% which podman
/opt/homebrew/bin/podman
% file $(which podman)
/opt/homebrew/bin/podman: Mach-O 64-bit executable arm64

carlca · 2024-01-18T12:44:07Z

Interesting! I'm not aware of been given a choice of architecture when I first installed podman...

which podman
/opt/podman/bin/podman
(base)
[Thu Jan 18 2024 12:41pm (GMT+0000)]  via  v18.16.1
~
file $(which podman)
/opt/podman/bin/podman: Mach-O 64-bit executable x86_64

arixmkii · 2024-01-18T12:52:49Z

@carlca It looks like official Podman package. Please try reinstalling latest for your actual arch from https://github.com/containers/podman/releases/. For macos there are 2 packages, one for amd64 and one for arm64 and they have some settings builtin during compile time w/o future checks in runtime. So, it is important to install the right one.

carlca · 2024-01-18T13:30:01Z

Thanks arixmkii - it's all sorted and working now 😃

lacosteque · 2024-01-20T10:54:35Z

TLDR: "i want something i can just run to patch it"

# pwd setup
mkdir ~/edk2-fix
cd edk2-fix

# download bz2
curl -LO https://gitlab.com/kraxel/qemu/-/raw/704f7cad5105246822686f65765ab92045f71a3b/pc-bios/edk2-aarch64-code.fd.bz2

# extract bz2
$ bzip2 -d edk2-aarch64-code.fd.bz2

# backup
$ cp /opt/homebrew/share/qemu/edk2-aarch64-code.fd ./edk2-aarch64-code.fd.old

# temporary manual patch of the edk2 fd file
$ cp ./edk2-aarch64-code.fd /opt/homebrew/share/qemu/edk2-aarch64-code.fd

# clean podman
$ podman machine stop
$ podman machine rm    # validate the prompt by Pressing 'y' + 'ENTER'

# reinit and and VM
$ podman machine init
$ podman machine start

Thank you, this solution helped on m1pro
After the last update, podman via brew encountered this problem.

tebeco · 2024-02-07T05:03:57Z

for anyone reading it I think it got fixed BTW

brew update
brew upgrade

podman machine stop
podman machine rm
podman machine init

podman machine start

Looking carefully at the date we can see the qcow2 image is now dated 28/01, the previous one I had was around 04/01 (yes my screenshot should otherwise because i didn't screened the first time)

Luap99 · 2024-04-04T09:40:47Z

podman 5.0 now uses the apple hypervisor and no longer support qemu

userbradley added the kind/bug Categorizes issue or PR as related to a bug. label Dec 27, 2023

arixmkii mentioned this issue Jan 2, 2024

[MacOS] podman machine start hangs indefinitely after reinstallation #21120

Closed

Luap99 mentioned this issue Jan 5, 2024

podman machine start stuck in podman 4.8.3 on Apple M1 #21169

Closed

arixmkii mentioned this issue Jan 9, 2024

podman struck at starting machine in macos #21205

Closed

tim3003 mentioned this issue Jan 12, 2024

podman machine start hangs at "Waiting for VM" (Mac Sonoma 14.1.2 M1 Pro Chip) #21237

Closed

arixmkii mentioned this issue Jan 26, 2024

Error: qemu exited unexpectedly with exit code -1, stderr: qemu-system-x86_64: Unknown Error #21375

Closed

Luap99 closed this as not planned Won't fix, can't repro, duplicate, stale Apr 4, 2024

stale-locking-app bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Jul 5, 2024

stale-locking-app bot locked as resolved and limited conversation to collaborators Jul 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Podman VM not starting on Arm mac #21096

Podman VM not starting on Arm mac #21096

userbradley commented Dec 27, 2023 •

edited

Loading

userbradley commented Dec 27, 2023

arixmkii commented Dec 28, 2023

mqj0712 commented Dec 30, 2023

tebeco commented Jan 10, 2024

michaelvanstraten commented Jan 10, 2024

michaelvanstraten commented Jan 10, 2024

tebeco commented Jan 10, 2024 •

edited

Loading

baigzeeshan commented Jan 11, 2024

JamesKunstle commented Jan 12, 2024

vhscom commented Jan 12, 2024

carlca commented Jan 17, 2024

arixmkii commented Jan 17, 2024

tebeco commented Jan 17, 2024

carlca commented Jan 17, 2024

tebeco commented Jan 17, 2024 •

edited

Loading

carlca commented Jan 17, 2024

carlca commented Jan 17, 2024

arixmkii commented Jan 18, 2024 •

edited

Loading

carlca commented Jan 18, 2024

arixmkii commented Jan 18, 2024

carlca commented Jan 18, 2024

lacosteque commented Jan 20, 2024

tebeco commented Feb 7, 2024

Luap99 commented Apr 4, 2024

Podman VM not starting on Arm mac #21096

Podman VM not starting on Arm mac #21096

Comments

userbradley commented Dec 27, 2023 • edited Loading

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

userbradley commented Dec 27, 2023

arixmkii commented Dec 28, 2023

mqj0712 commented Dec 30, 2023

tebeco commented Jan 10, 2024

michaelvanstraten commented Jan 10, 2024

michaelvanstraten commented Jan 10, 2024

tebeco commented Jan 10, 2024 • edited Loading

baigzeeshan commented Jan 11, 2024

JamesKunstle commented Jan 12, 2024

vhscom commented Jan 12, 2024

carlca commented Jan 17, 2024

arixmkii commented Jan 17, 2024

tebeco commented Jan 17, 2024

carlca commented Jan 17, 2024

tebeco commented Jan 17, 2024 • edited Loading

carlca commented Jan 17, 2024

carlca commented Jan 17, 2024

arixmkii commented Jan 18, 2024 • edited Loading

carlca commented Jan 18, 2024

arixmkii commented Jan 18, 2024

carlca commented Jan 18, 2024

lacosteque commented Jan 20, 2024

tebeco commented Feb 7, 2024

Luap99 commented Apr 4, 2024

userbradley commented Dec 27, 2023 •

edited

Loading

tebeco commented Jan 10, 2024 •

edited

Loading

tebeco commented Jan 17, 2024 •

edited

Loading

arixmkii commented Jan 18, 2024 •

edited

Loading