Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[macos] podman machine start on Apple M1 uses qemu with hvf which is not yet supported by qemu on aarch64 #10577

Closed
thomasw64 opened this issue Jun 6, 2021 · 29 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@thomasw64
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman machine start and the other podman machine commands use not supported qemu accelerator hvf on aarch64/darwin (MacOS 11.4 on M1).

podman/pkg/machine/qemu/options_darwin_arm64.go adds to the invocation of qemu --accel hvf.

The standard qemu built from brew formula did not contain hvf support, so I rebuilt qemu with configure option --enable-hvf which did not result in the expected outcome. Verifying the binary with
qemu-system-aarch64 -accel help states only tcg supported. I reviewed the qemu source, and as it looks there is no implementation of hvf for aarch64 in qemu yet.

If I read it correctly the qemu code for target arm or aarch64 does not contain any calls to the Hypervisor Framework (hvf).
See https://gitlab.com/qemu-project/qemu/-/tree/master/target/i386/hvf vs https://gitlab.com/qemu-project/qemu/-/tree/master/target/arm

Steps to reproduce the issue:

  1. Create machine
# podman machine init                                                                              
Extracting compressed file
  1. Verify the machine
# podman machine list                                                                                  
NAME                     VM TYPE     CREATED             LAST UP
podman-machine-default*  qemu        About a minute ago  About a minute ago
  1. Start the machine
# podman machine start                                                                                         
Waiting for VM ...
qemu-system-aarch64: -accel hvf: invalid accelerator hvf
Error: dial unix /var/folders/cm/5dyqbf0s0ld37xyl34xyb_100000gn/T/podman/podman-machine-default_ready.sock: connect: connection refused

Describe the results you received:

qemu reports that hvf is an invalid accelerator.

Describe the results you expected:

Machine is started with the right qemu parameters.

Additional information you deem important (e.g. issue happens only occasionally):

Issue happens every time.

Output of podman version:

# podman --version                                                                                            
podman version 3.2.0

Output of podman info --debug:

# podman info --debug                                                                                          
Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: Get "http://d/v3.2.0/libpod/_ping": dial unix ///var/folders/cm/5dyqbf0s0ld37xyl34xyb_100000gn/T/podman-run--1/podman/podman.sock: connect: no such file or directory

Package info (e.g. output of rpm -q podman or apt list podman):

Output of brew list podman

/opt/homebrew/Cellar/podman/3.2.0/bin/podman
/opt/homebrew/Cellar/podman/3.2.0/etc/bash_completion.d/podman
/opt/homebrew/Cellar/podman/3.2.0/share/fish/vendor_completions.d/podman.fish
/opt/homebrew/Cellar/podman/3.2.0/share/man/ (158 files)
/opt/homebrew/Cellar/podman/3.2.0/share/zsh/site-functions/_podman

Output of brew info podman

podman: stable 3.2.0 (bottled)
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/3.2.0 (167 files, 29.3MB) *
  Poured from bottle on 2021-06-06 at 09:38:46
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✘, go-md2man ✘
==> Caveats
fish completions have been installed to:
  /opt/homebrew/share/fish/vendor_completions.d
==> Analytics
install: 2,316 (30 days), 9,193 (90 days), 26,804 (365 days)
install-on-request: 2,315 (30 days), 9,190 (90 days), 26,365 (365 days)
build-error: 0 (30 days)

Output of brew list qemu

/opt/homebrew/Cellar/qemu/6.0.0/bin/elf2dmp
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-edid
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-img
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-io
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-nbd
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-storage-daemon
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-aarch64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-alpha
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-arm
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-avr
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-cris
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-hppa
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-i386
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-m68k
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-microblaze
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-microblazeel
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-mips
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-mips64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-mips64el
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-mipsel
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-moxie
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-nios2
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-or1k
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-ppc
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-ppc64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-riscv32
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-riscv64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-rx
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-s390x
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-sh4
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-sh4eb
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-sparc
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-sparc64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-tricore
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-x86_64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-xtensa
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-xtensaeb
/opt/homebrew/Cellar/qemu/6.0.0/share/applications/qemu.desktop
/opt/homebrew/Cellar/qemu/6.0.0/share/icons/ (10 files)
/opt/homebrew/Cellar/qemu/6.0.0/share/qemu/ (107 files)

Output of brew info qemu

qemu: stable 6.0.0 (bottled), HEAD
Emulator for x86 and PowerPC
https://www.qemu.org/
/opt/homebrew/Cellar/qemu/6.0.0 (161 files, 569.4MB) *
  Built from source on 2021-06-06 at 11:15:13
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/qemu.rb
License: GPL-2.0-only
==> Dependencies
Build: libtool ✔, meson ✔, ninja ✔, pkg-config ✔
Required: glib ✔, gnutls ✔, jpeg ✔, libpng ✔, libslirp ✔, libssh ✔, libusb ✔, lzo ✔, ncurses ✔, nettle ✔, pixman ✔, snappy ✔, vde ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 14,340 (30 days), 40,554 (90 days), 122,362 (365 days)
install-on-request: 14,262 (30 days), 40,376 (90 days), 121,152 (365 days)
build-error: 0 (30 days)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Hardware: Mac Mini M1 16G
ProductName: macOS
ProductVersion: 11.4
BuildVersion: 20F71

@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Jun 6, 2021
@thomasw64
Copy link
Author

I was able to start the machine, by changing -accel hvf to -accel tcg in .config/containers/podman/machine/qemu/podman-machine-default.json

...
  "-device",
  "virtserialport,chardev=podman-machine-default_ready,name=org.fedoraproject.port.0",
  "-accel",
  "tcg",
  "-cpu",
  "cortex-a57",
...

output of podman machine start

Waiting for VM ...
qemu-system-aarch64: -accel tcg: qemu_mprotect__osdep: mprotect failed: Permission denied

output of podman machine list

NAME                     VM TYPE     CREATED        LAST UP
podman-machine-default*  qemu        2 minutes ago  Currently running

Now, it looks, I found another issue: The coreos image contains (34.20210601.dev.0) podman 3.1.2 instead of 3.2.0.

@mheon
Copy link
Member

mheon commented Jun 6, 2021

This is expected - we're waiting on a few more patches to merge upstream in Qemu for this to be supported. podman machine is still in heavy development and very much experimental - we hope to have things more polished and ready in the next few releases.

The Podman issue may be my fault, I'll bring it up at scrum on Monday. 3.1.2 and 3.2.0 should be able to talk to each other. We are needing to put out a Podman 3.2.1 already for another bug, so fixing this as the same time is probably not a big deal.

@baude
Copy link
Member

baude commented Jun 7, 2021

stay tuned, I will have instructions for you shortly.

@github-actions
Copy link

github-actions bot commented Jul 8, 2021

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Jul 8, 2021

@baude any update?

@github-actions
Copy link

github-actions bot commented Aug 8, 2021

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Aug 9, 2021

@baude another month, and no merge?

@baude
Copy link
Member

baude commented Aug 9, 2021

correct ... i have checked with Sergio but he is now on PTO.

@lpil
Copy link

lpil commented Sep 1, 2021

With the change in Docker's pricing today this feature becomes a lot more desired! It would be great to get this in the near future. Thank you

@baude
Copy link
Member

baude commented Sep 1, 2021

@lpil unfortunately,we are subject to the merging of qemu patches upstream. once that is done, we will have it all set.

@Conan-Kudo
Copy link

@baude What's the current state of the patchset for QEMU? Is there a link to the latest discussions about them that I can follow?

@baude
Copy link
Member

baude commented Sep 1, 2021

no upstream change, the submitter needs to make some changes and resubmit

@thomasw64
Copy link
Author

@baude circumventing the QEMU issue temporarily with updating the -accel hvf to -accel tcg is not a good choice? At least people can run podman machine with that on M1 Macs.

@thomasw64
Copy link
Author

Before 3.3.1 I could start the machine with fixing the -accel tcg. Now any podman machine command says:

Error: due to missing upstream patches, Apple Silicon is not capable of running Podman machine yet

I think it would be better running with less optimal acceleration than tell you cannot do it at all.

@simnalamburt
Copy link
Contributor

simnalamburt commented Sep 5, 2021

FYI, I have patched and packed both qemu and podman due to this issue. You can check this out with the command below:

brew install simnalamburt/x/podman-apple-silicon

# Example
podman machine init
podman machine start
podman run -p 8080:80 docker.io/nginx

References

@jonpspri
Copy link
Contributor

jonpspri commented Sep 5, 2021

@simnalamburt I'm working on a PR with a podman patch for the podman side of the equation. I'll include awareness of /opt/homebrew directories (which I expect will create a separate firestorm) and backwards compatibility with non-HVF qemu.

Having said that, your qemu patches look much farther along than my experiment :). I suppose we'll have to see where the slow-moving upstream review of all that is at.

@simnalamburt
Copy link
Contributor

These are the patch that I used.

qemu: https://github.com/simnalamburt/qemu/tree/hvf, it's based on qemu 6.1.0
podman: https://github.com/simnalamburt/podman, it's based on the main branch of podman

@simnalamburt
Copy link
Contributor

@simnalamburt I'm working on a PR with a podman patch for the podman side of the equation.

By the way can I take a look of your podman patch? Maybe we can collaborate with it.

My PR is currently at #11449

@jonpspri
Copy link
Contributor

jonpspri commented Sep 5, 2021

Hi @simnalamburt . My podman patch is in https://github.com/jonpspri/podman/tree/homebrew.

I did what I could to make it backwards-compatible. I was working from head rather than 3.3.1, but I don't think that'll make much difference on a diff. Feel free to grab as much as you like and work it into your PR, since you've already started it. I'll go pull and build your qemu and re-test on my hardware.

@jonpspri
Copy link
Contributor

jonpspri commented Sep 5, 2021

@simnalamburt I looked at #11449 . I think we should add looking for the BIOS image in multiple locations into that PR as well (see my tree); however, to workaround this particular issue we may wish to add a separate PR for having qemu fallback to TCG acceleration if HVF acceleration is not available. That's a one-liner I can set up in a separate PR.

[Note: I created PR #11453 to address the BIOS FD issue, since it's very narrowly focused. The gvproxy conversation seems much more complicated.]

jonpspri added a commit to jonpspri/podman that referenced this issue Sep 5, 2021
Cause qemu to fall back to using TCG acceleration when HVP acceleration
is not available on Darwin Aarch64.  Qemu prints a warning which it is
desirable to leave to embarrass the upstream Qemu into approving the HVF
patches.

Signed-off-by: Jonathan Springer <jspringer@us.ibm.com>
jonpspri added a commit to jonpspri/podman that referenced this issue Sep 5, 2021
Cause qemu to fall back to using TCG acceleration when HVP acceleration
is not available on Darwin Aarch64.  Qemu prints a warning which it is
desirable to leave to embarrass the upstream Qemu into approving the HVF
patches.

[NO TESTS NEEDED]

Signed-off-by: Jonathan Springer <jspringer@us.ibm.com>
jonpspri added a commit to jonpspri/podman that referenced this issue Sep 5, 2021
Cause qemu to fall back to using TCG acceleration when HVP acceleration
is not available on Darwin Aarch64.  Qemu prints a warning which it is
desirable to leave to embarrass the upstream Qemu into approving the HVF
patches.

[NO TESTS NEEDED]

Signed-off-by: Jonathan Springer <jspringer@us.ibm.com>
Signed-off-by: Jonathan Springer <jonpspri@gmail.com>
@mheon
Copy link
Member

mheon commented Sep 5, 2021

I believe @ashley-cui is also maintaining a Qemu build for Homebrew which includes the appropriate patches to run on M1.

jonpspri added a commit to jonpspri/podman that referenced this issue Sep 6, 2021
Cause qemu to fall back to using TCG acceleration when HVP acceleration
is not available on Darwin Aarch64.  Qemu prints a warning which it is
desirable to leave to embarrass the upstream Qemu into approving the HVF
patches.

[NO TESTS NEEDED]

Signed-off-by: Jonathan Springer <jspringer@us.ibm.com>
Signed-off-by: Jonathan Springer <jonpspri@gmail.com>
openshift-merge-robot added a commit that referenced this issue Sep 6, 2021
Qemu TCG Accel fallback for Apple Silicon. Iss #10577
flouthoc pushed a commit to flouthoc/podman that referenced this issue Sep 6, 2021
…hvf-tcg

Qemu TCG Accel fallback for Apple Silicon. Iss containers#10577

Signed-off-by: Aditya Rajan <arajan@redhat.com>
@jonpspri
Copy link
Contributor

jonpspri commented Sep 8, 2021

Patch #11451 worked around this by allowing qemu to fall back onto TCG (software) acceleration. I don't know whether we want to keep the Issue open to track upstream progress? @baude , is there a label for that?

@simnalamburt
Copy link
Contributor

With #11451 being merged, I think there're nothing more we should do in the podman side for the podman's M1 support except #11449 and #11480.

@hurricanehrndz
Copy link

hurricanehrndz commented Sep 15, 2021

@simnalamburt

Did you use Alexander's patch for hvf support?
https://patchew.org/search?q=hvf%3A+Implement+Apple+Silicon+Support

Looks like there is a recent update to the patch, changing how qemu would be called:

To use hvf support, please make sure to run -M virt,highmem=off to fit
in M1's physical address space limits and use -cpu host.

@simnalamburt
Copy link
Contributor

@simnalamburt

Did you use Alexander's patch for hvf support?
https://patchew.org/search?q=hvf%3A+Implement+Apple+Silicon+Support

Looks like there is a recent update to the patch, changing how qemu would be called:

To use hvf support, please make sure to run -M virt,highmem=off to fit
in M1's physical address space limits and use -cpu host.

Yes I used that patch and podman is already using -M virt,highmem=off and -cpu option.

mheon pushed a commit to mheon/libpod that referenced this issue Sep 20, 2021
Cause qemu to fall back to using TCG acceleration when HVP acceleration
is not available on Darwin Aarch64.  Qemu prints a warning which it is
desirable to leave to embarrass the upstream Qemu into approving the HVF
patches.

[NO TESTS NEEDED]

Signed-off-by: Jonathan Springer <jspringer@us.ibm.com>
Signed-off-by: Jonathan Springer <jonpspri@gmail.com>
@daniellaera
Copy link

why my config file is not create when I installed podman ?

~/.config/containers/podman/machine/qemu/podman-machine-default.json

is not present

@ashley-cui
Copy link
Member

Closing as we now have support for M1 via Homebrew's Qemu

@fourstepper
Copy link

@ashley-cui not sure if this is in the scope of this issue, but should it be possible to run and build linux/amd64 with podman on MacOS via brew's qemu, or is this a feature possibly coming at a later date?

judaew added a commit to macports/macports-ports that referenced this issue Dec 4, 2021
@benfro
Copy link

benfro commented Jul 8, 2023

Closing as we now have support for M1 via Homebrew's Qemu

Hi.
I'm desperately trying to get Podman installed via brew to work. Version 4.5.1 does not. Error is "qemu exited unexpectedly with exit code -1, stderr: qemu-system-x86_64: Unknown Error"
In what way is there "support for M1 via Homebrew's Qemu"?

I have been looking into posts for three weeks now and not one single one of thm seem to give a good solution to this issue. As far as I am concerned, there is a big issue with Podman, brew and qemu on M1 machines still.

Regards,

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Oct 7, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

No branches or pull requests