Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to add hypervisor entitlement to qemu #4

Closed
jandubois opened this issue May 15, 2021 · 11 comments · Fixed by #10
Closed

Unable to add hypervisor entitlement to qemu #4

jandubois opened this issue May 15, 2021 · 11 comments · Fixed by #10
Labels
documentation Improvements or additions to documentation

Comments

@jandubois
Copy link
Member

I've installed qemu via brew. As soon as I invoke codesign with the parameters specified by the README.md, qemu no longer works and just prints Killed: 9 regardless of which options are specified, e.g.

$ cp /usr/local/bin/qemu-system-x86_64 .

$ ./qemu-system-x86_64 --version
QEMU emulator version 5.2.0
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers

$ codesign -v -s - --entitlements entitlements.xml ./qemu-system-x86_64
./qemu-system-x86_64: signed Mach-O thin (x86_64) [qemu-system-x86_64-5555494422424fa5348c3b5a936d67231b727a83]

$ ./qemu-system-x86_64 --version
Killed: 9

$ cat entitlements.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <!-- for OS X 10.10 - macOS 10.15 -->
    <key>com.apple.vm.hypervisor</key>
    <true/>
    <!-- for macOS 11 and later -->
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>

$ codesign -dv qemu-system-x86_64
Executable=/Users/jan/Downloads/qemu-system-x86_64
Identifier=qemu-system-x86_64-5555494422424fa5348c3b5a936d67231b727a83
Format=Mach-O thin (x86_64)
CodeDirectory v=20100 size=100716 flags=0x2(adhoc) hashes=3139+5 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.7
BuildVersion:	19H1030

The failure is due to the entitlement, not qemu: when I sign a different binary with the same XML file, it fails the same way. Same thing with just the macOS 10.10+ entitlement:

<dict>
    <key>com.apple.vm.hypervisor</key>
    <true/>
</dict>

I can sign with a different entitlement without problem, e.g. this works just fine:

<dict>
    <key>com.apple.security.cs.disable-library-validation</key>
    <true/>
</dict>
@jandubois jandubois mentioned this issue May 15, 2021
Closed
@christian-korneck
Copy link
Contributor

@jandubois I ran into similar issues and this worked for me (macOS 11.3, Intel):

  • only the relevant key in the e.xml (not both):
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
  • brew install qemu
  • codesign --remove-signature /usr/local/bin/qemu-system-x86_64
  • codesign -s - --entitlements e.xml --force /usr/local/bin/qemu-system-x86_64

and this should not be necessary, but if you're still getting a killed: 9 error you could also try to set a gatekeeper exception:

  • spctl --add /usr/local/bin/qemu-system-x86_64

@AkihiroSuda AkihiroSuda added the documentation Improvements or additions to documentation label May 15, 2021
@jandubois
Copy link
Member Author

@christian-korneck Thanks for your comments, but they don't seem to make any difference to me.

  • I already trimmed the entitlement.xml to only include the version used by Catalina.
  • The qemu version installed by brew is unsigned, so --remove-signature did nothing.
  • The codesign "works", but afterwards I get the "Killed: 9" response
  • spctl -a already says "rejected" even for the unsigned binary, but the binary works fine.
  • Running spctl --add didn't change anything; spctl -a still says "Rejected" and running the signed binary still says "Killed: 9".

This is all very mystifying for me; maybe qemu needs to be notarized in addition to being signed?

I guess it is time for me to try this on BigSur to see if that makes a difference.

@jandubois
Copy link
Member Author

I guess it is time for me to try this on BigSur to see if that makes a difference.

Same problems on my Big Sur laptop. 😞

@bric3
Copy link

bric3 commented May 16, 2021 •
edited
Loading

For the record running codesign with the entitlement with both keys didn't raise an error.

$ cat >entitlements.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"\>
<plist version="1.0">
<dict>
    <!-- for OS X 10.10 - macOS 10.15 -->
    <key>com.apple.vm.hypervisor</key>
    <true/>
    <!-- for macOS 11 and later -->
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
EOF
$ codesign -s - --entitlements entitlements.xml --force /usr/local/bin/qemu-system-x86_64

However when starting the fedora image, I got issues to install ssh, sshfs and guest agent. Which I didn't link rught away to the qemu signing.

$ limactl start fedora.yaml      # or later with    limactl start fedora
INFO[0000] Using the existing instance "fedora"
INFO[0000] Starting QEMU
INFO[0000] SSH: 127.0.0.1:60024
INFO[0000] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0010] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0020] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0030] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0040] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0050] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0060] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0070] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0080] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0090] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0100] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0110] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0120] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0130] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0140] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0150] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0160] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0170] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0180] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0190] Waiting for the essential requirement 1 of 3: "ssh"
... same message for a while

Once I followed @christian-korneck 's #4 (comment), starting lima worked:

$ limactl start fedora
INFO[0000] Using the existing instance "fedora"
INFO[0000] Starting QEMU
INFO[0000] SSH: 127.0.0.1:60024
INFO[0000] Waiting for the essential requirement 1 of 3: "ssh"
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EDX.spec-ctrl [bit 26]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:EDX.rdtscp [bit 27]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EDX.spec-ctrl [bit 26]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:EDX.rdtscp [bit 27]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EDX.spec-ctrl [bit 26]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:EDX.rdtscp [bit 27]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EDX.spec-ctrl [bit 26]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:EDX.rdtscp [bit 27]
INFO[0010] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0020] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0020] The essential requirement 1 of 3 is satisfied
INFO[0020] Waiting for the essential requirement 2 of 3: "sshfs binary to be installed"
INFO[0020] The essential requirement 2 of 3 is satisfied
INFO[0020] Waiting for the essential requirement 3 of 3: "the guest agent to be running"
INFO[0020] The essential requirement 3 of 3 is satisfied
INFO[0020] Mounting "/Users/bric3"
INFO[0021] Mounting "/tmp/lima"
INFO[0021] Waiting for the optional requirement 1 of 1: "containerd binaries to be installed"
INFO[0021] Forwarding "/run/user/501/lima-guestagent.sock" (guest) to "/Users/bric3/.lima/fedora/ga.sock" (host)
INFO[0021] Forwarding TCP port 5355
INFO[0021] Forwarding TCP port 5355
INFO[0021] The optional requirement 1 of 1 is satisfied
INFO[0021] READY. Run `lima bash` to open the shell.
$ lima bash
exit status 255
$ limactl shell fedora
bash: line 1: cd: /usr/local/opt/lima: No such file or directory
[bric3@lima-fedora bric3]$ uname -a
Linux lima-fedora 5.11.12-300.fc34.x86_64 #1 SMP Wed Apr 7 16:31:13 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

There's still some issues, that's probably due to my extract path (/usr/local/opt/lima)

@irlevesque
Copy link

I'm also having this issue. The workarounds mentioned haven't helped. I wonder if it's because SIP is enabled on my Mac?

@christian-korneck
Copy link
Contributor

christian-korneck commented May 16, 2021 •
edited
Loading

I wonder if it's because SIP is enabled on my Mac?

@irlevesque - the above worked for me on macOS 11.3.1 (Intel), with SIP enabled (for completeness: I have dtrace allowed csrutil enable --without dtrace, but I don't think this matters here).

I also briefly tried to get it to work on github actions (which uses MacOS 10.15 on Intel) and I ran into "killed: 9" problems. Github Actions currently has an outage, so can't test any further today.

@christian-korneck
Copy link
Contributor

I also briefly tried to get it to work on github actions (which uses MacOS 10.15 on Intel) and I ran into "killed: 9" problems.

so I've now tried it twice on a vanilla MacOS 10.15.7 (Intel) VM with SIP kept enabled (aws ec2 and github actions) and this worked for me:

TL'DR: use com.apple.security.hypervisor (and only), even on MacOS 10.15. I think the README is misleading here (I'll submit a pr shortly).

assuming homebrew is already installed:

brew install qemu coreutils

cat >entitlements.xml <<EOF
> <?xml version="1.0" encoding="UTF-8"?>
> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
> <plist version="1.0">
> <dict>
>     <key>com.apple.security.hypervisor</key>
>     <true/>
> </dict>
> </plist>
> EOF


codesign -s - --entitlements entitlements.xml --force /usr/local/bin/qemu-system-x86_64
curl -L https://github.com/AkihiroSuda/lima/releases/download/v0.1.0/lima-0.1.0-Darwin-x86_64.tar.gz -o lima.tar.gz
sudo tar xf lima.tar.gz -C /usr/local/
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa <<<n 2>&1 >/dev/null #create ssh keypair, needed only on vanilla machine
touch ~/.ssh/known_hosts #create ssh hosts file, needed only on vanilla machine
limactl start

@irlevesque
Copy link

TL'DR: use com.apple.security.hypervisor (and only), even on MacOS 10.15. I think the README is misleading here (I'll submit a pr shortly).

Yep, that worked for me. Thanks for digging in, @christian-korneck

@christian-korneck christian-korneck mentioned this issue May 16, 2021
Merged
@jandubois
Copy link
Member Author

TL'DR: use com.apple.security.hypervisor (and only), even on MacOS 10.15. I think the README is misleading here (I'll submit a pr shortly).

Thank you @christian-korneck, this worked for me as well.

However, how can I tell if qemu is actually using the hvf acceleration, as limactl start default worked for me even when I didn't codesign the qemu binary.

I get exactly the same output on stdout/stderr after codesigning than I did before?

Given that I get qemu-system-x86_64: Error: HV_ERROR on Big Sur if I don't codesign the qemu executable, but it runs just fine on Catalina without it, I now think that -accel=hvf is maybe a no-op on Catalina, and that there might be no point in bothering with signing qemu on that macOS version.

@jandubois
Copy link
Member Author

I now think that -accel=hvf is maybe a no-op on Catalina,

After reading the com.apple.vm.hypervisor description again, I think it is also possible that we simply don't need the old entitlement on Catalina because it was only required for sandboxed processes before:

The entitlement is required to use the Hypervisor APIs in a sandboxed process.

So maybe Big Sur needs the new entitlement for all processes, including non-sandboxed ones, but things work fine on Catalina without it as long as the process is not sandboxed?

At least with a quick search through the qemu repo I couldn't find any commits that would indicate that -accel=hvf would only work on Big Sur, so the theory that the entitlement is not needed on Catalina seems more plausible to me now.

@christian-korneck
Copy link
Contributor

@jandubois - I think so, too. Also browsing the qemu sources I think there would be an error or at least a log message if acceleration didn't work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Docs: Discourage use of com.apple.vm.hypervisor on macOS >=10.15.7 christian-korneck/lima
5 participants
@jandubois @bric3 @irlevesque @christian-korneck @AkihiroSuda