Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run xhyve with network without sudo #60

Closed
rickard-von-essen opened this issue Sep 15, 2015 · 54 comments
Closed

Run xhyve with network without sudo #60

rickard-von-essen opened this issue Sep 15, 2015 · 54 comments

Comments

@rickard-von-essen
Copy link
Contributor

This is the main pain point with xhyve. I'm very interested in getting this fixed or getting a workaround.

Could we detail the problem and possible solutions/work arounds, here.

@rimusz
Copy link

rimusz commented Sep 15, 2015

+1

@xez
Copy link
Contributor

xez commented Sep 15, 2015

Perhaps as a workaround one could use setuid at install time:

sudo chown root build/xhyve
sudo chmod +s build/xhyve

So xhyve will not require an explicit sudo to run with networking. Even better would be to add a patch to xhyve to drop the root privileges as soon as it has instantiated the network connections.

@AntonioMeireles
Copy link
Member

@xez the sudo is a OSX's Hypervisor framework 'defense'. afaict one would just need to run a signed build of xhyve for the sudo stop being required.

@xez
Copy link
Contributor

xez commented Sep 15, 2015

Hypervisor.framework does not require sudo. If it is sandboxed it requires an entitlement. vmnet.framework however requires sudo or an entitlement. From my experiments these don't seem to be public entitlements (App Store only). Bummer :(

@AntonioMeireles
Copy link
Member

@xez fwiw if i'm reading things right one doesn't really need to go thru App store just have a valid apple developer cert to use with codesign (which implies an Apple's Developer Program membership and USD $99/yr). something along what's bellow may work (would be nice, to get sure, if there's someone here which is a Apple registered dev could try snippet bellow (just replacing "VALID APPLE ID" in the Makefile by the real thing...)

❯❯❯ git diff edgy                                                                                                                                                                                                                   
diff --git a/Makefile b/Makefile
index 3294494..306708c 100644
--- a/Makefile
+++ b/Makefile
@@ -77,6 +77,7 @@ DEP := $(OBJ:%.o=%.d)
 INC := -Iinclude

 CFLAGS += -DVERSION=\"$(GIT_VERSION)\"
+LDFLAGS += -mmacosx-version-min=10.10 -sectcreate __TEXT __info_plist xhyve.info.plist

 TARGET = build/xhyve

@@ -95,6 +96,7 @@ build/%.o: src/%.c
        @mkdir -p $(dir $@)
        $(VERBOSE) $(ENV) $(CC) $(CFLAGS) $(INC) $(DEF) -MMD -MT $@ -MF build/$*.d -o $@ -c $<

+
 $(TARGET).sym: $(OBJ)
        @echo ld $(notdir $@)
        $(VERBOSE) $(ENV) $(LD) $(LDFLAGS) -Xlinker $(TARGET).lto.o -o $@ $(OBJ)
@@ -104,6 +106,8 @@ $(TARGET).sym: $(OBJ)
 $(TARGET): $(TARGET).sym
        @echo strip $(notdir $@)
        $(VERBOSE) $(ENV) $(STRIP) $(TARGET).sym -o $@
+       codesign -fs "VALID APPLE ID" --prefix hypervisor. -v --entitlements xhyve.entitlements.plist -r xhyve.rqset  $(TARGET) --deep 
+       codesign -d -r- $(TARGET)

 clean:
        @rm -rf build
diff --git a/xhyve.entitlements.plist b/xhyve.entitlements.plist
new file mode 100644
index 0000000..83e90e0
--- /dev/null
+++ b/xhyve.entitlements.plist
@@ -0,0 +1,15 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
+"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+  <dict>
+    <key>com.apple.security.app-sandbox</key>
+    <true/>
+    <key>com.apple.security.inherit</key>
+    <true/>
+    <key>com.apple.security.network.client</key>
+    <true/>
+    <key>com.apple.security.network.server</key>
+    <true/>
+  </dict>
+</plist>
diff --git a/xhyve.info.plist b/xhyve.info.plist
new file mode 100644
index 0000000..d5fcee9
--- /dev/null
+++ b/xhyve.info.plist
@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
+"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+    <key>CFBundleDevelopmentRegion</key>
+    <string>English</string>
+    <key>CFBundleIdentifier</key>
+    <string>hypervisor.xhyve</string>
+    <key>CFBundleInfoDictionaryVersion</key>
+    <string>6.0</string>
+    <key>CFBundleName</key>
+    <string>xhyve</string>
+    <key>CFBundleVersion</key>
+    <string>0.3</string>
+</dict>
+</plist>
diff --git a/xhyverun.sh b/xhyverun.sh
index fcd7185..4f6dedf 100755
--- a/xhyverun.sh
+++ b/xhyverun.sh
@@ -12,7 +12,7 @@ CMDLINE="earlyprintk=serial console=ttyS0"

 MEM="-m 1G"
 #SMP="-c 2"
-#NET="-s 2:0,virtio-net"
+NET="-s 2:0,virtio-net"
 #IMG_CD="-s 3,ahci-cd,/somepath/somefile.iso"
 #IMG_HDD="-s 4,virtio-blk,/somepath/somefile.img"
 PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
@@ -21,6 +21,7 @@ ACPI="-A"
 #UUID="-U deadbeef-dead-dead-dead-deaddeafbeef"

@rimusz
Copy link

rimusz commented Sep 15, 2015

@AntonioMeireles is right, the same way I sign my https://github.com/rimusz/coreos-xhyve-ui App

@AntonioMeireles
Copy link
Member

@rimusz so you 're volunteering to get us all signed builds 😸 (hint, hint) ?
@mist64, while on this, fwiw, it may start to make sense to think in simething along a xhyve org in github to distribute load, etc (and keep entropy manageable)...

@rimusz
Copy link

rimusz commented Sep 15, 2015

@AntonioMeireles Yes, I'm, as that should take that pain of sudo away I hope :)

@alimoeeny
Copy link
Contributor

It's not all about "the pain of sudo" right? if you require sudo you leave open the possibility of a whole set of problems and exploits and stuff, if I understand things correctly. That's why we love homebrew :)

@avsm
Copy link

avsm commented Sep 22, 2015

@rimusz Do you have a signed binary of xhyve available that I could try out? When I try signing using a developer key and App Sandboxing active, I end up with an illegal instruction exception in the Hypervisor.framework. The binary releases of xhyve in coreos-xhyve-ui don't seem to have any entitlements in them.

@rimusz
Copy link

rimusz commented Sep 26, 2015

@avsm no, I do not have the signed binary yet, as I was too busy on different things for a while.
I will start working on updated coreos-xhyve-ui in the next week and see what I can come with the xhyve without sudo.

@rimusz
Copy link

rimusz commented Oct 23, 2015

right, finally got a time to try to build the signed xhyve binary, still no joy without the sudo :(

@AntonioMeireles
Copy link
Member

would need more details... but at sight something would be missing along the line in the entitlements part as it should just work afaict.

@gpolitis
Copy link

@AntonioMeireles I'm using a self-signed certificate and your patch to codesign xhyve, but xhyve.rqset is missing from your patch. If I codesign without it, xhyve fails to create the VM (with or without virtio-net) with:

vmx_init: processor not supported by Hypervisor.framework
Unable to create VM (-85377018)

I'm not very familiar with the code signing process on OSX but I'm curious to see what's inside the missing xhyve.rqset.

@rimusz
Copy link

rimusz commented Oct 25, 2015

@AntonioMeireles @gpolitis I have got the same problem with that missing xhyve.rqset and without it
xhyve fails to create the VM too

@AntonioMeireles
Copy link
Member

@rimusz @gpolitis et al.

sorry, xhyve.rqsetshouldn't had been there - was a leftover/ oversight of my part (when i was trying to 'fool' OS X to let me sign stuff without (a valid but) non apple issues certificate)...
(from here) what does codesign -d -r $BLOB / codesign -dvvvv -r $BLOB / spctl --assess --type execute $BLOB give on your side after you build them signed ?

@gpolitis
Copy link

$ codesign -dvvv -r- build/xhyve
Executable=/Users/gp/Projects/xhyve/build/xhyve
Identifier=hypervisor.xhyve
Format=Mach-O thin (x86_64)
CodeDirectory v=20100 size=1225 flags=0x0(none) hashes=53+5 location=embedded
Hash type=sha1 size=20
CDHash=b96eea61753977602e62ca41560ec225fb439b4a
Signature size=1278
Authority=gp
Signed Time=Oct 25, 2015, 3:09:33 PM
Info.plist entries=5
TeamIdentifier=not set
Sealed Resources=none
designated => identifier "hypervisor.xhyve" and certificate leaf = H"92ce583d98ed5a0762f95365157ee78614afb4ed"
$ codesign -d -r- build/xhyve
Executable=/Users/gp/Projects/xhyve/build/xhyve
designated => identifier "hypervisor.xhyve" and certificate leaf = H"92ce583d98ed5a0762f95365157ee78614afb4ed"
$ spctl --assess --type execute build/xhyve
build/xhyve: rejected

@gpolitis
Copy link

When I run the sandboxed xhyve executable, sandboxd refuses to open some device needed by the hv_vm_create function:

xhyve(16791) deny iokit-open AppleHVClient

Process:         xhyve [16791]
Path:            /Users/gp/Projects/xhyve/build/xhyve
Load Address:    0x1033f6000
Identifier:      xhyve
Version:         ??? (???)
Code Type:       x86_64 (Native)
Parent Process:  sh [16790]

Date/Time:       2015-10-29 21:29:03.419 -0500
OS Version:      Mac OS X 10.11.1 (15B42)
Report Version:  8

Thread 0:
0   libsystem_kernel.dylib          0x00007fff89e15c96 mach_msg_trap + 10
1   IOKit                           0x00007fff928548c9 io_service_open_extended + 128
2   IOKit                           0x00007fff927f7332 IOServiceOpen + 45
3   Hypervisor                      0x0000000103635e65 hv_vm_create + 149
4   xhyve                           0x000000010341324d main + 1957 (vmx.c:468)
5   libdyld.dylib                   0x00007fff881225ad start + 1

Binary Images:
       0x1033f6000 -        0x10341fff7  xhyve (0) <b2d864dc-b8c8-3d94-b0af-3cc5f8abe391> /Users/gp/Projects/xhyve/build/xhyve
       0x103635000 -        0x103639fff  com.apple.Hypervisor (1.0 - 1) <4deeba77-3b83-3b64-bbb8-25ca44d31e38> /System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor
    0x7fff8811f000 -     0x7fff88122ffb  libdyld.dylib (360.17) <99396189-e53a-3a78-bc49-449236da7cff> /usr/lib/system/libdyld.dylib
    0x7fff89e04000 -     0x7fff89e22fff  libsystem_kernel.dylib (3247.10.11) <fa4dcf42-27b0-3878-9158-e148d208b21e> /usr/lib/system/libsystem_kernel.dylib
    0x7fff927f1000 -     0x7fff92864fff  com.apple.framework.IOKit (2.0.2) <6a9d7e11-156a-3fdb-850e-114ecb1ae7fd> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit

The protected device could be the raw CPU but I'm not sure because I have no way of verifying which device it is.

@gpolitis
Copy link

@xez you mention that

If it is sandboxed it requires an entitlement. vmnet.framework however requires sudo or an entitlement.

Do you know which entitlements are required?

@xez
Copy link
Contributor

xez commented Nov 3, 2015

Mh, just going by /System/Library/Sandbox/Profiles/application.sb:

(if (entitlement "com.apple.vm.hypervisor")
  (allow iokit-open (iokit-user-client-class "AppleHVClient")))
(if (entitlement "com.apple.vm.networking")
  (allow mach-lookup (global-name "com.apple.NetworkSharing")))

so com.apple.vm.hypervisor for Hypervisor.framework and com.apple.vm.networkingfor vmnet.framework, I guess.

@gpolitis
Copy link

gpolitis commented Nov 4, 2015

11/3/15 7:14:09.954 PM taskgated[219]: no application identifier provided, can't use provisioning profiles [pid=7146]
11/3/15 7:14:09.954 PM taskgated[219]: killed hypervisor.xhyve[pid 7146] because its use of the com.apple.vm.hypervisor entitlement is not allowed (error code -67050)

As @xez would have expected, I guess :-)

@jeremyhu jeremyhu mentioned this issue Nov 14, 2015
@jeremyhu
Copy link
Member

The changes that I made in pull request #75 should allow for easier development in this area. I just grabbed the entitlements from Antonio's comment above.

@geoff-nixon
Copy link

@AntonioMeireles 👍 Thanks for this.

@AntonioMeireles
Copy link
Member

@geoff-codes welcome!

@dalehamel
Copy link
Member

Has anyone managed to come up with a working, code-signing+entitlements based solution?

I have an apple developer account and I provisioned an app with a developer ID deployment and certificate for this. I added the certificate to my keyring, and applied the patch provided by @AntonioMeireles above. I ran into the same issue as @gpolitis (xhyve killed because of missing entitlement).

2015-12-05 11:51:15.821 PM taskgated[1143]: killed hypervisor.xhyve[pid 30819] because its use of the com.apple.vm.hypervisor entitlement is not allowed (error code -67050)

I tried patching the entitlements to include networking and hypervisor, as below. I'm still seeing the app get killed, only now it happens immediately when even trying to invoke xhyve (won't even print help), where previously without these entitlements xhyve would at least run.

If anyone has gotten this working, I'd like to know how. It seems like these entitlements may not be supported yet, as the app gets killed when trying to use them. Further, they don't show up in any documentation I could find, and don't have corresponding xcode options.

I am trying to build a rubygem that uses xhyve (https://github.com/dalehamel/xhyve-ruby), and would prefer users not have to run the bundled xhyve binary code as root for security reasons.

Below is my adjusted patch:

iff --git a/Makefile b/Makefile
index 3294494..eedf336 100644
--- a/Makefile
+++ b/Makefile
@@ -77,6 +77,8 @@ DEP := $(OBJ:%.o=%.d)
 INC := -Iinclude
 CFLAGS += -DVERSION=\"$(GIT_VERSION)\"
+LDFLAGS += -mmacosx-version-min=10.10 -sectcreate __TEXT __info_plist xhyve.info.plist
@@ -104,6 +106,8 @@ $(TARGET).sym: $(OBJ)
 $(TARGET): $(TARGET).sym
        @echo strip $(notdir $@)
        $(VERBOSE) $(ENV) $(STRIP) $(TARGET).sym -o $@
+       codesign -fs "VALID APPLE ID" --prefix hypervisor. -v --entitlements xhyve.entitlements.plist -r xhyve.rqset  $(TARGET) --deep 
+       codesign -d -r- $(TARGET)
diff --git a/xhyve.entitlements.plist b/xhyve.entitlements.plist
new file mode 100644
index 0000000..e0e11cf
--- /dev/null
+++ b/xhyve.entitlements.plist
@@ -0,0 +1,19 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
+"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+  <dict>
+    <key>com.apple.security.app-sandbox</key>
+    <true/>
+    <key>com.apple.security.inherit</key>
+    <true/>
+    <key>com.apple.security.network.client</key>
+    <true/>
+    <key>com.apple.security.network.server</key>
+    <true/>
+    <key>com.apple.vm.hypervisor</key>
+    <true/>
+    <key>com.apple.vm.networking</key>
+    <true/>
+  </dict>
+</plist>
diff --git a/xhyve.info.plist b/xhyve.info.plist
new file mode 100644
index 0000000..d5fcee9
--- /dev/null
+++ b/xhyve.info.plist
@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
+"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+    <key>CFBundleDevelopmentRegion</key>
+    <string>English</string>
+    <key>CFBundleIdentifier</key>
+    <string>hypervisor.xhyve</string>
+    <key>CFBundleInfoDictionaryVersion</key>
+    <string>6.0</string>
+    <key>CFBundleName</key>
+    <string>xhyve</string>
+    <key>CFBundleVersion</key>
+    <string>0.3</string>
+</dict>
+</plist>

From my reading of this issue and #75 it seems as though no one actually has this working yet?

@rimusz it looks like you never got this working for your coreos-ui app?

But, if anyone has an entitlement /code signing/ sandbox solution working, please help me know what I'm doing wrong :)

@geoff-nixon
Copy link

@dalehamel No, I never did get it working; I ran into the same issue you did.

Maybe one of us should open a radar bug (or a TSI) and see if we can get a response from Apple?

@jeremyhu
Copy link
Member

jeremyhu commented Dec 7, 2015

@geoff-codes taskgate isn't complaining here when building it from my Xcode project:

$ git clone git@github.com:jeremyhu/xhyve.git
$ cd xhyve
$ xcodebuild
$ build/Release/xhyve --help

Do you have issues when building like that? If not, I suggest you figure out what is different about the Makefile build failing.

@dalehamel
Copy link
Member

Are you able to actually start a Guest with networking without sudo using
the Xcode build? @jeremyhu

On Monday, 7 December 2015, Jeremy Huddleston Sequoia <
notifications@github.com> wrote:

@geoff-codes https://github.com/geoff-codes taskgate isn't complaining
here when building it from my Xcode project:

$ git clone git@github.com:jeremyhu/xhyve.git
$ cd xhyve
$ xcodebuild
$ build/Release/xhyve --help

Do you have issues when building like that? If not, I suggest you figure
out what is different about the Makefile build failing.


Reply to this email directly or view it on GitHub
#60 (comment).

@dalehamel
Copy link
Member

@geoff-codes i could file one, i found out my company has access to two.

Perhaps we can all put our heads together to come up with a succinct description of the problem?

It would be great to submit the xcode stuff, as it is likely going to be better supported by apple. Once that's working we can always rev eng it.

@geoff-nixon
Copy link

@jeremyhu

Do you have issues when building like that?

Yep.

~> git clone git://github.com/jeremyhu/xhyve
~> cd xhyve
~> xcodebuild CODE_SIGN_IDENTITY="Developer ID Application" # Actually, ad-hoc signing seems to do fine with these four entitlements.
~> build/Release/xhyve --help # Yes, this is fine...
Usage: xhyve [-behuwxACHPWY] [-c vcpus] [-g <gdb port>] [-l <lpc>]
...
~> ln build/Release/xhyve build/xhyve
~> ./xhyverun.sh # ... but this isn't. :(
vmx_init: processor not supported by Hypervisor.framework
Unable to create VM (-85377018)

In system.log we get hyve(12345) deny iokit-open AppleHVClient (as @gpolitis did, above), so I edit src/xhyve-entitlements.plist to add com.apple.vm.hypervisor and com.apple.vm.networking, and ...

~> xcodebuild clean
~> xcodebuild CODE_SIGN_IDENTITY="Developer ID Application"
~> ./build/Release/xhyve --help
'./build/Release/xhyve --help' terminated by signal SIGKILL (Forced quit)

... and now its taskgate complaining:
taskgated[236]: no application identifier provided, can't use provisioning profiles [pid=44315]
taskgated[236]: killed xyz.xhyve.xhyve[pid 44315] because its use of the com.apple.vm.hypervisor entitlement is not allowed (error code -67050)

and we get this crash report:

Process:               xhyve [44315]
Path:                  /Users/USER/*/xhyve
Identifier:            xhyve
Version:               ???
Code Type:             X86-64 (Native)
Parent Process:        fish [7159]
Responsible:           xhyve [44315]
User ID:               501

Date/Time:             2015-12-07 07:46:27.519 -0800
OS Version:            Mac OS X 10.11.1 (15B42)
Report Version:        11
Anonymous UUID:        7738B1E3-9213-F179-1B36-9C45C052E7DD


Time Awake Since Boot: 10000 seconds

System Integrity Protection: enabled

Crashed Thread:        0

Exception Type:        EXC_CRASH (Code Signature Invalid)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

@dalehamel I would say maybe lets hold off on that for now, as long as we have our man @jeremyhu on the case. He might actually be able to see something on the rdar://, unlike the rest of us. 😉

@jeremyhu
Copy link
Member

jeremyhu commented Dec 7, 2015

Ah, /facepalm. I copied an earlier entitlements.plist into my xcode patch and didn't look back. I just assumed that it was the same and based my earlier reply on that false assumption (and the comment that the error occurred by taskgated denying the entitlement). Sorry for the confusion. I'm looking into it.

@jeremyhu
Copy link
Member

jeremyhu commented Dec 7, 2015

I asked around, and it turns out that those entitlements are "App Store”-only entitlements and can not be self-signed or signed with a Developer ID. As such, I think the current options are unfortunately:

  1. do not use the com.apple.vm.hypervisor entitlement, and run under sudo (or setuid root). =(
  2. use the entitlements, but disable code signature verification on your system. =(

Both have obvious downsides and are sub-optimal. I'll continue looking into this problem space, but I likely won't have updates on this until (and if) changes in the base OS allow it.

@AntonioMeireles
Copy link
Member

@jeremyhu so, if i got you right, you're saying that ALL OSX's sandboxing features regarding whitelisting of priv escalation ONLY work when apps are signed with an (Apple issued thru a valid ADC membership) certificate (which may look brutal but actually makes tons of sense, as it gives AAPL a quick way to revoke/blacklist it if need arises) but ALSO and ONLY got distributed via the App Store ? is this specific to Hypervisor.framework or, in general, codesign only gets really usefull is app goes via the App Store ? (also, sorry in advance if this is in some obvious place i've missed - where is (url) this stated on some docs ?)
/cc @rimusz

@dalehamel
Copy link
Member

Looking. At Xcode, it only lets you do sandboxing on App Store apps. So it
looks like it's a matter of any sandboxing/entitlements.

Of course, a workaround would be for someone to make a free xhyve app. I'll
give this a try.

On Monday, 7 December 2015, António Meireles notifications@github.com
wrote:

@jeremyhu https://github.com/jeremyhu so, if i got you right, you're
saying that ALL OSX's sandboxing features regarding whitelisting of priv
escalation ONLY work when apps are signed with an (Apple issued thru a
valid ADC membership) certificate (which may look brutal but actually makes
tons of sense, as it gives AAPL a quick way to revoke/blacklist it if need
arises) but ALSO and ONLY got distributed via the App Store ? is this
specific to Hypervisor.framework or, in general, codesign only gets
really usefull is app goes via the App Store ? (also, sorry in advance
if this is in some obvious place i've missed - where is (url) this stated
on some docs ?)
/cc @rimusz https://github.com/rimusz


Reply to this email directly or view it on GitHub
#60 (comment).

@geoff-nixon
Copy link

I wonder if an App Store App would get approved. Are pure-CLI tools allowed? Looking at the Mac App Store Review Guidelines, it looks like this might be trouble:

Apps that do not use system provided items, such as buttons and icons, correctly and as described in the OS X Human Interface Guidelines will be rejected.

So maybe we need to put the xhyve binary in Resources and make the application's executable be something like:

#!/usr/bin/osascript
display dialog "The xhyve binary is in the application bundle." buttons {"Awesome"} with icon path to resource "Terminal.icns" in bundle {POSIX path of "/Applications/Utilities/Terminal.app"}

awesome

😆 🙊

@geoff-nixon
Copy link

@AntonioMeireles Yes, after doing some move digging/experimenting it appears that is correct.

I've tried:

  • Rebooting into Recovery and disabling System Integrity Protection (csrutil disable). No dice.
  • Creating a custom provisioning profile (I have a paid developer program account), using my machine's UDID, installing the profile on my machine, and code-signing with this profile and the corresponding "Mac Developer" certificate – which is how you're supposed to be able to beta-test App Store apps. It still rejects the com.apple.vm.* entitlements.
    • So either I'm doing something wrong (quite possible - I've never actually done that before, nor submitted anything to the App Store),
    • or you can't even test a program using these entitlements before submitting (?).

Apparently, this business of App Store-only entitlements has actually been around for a while. The most illuminating piece of "documentation" regarding this business is @landonf's task-unchain. Alas, I even tried applying this patch to my 10.11 install; no dice.

I still think this is pretty bizarre. @jeremyhu, you mentioned,

  1. use the entitlements, but disable code signature verification on your system. =(

Have you actually been able to do this? What's the secret?

@jeremyhu
Copy link
Member

jeremyhu commented Dec 8, 2015

use the entitlements, but disable code signature verification on your system. =(

Have you actually been able to do this? What's the secret?

My boot-args are "-v debug=0x146 kext-dev-mode=1 amfi=0xff cs_enforcement_disable=1"

-v : verbose boot
debug=0x146 : DB_LOG_PI_SCRN | DB_ARP | DB_NMI | DB_PRT, see Kernel documentation (currently Table 20-1)
kext-dev-mode=1 : Disable codesigning verification of kexts
amfi=0xff : Disable AMFI
cs_enforcement_disable=1 : Disable codesignature enforcement

Plus, I set the following preferences:

sudo defaults write /Library/Preferences/com.apple.security.coderequirements Entitlements -string always
sudo defaults write /Library/Preferences/com.apple.security.coderequirements AllowUnsafeDynamicLinking -bool YES

The first overrides requirements for restricted entitlements. The second causes the system to not mark the process as having codesignature restrictions (specifically, csops(pid, CS_OPS_MARKRESTRICT, NULL, 0) is not called on the process).

What you require is a subset of all of that.

@jeremyhu
Copy link
Member

jeremyhu commented Dec 8, 2015

Out of curiosity, why do you have com.apple.security.inherit in your entitlements.plist? I don't see any calls to fork(2), posix_spawn(2), system(3), or popen(3) in xhyve, so there should be no child processes. Am I missing something?

@AntonioMeireles
Copy link
Member

@jeremyhu no, you're not missing anything. that is/was specific to my use case. there (https://github.com/TheNewNormal/corectl) i hacked things around a cli so that one could call xhyve in detached or non-detached mode, or even lunch multiple VMs in a single pass, and for that com.apple.security.inherit would end handy.

@darkn3rd
Copy link

darkn3rd commented May 3, 2016

👍

@dalehamel
Copy link
Member

I think this issue should be closed as it doesn't not appear to be possible
to do this without setuid

On Tuesday, 3 May 2016, Joaquin Menchaca notifications@github.com wrote:

👍


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#60 (comment)

@rickard-von-essen
Copy link
Contributor Author

Closing since it seems impossible.

@xpepermint
Copy link

Is this still the case?

@jeremyhu
Copy link
Member

Is this still the case?

No. If you build using the Xcode project, it should setup the entitlements correctly for you.

@dexwolfram
Copy link

Hi, I've been spending quite some time trying to build xhyve using xcodebuild according to the instructions. Trying to run the resulting xcodebuild app, results in:

$ xhyve -h
Killed: 9

I've tried various methods from within Xcode as well.

Instructions for xhyve say that:

macOS limits access to the networking API (vmnet) to builds that are code signed and have the appropriate entitlement. The code signing/entitlement requirement can be bypassed by running xhyve as root (via sudo)... Building xhyve via xcodebuild signs the build. Building xhyve via make does not.

Is my reading of this thread correct, that build with xcodebuild as per the instructions is not longer possible? It's possible that make is working, but I can't get xcodebuild to produce a functional exectuable, and it appears from this thread that there may be no point in this, since root may well be required regardless of whether or not the executable is signed. Is that correct? (I'm not a programmer.)

FYI, it appears others have run into this problem as well (google translate helps):

https://cloud-atlas.readthedocs.io/zh_CN/latest/kvm/macos/xhyve.html

编译后执行程序位于 build/Release/xhyve 。 在最新的 macOS Mojave 10.14.1 编译成功,运行 xhyve -h 失败,显示
Killed: 9
不过,实际上发现,使用完整的路径运行 buid/xhyve 则可以正常工作:
/Users/huatai/github/xhyve/build/xhyve -h

https://houraku365.hatenablog.com/entry/2019/01/06/ubuntu18-on-xhyve-on-macosx-failed-again

しかし下記のように起動できなかったので、build 結果を削除しました。
$ ./build/Release/xhyve -h
Killed: 9
$ sudo ./build/Release/xhyve -h
Killed: 9
$ xcodebuild clean

@dexwolfram
Copy link

Hm. Looks like from the last couple of posts that build & run without su using xcodebuild is now supported. I am using a self-signed certificate. Do I need to pay $99 for a certificate-authority issued cert to get this to work?

@dexwolfram
Copy link

FYI, originally on xcodebuild I would get an app xhyve that terminates with "killed -9" (with or without sudo).

I just tried following these instructions:

https://kubadownload.com/news/codesign-sign-app

codesign --force --deep --sign - xhyve

... which re-codesigns xhyve. Now the app runs again with sudo, but still no networking.

I also tried turning on sandboxing with networking from within Xcode, then re- code signing with "codesign --force --deep --sign - xhyve", with the same outcome. I must be doing something wrong if they are saying this should now work. I don't have a developer's account, but Apple support tells me one should not be necessary to run on my own machine. They say to follow these instructions:

https://help.apple.com/xcode/mac/current/#/dev5a825a1ca

which... has a link to https://help.apple.com/xcode/mac/current/#/dev23aab79b4 where it says "If you are not a member of the Apple Developer Program, Xcode will create a personal team for you." But still this produces xhyve that terminates with killed -9, and that on re-sign "codesign --force --deep --sign - xhyve" only runs with sudo.

@jeremyhu
Copy link
Member

jeremyhu commented May 16, 2020

FYI, originally on xcodebuild I would get an app xhyve that terminates with "killed -9" (with or without sudo).

I just tried following these instructions:

https://kubadownload.com/news/codesign-sign-app

codesign --force --deep --sign - xhyve

... which re-codesigns xhyve. Now the app runs again with sudo, but still no networking.

because your re-signing nuked the entitlements.

The build coming out of Xcode should be signed. What's the output of codesign -vvv /path/to/xhyve?

I also tried turning on sandboxing with networking from within Xcode

I don't know what you mean by this.

, then re- code signing with "codesign --force --deep --sign - xhyve", with the same outcome. I must be doing something wrong if they are saying this should now work. I don't have a developer's account, but Apple support tells me one should not be necessary to run on my own machine. They say to follow these instructions:

https://help.apple.com/xcode/mac/current/#/dev5a825a1ca

which... has a link to https://help.apple.com/xcode/mac/current/#/dev23aab79b4 where it says "If you are not a member of the Apple Developer Program, Xcode will create a personal team for you." But still this produces xhyve that terminates with killed -9, and that on re-sign "codesign --force --deep --sign - xhyve" only runs with sudo.

You should not need to re-sign. The binary is signed by Xcode. What does the crash log say? I suspect the SIGKILL is due to you signing adhoc. Please select a valid developer profile (pretty sure your free one is fine) to sign with, and you should be ok.

@dexwolfram
Copy link

dexwolfram commented May 22, 2020

Was finally able to get a developer account with Apple, but still no luck, now having made two code sign certificates, "Apple Development" and "Developer ID Application." I start with:

$ xcode-select --print-path
/Applications/Xcode.app/Contents/Developer
$ xcode-select --version
xcode-select version 2373.

Catalina, ran 'sudo spctl --master-disable' to allow possibility of "Allow apps downloaded from anywhere" in MacOS security control panel.

Running xcodebuild from within xhyve directory appears to go well until the code signing part at the end, where there is an error:

$ xcodebuild

CodeSign /Users/dad/xhyve/build/Release/xhyve (in target 'xhyve' from project 'xhyve')
cd /Users/dad/xhyve
export CODESIGN_ALLOCATE=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate

Signing Identity: "-"

/usr/bin/codesign --force --sign - -o library --entitlements /Users/dad/xhyve/build/xhyve.build/Release/xhyve.build/xhyve.xcent --timestamp=none /Users/dad/xhyve/build/Release/xhyve

RegisterExecutionPolicyException /Users/dad/xhyve/build/Release/xhyve (in target 'xhyve' from project 'xhyve')
cd /Users/dad/xhyve
builtin-RegisterExecutionPolicyException /Users/dad/xhyve/build/Release/xhyve
note: Execution policy exception registration failed and was skipped: Error Domain=NSPOSIXErrorDomain Code=1 "Operation not permitted" (in target 'xhyve' from project 'xhyve')

** BUILD SUCCEEDED **

And again:

$ ~/xhyve/build/Release/xhyve -h

Killed: 9'

Based on a nebulous tip here: https://forums.developer.apple.com/thread/124529 , I tried alternatively:

$ xcodebuild -UseModernBuildSystem=NO

Interestingly, the RegisterExecutionPolicyException error at the end disappears, but the resulting binary is still killed.

ProcessProductPackaging "" build/xhyve.build/Release/xhyve.build/xhyve.xcent
cd /Users/dad/xhyve

Entitlements:

{
"com.apple.security.get-task-allow" = 1;
"com.apple.security.network.client" = 1;
"com.apple.security.network.server" = 1;
"com.apple.vm.hypervisor" = 1;
"com.apple.vm.networking" = 1;
}

builtin-productPackagingUtility -entitlements -format xml -o /Users/dad/xhyve/build/xhyve.build/Release/xhyve.build/xhyve.xcent

CodeSign build/Release/xhyve
cd /Users/dad/xhyve
export CODESIGN_ALLOCATE=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate

Signing Identity: "-"

/usr/bin/codesign --force --sign - -o library --entitlements /Users/dad/xhyve/build/xhyve.build/Release/xhyve.build/xhyve.xcent --timestamp=none /Users/dad/xhyve/build/Release/xhyve

** BUILD SUCCEEDED **

$ ~/xhyve/build/Release/xhyve -h

Killed: 9

With:

$ codesign -vvv ~/xhyve/build/Release/xhyve

/Users//xhyve/build/Release/xhyve: valid on disk
/Users/
/xhyve/build/Release/xhyve: satisfies its Designated Requirement

I noticed that xcodebuild is not automatically selecting one of my signing certificates. Tried:

$ xcodebuild -SigningIdentity="Developer ID Application" -UseModernBuildSystem=NO

Still, the signing certificate is not chosen at the end

Signing Identity: "-"

So I went ahead and built again and followed up with codesign command to re-codesign using the same parameters as were invoked in the build, but still no luck:

$ xcodebuild -UseModernBuildSystem=NO
$ codesign --verbose --force --sign "Developer ID Application" -o library --entitlements /Users/*/xhyve/build/xhyve.build/Release/xhyve.build/xhyve.xcent --timestamp=none /Users/*/xhyve/build/Release/xhyve
$ ~/xhyve/build/Release/xhyve -h
Killed: 9

Same with this:

$ xcodebuild -UseModernBuildSystem=NO
$ codesign --verbose --force --sign "Apple Development" -o library --entitlements /Users/*/xhyve/build/xhyve.build/Release/xhyve.build/xhyve.xcent --timestamp=none /Users/*/xhyve/build/Release/xhyve
$ ~/xhyve/build/Release/xhyve -h
Killed: 9

Crash log for xhyve always shows the same thing:

System Integrity Protection: enabled

Crashed Thread: Unknown

Exception Type: EXC_CRASH (Code Signature Invalid)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY

Termination Reason: Namespace CODESIGNING, Code 0x1

A bit of a loss here. I feel like invoking 'xcodebuild -UseModernBuildSystem=NO' should have picked out a better signing certificate from the get-go, and not gone with 'Signing Identity: "-"'. But I'm not able to force it.

Must be doing something wrong, just not sure what. Change the bundle identifier? Add the mac to a provisioning profile and again try to codesign with the "Apple Development" certificate as a development version? Seems like the executable might expire before long in that case?

@dexwolfram
Copy link

dexwolfram commented May 22, 2020

On the bright side, I was able to get Ubuntu Server 20 running on xhyve very nicely (with sudo, nothing to do with this thread). I used the legacy BIOS server install image (as opposed to regular UEFI), found here:

http://cdimage.ubuntu.com/ubuntu-legacy-server/releases/20.04/release/

... not absolutely sure if that was critical, but at the very least it made things a lot easier. Also, I removed "acpi=off" from CMDLINE in the install script.

Absolutely critical was to pay close attention to the disk partitioning during install, and to apply a manual disk partition. Perhaps because of disk lettering (vba1, vba2...), the installer does not apply appropriate formatting to the partitions. I manually made a single ext4 partition for ubuntu, and manually marked it as bootable.

People are having trouble with Ubuntu 18 (#161) and I suspect are running into the same troubles with 20.

@jeremyhu
Copy link
Member

Was finally able to get a developer account with Apple, but still no luck, now having made two code sign certificates, "Apple Development" and "Developer ID Application." I start with:

...

CodeSign /Users/dad/xhyve/build/Release/xhyve (in target 'xhyve' from project 'xhyve')
cd /Users/dad/xhyve
export CODESIGN_ALLOCATE=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate
Signing Identity: "-"

"-" means that you're still AdHoc signing. You're not using the signing certificate you got. You need to select it using the CODE_SIGN_IDENTITY build setting when you build.

If you don't know how to do that, just open the project in Xcode, select the project to open the project editor. Select the target and then open the "Signing & Capabilities" tab.

Based on a nebulous tip here: https://forums.developer.apple.com/thread/124529 , I tried alternatively:

Don't believe everything you read. That has nothing to do with it. You just need to select the right signing identity.

I noticed that xcodebuild is not automatically selecting one of my signing certificates.

Correct. You need to specify it.

So I went ahead and built again and followed up with codesign command to re-codesign using the same parameters as were invoked in the build, but still no luck:

$ xcodebuild -UseModernBuildSystem=NO
$ codesign --verbose --force --sign "Developer ID Application" -o library --entitlements /Users/*/xhyve/build/xhyve.build/Release/xhyve.build/xhyve.xcent --timestamp=none /Users/*/xhyve/build/Release/xhyve

You could just resign it, preserving the existing entitlements in the ad-hoc signature. Check
man codesign for info on --preserve-metadata=...

$ ~/xhyve/build/Release/xhyve -h
Killed: 9

System Integrity Protection: enabled
Crashed Thread: Unknown
Exception Type: EXC_CRASH (Code Signature Invalid)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: Namespace CODESIGNING, Code 0x1

Please bare in mind that the code signature is cached on an inode. If you try copying the (valid) contents over an existing file, the system will terminate the process. You will need to unlink the target (eg: with mv instead of cp). That is one of the most common mistakes folks run into with invalid code signatures that appear valid on disk.

A bit of a loss here. I feel like invoking 'xcodebuild -UseModernBuildSystem=NO' should have picked out a better signing certificate from the get-go

What makes you think that? I suggest using XCBuild (the modern build system).

@dexwolfram
Copy link

Jeremy, thanks for all your help here. Not sure I know enough about what I'm doing here to get this to work, and I'll just wait it out until someone else who runs into my problem figures out a solution. No biggie.

"-" means that you're still AdHoc signing.

FYI, I was finally able to get xcodebuild to attach my valid signature, by changing out "-" for my certificate ID (e.g. CODE_SIGN_IDENTITY = Apple Development: Dex Wolfram (GHJRE3745T) )on line 6 of common.xcconfig: https://github.com/machyve/xhyve/blob/master/xcconfigs/common.xcconfig . Otherwise, I couldn't get xcodebuild to override the "-" using CODE_SIGN_IDENTITY on xcodebuild command line, so switching it out in the xcconfig file worked. It signs with my ID, but I still get:

$ ~/xhyve/build/Release/xhyve -h
Killed: 9

Console gives me:

/Users/~/Downloads/xhyve/build/Release/xhyve signature not valid: -67050
Security policy would not allow process: 5621, /Users/dad/Downloads/xhyve/build/Release/xhyve

And the crash log is pretty much the same. Signature doesn't look obviously bad:

$ codesign -dv --verbose=4 /Users//Downloads/xhyve/build/Release/xhyve
Executable=/Users/
/Downloads/xhyve/build/Release/xhyve
Identifier=xyz.xhyve.xhyve
Format=Mach-O thin (x86_64)
CodeDirectory v=20200 size=2639 flags=0x2000(library-validation) hashes=75+5 location=embedded
VersionPlatform=1
VersionMin=657920
VersionSDK=659204
Hash type=sha256 size=32
CandidateCDHash sha1=c98f8305df01b2af2f1c18081f346c40b55c8699
CandidateCDHashFull sha1=c98f8305df01b2af2f1c18081f346c40b55c8699
CandidateCDHash sha256=5ca9d593850868b19426d4d8df6c705143c8156e
CandidateCDHashFull sha256=5ca9d593850868b19426d4d8df6c705143c8156ec8af7f01caf4521efe70bd26
Hash choices=sha1,sha256
CMSDigest=5cf9d478c8c4d8c90e92eca01abaf3f5fa72ae78f1c6e90e3af7fd7714a269f8
CMSDigestType=2
Page size=4096
CDHash=5ca9d593850868b19426d4d8df6c705143c8156e
Signature size=4820
Authority=Apple Distribution: Dex Wolfram (RJGRE3GHJF)
Authority=Apple Worldwide Developer Relations Certification Authority
Authority=Apple Root CA
Signed Time=May 30, 2020 at 1:38:58 AM
Info.plist entries=16
TeamIdentifier=RJGRE3GHJF
Sealed Resources=none
Internal requirements count=1 size=180

I also tried to disable gatekeeper ("Allow apps downloaded from anywhere"):

sudo spctl --master-disable

... and this little trick: https://eclecticlight.co/2019/10/22/catalina-crashes-non-notarized-command-tools-with-a-quarantine-flag/ (though I didn't use his app, just copied the quarantine flag attribute from another executable to xhyve), still no luck.

Again, thanks for you help. I'll just let this sit for a while. Who knows, maybe I'll come across the problem later. Could be something with the certificates. I'll post back if I do.

@mrpippy
Copy link

mrpippy commented Jun 24, 2020

$ ~/xhyve/build/Release/xhyve -h
Killed: 9

I'm also having this problem with a fresh xhyve checkout. I've tried picking my own (paid) team and using the Mac Developer certificate, but no go. But if I remove com.apple.vm.networking and com.apple.vm.hypervisor from xhyve-entitlements.plist, it is able to launch.

This is Xcode 11.3.1 on 10.14.6.

xhyve pip$ build/Release/xhyve
Killed: 9
xhyve pip$ codesign -dvvv build/Release/xhyve
Executable=/Users/pip/no_backup/xhyve/build/Release/xhyve
Identifier=xyz.xhyve.xhyve
Format=Mach-O thin (x86_64)
CodeDirectory v=20200 size=2639 flags=0x2000(library-validation) hashes=75+5 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha1=1d168d9c58a6f09c5b01cf53ab3f2655d5ccb34d
CandidateCDHash sha256=2f5e246741f8ba8eb044068f1a1064649cb59472
Hash choices=sha1,sha256
CDHash=2f5e246741f8ba8eb044068f1a1064649cb59472
Signature size=4821
Authority=Apple Development: Brendan Shanks (WEGLPW842F)
Authority=Apple Worldwide Developer Relations Certification Authority
Authority=Apple Root CA
Signed Time=Jun 23, 2020 at 11:17:13 PM
Info.plist entries=16
TeamIdentifier=39U85C54H6
Sealed Resources=none
Internal requirements count=1 size=180

From Console.app:

default	23:21:47.602894 -0700	amfid	Requirements for restricted entitlements failed to validate, error -67050, requirements: '<private>', error: (null)
default	23:21:47.602926 -0700	amfid	Restricted entitlements not validated, bailing out. Error: (null)
default	23:21:47.603168 -0700	amfid	/Users/pip/xhyve/build/Release/xhyve signature not valid: -67050
default	23:21:47.603208 -0700	kernel	AMFI: code signature validation failed.
default	23:21:47.603218 -0700	kernel	AMFI: bailing out because of restricted entitlements.
default	23:21:47.603242 -0700	kernel	mac_vnode_check_signature: /Users/pip/xhyve/build/Release/xhyve: code signature validation failed fatally: When validating /Users/pip/xhyve/build/Release/xhyve:
  Code has restricted entitlements, but the validation of its code signature failed.
Unsatisfied Entitlements:
default	23:21:47.603287 -0700	kernel	proc 225: load code signature error 4 for file "xhyve"

@micahflee
Copy link

I'm trying to figure out how to do a similar thing, and for what it's worth I found this: https://developer.apple.com/forums/thread/116870?answerId=361392022#361392022

However,

com.apple.vm.networking

is a special entitlement, one that you can’t enable via the developer web site. Such entitlements must be explicitly granted to you by Apple. You can do that by dropping DTS a line [2]. However, I have to set some expectations here.

com.apple.vm.networking

was intended to allow VM apps to ship in the Mac App Store (where privilege escalation is outlawed) and thus it’s going to be a hard sell for non-app products shipping outside the store.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

16 participants