-
Notifications
You must be signed in to change notification settings - Fork 356
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run xhyve with network without sudo #60
Comments
+1 |
Perhaps as a workaround one could use setuid at install time:
So xhyve will not require an explicit |
@xez the |
|
@xez fwiw if i'm reading things right one doesn't really need to go thru App store just have a valid apple developer cert to use with ❯❯❯ git diff edgy
diff --git a/Makefile b/Makefile
index 3294494..306708c 100644
--- a/Makefile
+++ b/Makefile
@@ -77,6 +77,7 @@ DEP := $(OBJ:%.o=%.d)
INC := -Iinclude
CFLAGS += -DVERSION=\"$(GIT_VERSION)\"
+LDFLAGS += -mmacosx-version-min=10.10 -sectcreate __TEXT __info_plist xhyve.info.plist
TARGET = build/xhyve
@@ -95,6 +96,7 @@ build/%.o: src/%.c
@mkdir -p $(dir $@)
$(VERBOSE) $(ENV) $(CC) $(CFLAGS) $(INC) $(DEF) -MMD -MT $@ -MF build/$*.d -o $@ -c $<
+
$(TARGET).sym: $(OBJ)
@echo ld $(notdir $@)
$(VERBOSE) $(ENV) $(LD) $(LDFLAGS) -Xlinker $(TARGET).lto.o -o $@ $(OBJ)
@@ -104,6 +106,8 @@ $(TARGET).sym: $(OBJ)
$(TARGET): $(TARGET).sym
@echo strip $(notdir $@)
$(VERBOSE) $(ENV) $(STRIP) $(TARGET).sym -o $@
+ codesign -fs "VALID APPLE ID" --prefix hypervisor. -v --entitlements xhyve.entitlements.plist -r xhyve.rqset $(TARGET) --deep
+ codesign -d -r- $(TARGET)
clean:
@rm -rf build
diff --git a/xhyve.entitlements.plist b/xhyve.entitlements.plist
new file mode 100644
index 0000000..83e90e0
--- /dev/null
+++ b/xhyve.entitlements.plist
@@ -0,0 +1,15 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
+"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+ <dict>
+ <key>com.apple.security.app-sandbox</key>
+ <true/>
+ <key>com.apple.security.inherit</key>
+ <true/>
+ <key>com.apple.security.network.client</key>
+ <true/>
+ <key>com.apple.security.network.server</key>
+ <true/>
+ </dict>
+</plist>
diff --git a/xhyve.info.plist b/xhyve.info.plist
new file mode 100644
index 0000000..d5fcee9
--- /dev/null
+++ b/xhyve.info.plist
@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
+"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+ <key>CFBundleDevelopmentRegion</key>
+ <string>English</string>
+ <key>CFBundleIdentifier</key>
+ <string>hypervisor.xhyve</string>
+ <key>CFBundleInfoDictionaryVersion</key>
+ <string>6.0</string>
+ <key>CFBundleName</key>
+ <string>xhyve</string>
+ <key>CFBundleVersion</key>
+ <string>0.3</string>
+</dict>
+</plist>
diff --git a/xhyverun.sh b/xhyverun.sh
index fcd7185..4f6dedf 100755
--- a/xhyverun.sh
+++ b/xhyverun.sh
@@ -12,7 +12,7 @@ CMDLINE="earlyprintk=serial console=ttyS0"
MEM="-m 1G"
#SMP="-c 2"
-#NET="-s 2:0,virtio-net"
+NET="-s 2:0,virtio-net"
#IMG_CD="-s 3,ahci-cd,/somepath/somefile.iso"
#IMG_HDD="-s 4,virtio-blk,/somepath/somefile.img"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
@@ -21,6 +21,7 @@ ACPI="-A"
#UUID="-U deadbeef-dead-dead-dead-deaddeafbeef"
|
@AntonioMeireles is right, the same way I sign my https://github.com/rimusz/coreos-xhyve-ui App |
@AntonioMeireles Yes, I'm, as that should take that pain of sudo away I hope :) |
It's not all about "the pain of sudo" right? if you require |
@rimusz Do you have a signed binary of |
@avsm no, I do not have the signed binary yet, as I was too busy on different things for a while. |
right, finally got a time to try to build the signed xhyve binary, still no joy without the sudo :( |
would need more details... but at sight something would be missing along the line in the entitlements part as it should just work afaict. |
@AntonioMeireles I'm using a self-signed certificate and your patch to codesign xhyve, but
I'm not very familiar with the code signing process on OSX but I'm curious to see what's inside the missing |
@AntonioMeireles @gpolitis I have got the same problem with that missing |
sorry, |
|
When I run the sandboxed xhyve executable, sandboxd refuses to open some device needed by the
The protected device could be the raw CPU but I'm not sure because I have no way of verifying which device it is. |
@xez you mention that
Do you know which entitlements are required? |
Mh, just going by
so |
As @xez would have expected, I guess :-) |
The changes that I made in pull request #75 should allow for easier development in this area. I just grabbed the entitlements from Antonio's comment above. |
@AntonioMeireles 👍 Thanks for this. |
@geoff-codes welcome! |
Has anyone managed to come up with a working, code-signing+entitlements based solution? I have an apple developer account and I provisioned an app with a developer ID deployment and certificate for this. I added the certificate to my keyring, and applied the patch provided by @AntonioMeireles above. I ran into the same issue as @gpolitis (xhyve killed because of missing entitlement).
I tried patching the entitlements to include networking and hypervisor, as below. I'm still seeing the app get killed, only now it happens immediately when even trying to invoke xhyve (won't even print help), where previously without these entitlements xhyve would at least run. If anyone has gotten this working, I'd like to know how. It seems like these entitlements may not be supported yet, as the app gets killed when trying to use them. Further, they don't show up in any documentation I could find, and don't have corresponding xcode options. I am trying to build a rubygem that uses xhyve (https://github.com/dalehamel/xhyve-ruby), and would prefer users not have to run the bundled xhyve binary code as root for security reasons. Below is my adjusted patch: iff --git a/Makefile b/Makefile
index 3294494..eedf336 100644
--- a/Makefile
+++ b/Makefile
@@ -77,6 +77,8 @@ DEP := $(OBJ:%.o=%.d)
INC := -Iinclude
CFLAGS += -DVERSION=\"$(GIT_VERSION)\"
+LDFLAGS += -mmacosx-version-min=10.10 -sectcreate __TEXT __info_plist xhyve.info.plist
@@ -104,6 +106,8 @@ $(TARGET).sym: $(OBJ)
$(TARGET): $(TARGET).sym
@echo strip $(notdir $@)
$(VERBOSE) $(ENV) $(STRIP) $(TARGET).sym -o $@
+ codesign -fs "VALID APPLE ID" --prefix hypervisor. -v --entitlements xhyve.entitlements.plist -r xhyve.rqset $(TARGET) --deep
+ codesign -d -r- $(TARGET)
diff --git a/xhyve.entitlements.plist b/xhyve.entitlements.plist
new file mode 100644
index 0000000..e0e11cf
--- /dev/null
+++ b/xhyve.entitlements.plist
@@ -0,0 +1,19 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
+"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+ <dict>
+ <key>com.apple.security.app-sandbox</key>
+ <true/>
+ <key>com.apple.security.inherit</key>
+ <true/>
+ <key>com.apple.security.network.client</key>
+ <true/>
+ <key>com.apple.security.network.server</key>
+ <true/>
+ <key>com.apple.vm.hypervisor</key>
+ <true/>
+ <key>com.apple.vm.networking</key>
+ <true/>
+ </dict>
+</plist>
diff --git a/xhyve.info.plist b/xhyve.info.plist
new file mode 100644
index 0000000..d5fcee9
--- /dev/null
+++ b/xhyve.info.plist
@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
+"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+ <key>CFBundleDevelopmentRegion</key>
+ <string>English</string>
+ <key>CFBundleIdentifier</key>
+ <string>hypervisor.xhyve</string>
+ <key>CFBundleInfoDictionaryVersion</key>
+ <string>6.0</string>
+ <key>CFBundleName</key>
+ <string>xhyve</string>
+ <key>CFBundleVersion</key>
+ <string>0.3</string>
+</dict>
+</plist> From my reading of this issue and #75 it seems as though no one actually has this working yet? @rimusz it looks like you never got this working for your coreos-ui app? But, if anyone has an entitlement /code signing/ sandbox solution working, please help me know what I'm doing wrong :) |
@dalehamel No, I never did get it working; I ran into the same issue you did. Maybe one of us should open a radar bug (or a TSI) and see if we can get a response from Apple? |
@geoff-codes taskgate isn't complaining here when building it from my Xcode project:
Do you have issues when building like that? If not, I suggest you figure out what is different about the Makefile build failing. |
Are you able to actually start a Guest with networking without sudo using On Monday, 7 December 2015, Jeremy Huddleston Sequoia <
|
@geoff-codes i could file one, i found out my company has access to two. Perhaps we can all put our heads together to come up with a succinct description of the problem? It would be great to submit the xcode stuff, as it is likely going to be better supported by apple. Once that's working we can always rev eng it. |
Yep. ~> git clone git://github.com/jeremyhu/xhyve
~> cd xhyve
~> xcodebuild CODE_SIGN_IDENTITY="Developer ID Application" # Actually, ad-hoc signing seems to do fine with these four entitlements.
~> build/Release/xhyve --help # Yes, this is fine...
Usage: xhyve [-behuwxACHPWY] [-c vcpus] [-g <gdb port>] [-l <lpc>]
...
~> ln build/Release/xhyve build/xhyve
~> ./xhyverun.sh # ... but this isn't. :(
vmx_init: processor not supported by Hypervisor.framework
Unable to create VM (-85377018) In ~> xcodebuild clean
~> xcodebuild CODE_SIGN_IDENTITY="Developer ID Application"
~> ./build/Release/xhyve --help
'./build/Release/xhyve --help' terminated by signal SIGKILL (Forced quit) ... and now its taskgate complaining: and we get this crash report:
@dalehamel I would say maybe lets hold off on that for now, as long as we have our man @jeremyhu on the case. He might actually be able to see something on the |
Ah, /facepalm. I copied an earlier entitlements.plist into my xcode patch and didn't look back. I just assumed that it was the same and based my earlier reply on that false assumption (and the comment that the error occurred by taskgated denying the entitlement). Sorry for the confusion. I'm looking into it. |
I asked around, and it turns out that those entitlements are "App Store”-only entitlements and can not be self-signed or signed with a Developer ID. As such, I think the current options are unfortunately:
Both have obvious downsides and are sub-optimal. I'll continue looking into this problem space, but I likely won't have updates on this until (and if) changes in the base OS allow it. |
@jeremyhu so, if i got you right, you're saying that ALL OSX's sandboxing features regarding whitelisting of priv escalation ONLY work when apps are signed with an (Apple issued thru a valid ADC membership) certificate (which may look brutal but actually makes tons of sense, as it gives AAPL a quick way to revoke/blacklist it if need arises) but ALSO and ONLY got distributed via the App Store ? is this specific to Hypervisor.framework or, in general, |
Looking. At Xcode, it only lets you do sandboxing on App Store apps. So it Of course, a workaround would be for someone to make a free xhyve app. I'll On Monday, 7 December 2015, António Meireles notifications@github.com
|
I wonder if an App Store App would get approved. Are pure-CLI tools allowed? Looking at the Mac App Store Review Guidelines, it looks like this might be trouble:
So maybe we need to put the xhyve binary in #!/usr/bin/osascript
display dialog "The xhyve binary is in the application bundle." buttons {"Awesome"} with icon path to resource "Terminal.icns" in bundle {POSIX path of "/Applications/Utilities/Terminal.app"} 😆 🙊 |
@AntonioMeireles Yes, after doing some move digging/experimenting it appears that is correct. I've tried:
Apparently, this business of App Store-only entitlements has actually been around for a while. The most illuminating piece of "documentation" regarding this business is @landonf's task-unchain. Alas, I even tried applying this patch to my 10.11 install; no dice. I still think this is pretty bizarre. @jeremyhu, you mentioned,
Have you actually been able to do this? What's the secret? |
My boot-args are "-v debug=0x146 kext-dev-mode=1 amfi=0xff cs_enforcement_disable=1" -v : verbose boot Plus, I set the following preferences:
The first overrides requirements for restricted entitlements. The second causes the system to not mark the process as having codesignature restrictions (specifically, csops(pid, CS_OPS_MARKRESTRICT, NULL, 0) is not called on the process). What you require is a subset of all of that. |
Out of curiosity, why do you have com.apple.security.inherit in your entitlements.plist? I don't see any calls to fork(2), posix_spawn(2), system(3), or popen(3) in xhyve, so there should be no child processes. Am I missing something? |
@jeremyhu no, you're not missing anything. that is/was specific to my use case. there (https://github.com/TheNewNormal/corectl) i hacked things around a cli so that one could call xhyve in detached or non-detached mode, or even lunch multiple VMs in a single pass, and for that com.apple.security.inherit would end handy. |
👍 |
I think this issue should be closed as it doesn't not appear to be possible On Tuesday, 3 May 2016, Joaquin Menchaca notifications@github.com wrote:
|
Closing since it seems impossible. |
Is this still the case? |
No. If you build using the Xcode project, it should setup the entitlements correctly for you. |
Hi, I've been spending quite some time trying to build xhyve using xcodebuild according to the instructions. Trying to run the resulting xcodebuild app, results in: $ xhyve -h I've tried various methods from within Xcode as well. Instructions for xhyve say that:
Is my reading of this thread correct, that build with xcodebuild as per the instructions is not longer possible? It's possible that make is working, but I can't get xcodebuild to produce a functional exectuable, and it appears from this thread that there may be no point in this, since root may well be required regardless of whether or not the executable is signed. Is that correct? (I'm not a programmer.) FYI, it appears others have run into this problem as well (google translate helps): https://cloud-atlas.readthedocs.io/zh_CN/latest/kvm/macos/xhyve.html
https://houraku365.hatenablog.com/entry/2019/01/06/ubuntu18-on-xhyve-on-macosx-failed-again
|
Hm. Looks like from the last couple of posts that build & run without su using xcodebuild is now supported. I am using a self-signed certificate. Do I need to pay $99 for a certificate-authority issued cert to get this to work? |
FYI, originally on xcodebuild I would get an app xhyve that terminates with "killed -9" (with or without sudo). I just tried following these instructions: https://kubadownload.com/news/codesign-sign-app codesign --force --deep --sign - xhyve ... which re-codesigns xhyve. Now the app runs again with sudo, but still no networking. I also tried turning on sandboxing with networking from within Xcode, then re- code signing with "codesign --force --deep --sign - xhyve", with the same outcome. I must be doing something wrong if they are saying this should now work. I don't have a developer's account, but Apple support tells me one should not be necessary to run on my own machine. They say to follow these instructions: https://help.apple.com/xcode/mac/current/#/dev5a825a1ca which... has a link to https://help.apple.com/xcode/mac/current/#/dev23aab79b4 where it says "If you are not a member of the Apple Developer Program, Xcode will create a personal team for you." But still this produces xhyve that terminates with killed -9, and that on re-sign "codesign --force --deep --sign - xhyve" only runs with sudo. |
because your re-signing nuked the entitlements. The build coming out of Xcode should be signed. What's the output of
I don't know what you mean by this.
You should not need to re-sign. The binary is signed by Xcode. What does the crash log say? I suspect the SIGKILL is due to you signing adhoc. Please select a valid developer profile (pretty sure your free one is fine) to sign with, and you should be ok. |
Was finally able to get a developer account with Apple, but still no luck, now having made two code sign certificates, "Apple Development" and "Developer ID Application." I start with:
Catalina, ran 'sudo spctl --master-disable' to allow possibility of "Allow apps downloaded from anywhere" in MacOS security control panel. Running xcodebuild from within xhyve directory appears to go well until the code signing part at the end, where there is an error:
And again:
Based on a nebulous tip here: https://forums.developer.apple.com/thread/124529 , I tried alternatively:
Interestingly, the RegisterExecutionPolicyException error at the end disappears, but the resulting binary is still killed.
With:
I noticed that xcodebuild is not automatically selecting one of my signing certificates. Tried:
Still, the signing certificate is not chosen at the end
So I went ahead and built again and followed up with codesign command to re-codesign using the same parameters as were invoked in the build, but still no luck:
Same with this:
Crash log for xhyve always shows the same thing:
A bit of a loss here. I feel like invoking 'xcodebuild -UseModernBuildSystem=NO' should have picked out a better signing certificate from the get-go, and not gone with 'Signing Identity: "-"'. But I'm not able to force it. Must be doing something wrong, just not sure what. Change the bundle identifier? Add the mac to a provisioning profile and again try to codesign with the "Apple Development" certificate as a development version? Seems like the executable might expire before long in that case? |
On the bright side, I was able to get Ubuntu Server 20 running on xhyve very nicely (with sudo, nothing to do with this thread). I used the legacy BIOS server install image (as opposed to regular UEFI), found here: http://cdimage.ubuntu.com/ubuntu-legacy-server/releases/20.04/release/ ... not absolutely sure if that was critical, but at the very least it made things a lot easier. Also, I removed "acpi=off" from CMDLINE in the install script. Absolutely critical was to pay close attention to the disk partitioning during install, and to apply a manual disk partition. Perhaps because of disk lettering (vba1, vba2...), the installer does not apply appropriate formatting to the partitions. I manually made a single ext4 partition for ubuntu, and manually marked it as bootable. People are having trouble with Ubuntu 18 (#161) and I suspect are running into the same troubles with 20. |
"-" means that you're still AdHoc signing. You're not using the signing certificate you got. You need to select it using the CODE_SIGN_IDENTITY build setting when you build. If you don't know how to do that, just open the project in Xcode, select the project to open the project editor. Select the target and then open the "Signing & Capabilities" tab.
Don't believe everything you read. That has nothing to do with it. You just need to select the right signing identity.
Correct. You need to specify it.
You could just resign it, preserving the existing entitlements in the ad-hoc signature. Check
Please bare in mind that the code signature is cached on an inode. If you try copying the (valid) contents over an existing file, the system will terminate the process. You will need to unlink the target (eg: with mv instead of cp). That is one of the most common mistakes folks run into with invalid code signatures that appear valid on disk.
What makes you think that? I suggest using XCBuild (the modern build system). |
Jeremy, thanks for all your help here. Not sure I know enough about what I'm doing here to get this to work, and I'll just wait it out until someone else who runs into my problem figures out a solution. No biggie.
FYI, I was finally able to get xcodebuild to attach my valid signature, by changing out "-" for my certificate ID (e.g. CODE_SIGN_IDENTITY = Apple Development: Dex Wolfram (GHJRE3745T) )on line 6 of common.xcconfig: https://github.com/machyve/xhyve/blob/master/xcconfigs/common.xcconfig . Otherwise, I couldn't get xcodebuild to override the "-" using CODE_SIGN_IDENTITY on xcodebuild command line, so switching it out in the xcconfig file worked. It signs with my ID, but I still get: $ ~/xhyve/build/Release/xhyve -h Console gives me:
And the crash log is pretty much the same. Signature doesn't look obviously bad: $ codesign -dv --verbose=4 /Users/ I also tried to disable gatekeeper ("Allow apps downloaded from anywhere"):
... and this little trick: https://eclecticlight.co/2019/10/22/catalina-crashes-non-notarized-command-tools-with-a-quarantine-flag/ (though I didn't use his app, just copied the quarantine flag attribute from another executable to xhyve), still no luck. Again, thanks for you help. I'll just let this sit for a while. Who knows, maybe I'll come across the problem later. Could be something with the certificates. I'll post back if I do. |
I'm also having this problem with a fresh xhyve checkout. I've tried picking my own (paid) team and using the Mac Developer certificate, but no go. But if I remove This is Xcode 11.3.1 on 10.14.6.
From Console.app:
|
I'm trying to figure out how to do a similar thing, and for what it's worth I found this: https://developer.apple.com/forums/thread/116870?answerId=361392022#361392022
|
This is the main pain point with xhyve. I'm very interested in getting this fixed or getting a workaround.
Could we detail the problem and possible solutions/work arounds, here.
The text was updated successfully, but these errors were encountered: