Cannot load 3rd party kexts #4026
Comments
|
Same root cause as #3526 |
|
This bug still exists in macOS 13.0 Beta (build 22A5266r), running as both host and guest. I tested with UTM 3.2.4. So Apple hasn't fixed it in Ventura -- at least not yet. |
|
This bug (Apple's bug) still exists in macOS 13.0 Beta 2 (build 22A5286j). |
|
Apple's bug still exists in macOS 13.0 beta 3 (build 22A5295h). |
|
Perhaps Applespecified it that way. They have moving away from kexts for awhile.
… Apple's bug still exists in macOS 13.0 beta 3 (build 22A5295h).
|
|
That's possible. But it may also just be an oversight -- Apple still supports them in macOS 13 "on bare metal". Only time will tell. I'll keep testing new Ventura betas. But I'll only report here if I find one that works. |
|
Ah just came across this problem myself, sadly after I posted about it on the dev forums. Oh well, maybe I'll get a reply. |
|
Still doesn't work in Ventura 13.0 and UTM v4.0.8 |
|
I reported the inability of enabling kexts on machines running via virtualization.framework as to Apple as FB10145502 (third party kexts cannot be used in Virtualization framework apps) a bit ago and as yet have heard nothing, for what it's worth |
|
To follow up on this, using bputil from a terminal in recovery mode gives what's probably the most useful hint: Error Domain 401, "Failed to create local policy", underlying error 0x600001f64150, domain=com.apple.bootpolicy, code=2, "SEP communication (2)" So it's almost certainly going to be that Virtualization.framework neither provides access to the real SEP nor emulates one fully, and so, until Apple decides to properly support Virtualization, we're just sad. |
|
@dariaphoebe, I'm thinking of trying to debug this on an Intel Mac, since I can use HookCase there. Any suggestions how I might go about it, or is it a lost cause? Is the Intel environment just too different from the Apple Silicon environment? If I'm really lucky, I might be able to find a hackish workaround for the problem. Something like #3904 (comment). But I might also end up wasting a lot of time. |
|
Ah I only shared the log output in the Apple Dev Forums; the only lines I see are: but could be unrelated. |
|
I've made some progress debugging this problem. But I haven't, unfortunately, found a workaround. And what I have found indicates it happens at a very low level, and is almost certainly the result of a deliberate policy decision by Apple. After much digging around in One of the things I first noticed is that there isn't any "auxiliary kext collection" at all in macOS 12 and 13 VMs (created using the Virtualization framework). (Note the output from macOS 13 macOS 12 So a "linked manifest" is an expression of "local policy", and also of "boot policy". I'd guess "SEP" stands for something like "Secure Enclave Proxy". It's accessed through several software layers (including The "SEP command" that fails in VMs is "SEP command 33". It always succeeds on bare metal (at least in my experience). I see no references to it online. Over time I'll try to learn more about it. If Virtualization framework VMs emulate the "Secure Enclave", it should be possible, at least in principle, to mess with that emulation, to make "command 33" succeed. I'm sure this won't be easy, though, and I may never learn how to do it. Someone at Apple must have made a deliberate decision to make "command 33" fail in Virtualization framework VMs. That sits oddly with other indications that they intended to support third-party kexts in these VMs -- for example they let you allow third-party kexts in the Recovery Partition. But Apple is a large company, and this won't be the first time the left hand doesn't know what the right hand is doing. My logs have lots of entries headed "kcgend:". It's completely undocumented. But as best I can tell it's what actually writes "linked manifests" and kernel caches as your Mac boots. It's not meant to run in a VM, and normally doesn't (the result of another deliberate decision by someone at Apple). But I was able to find a way around this: Just run |
|
I've learned that "SEP" means "Secure Enclave Processor" ... probably. And "ACM" might mean "Authenticated Code Module". |
|
After more digging around, I discovered there are three kernel extensions that can receive "SEP commands". (These are the only three kexts that have Then I wrote a dtrace script to trace when any of them receives a SEP command. Rename this file from The easiest way to trigger SEP commands to But you don't see anything when you run or reboot a UTM VM (that uses the Virtualization framework). This tells me that Apple's implementation of the Secure Enclave in the Virtualization framework is an emulation -- possibly not a complete one. It doesn't call out to the Secure Enclave Processor in the host. Another interesting bit: There are three "AppleVP" kexts, one of which ( More interesting stuff: All but one of the "SEP methods" in |
|
I've found out what "SEP command 33" is -- I've also found where So next I need to figure out why the call to |
|
I've discovered some information about why There are many other cases where For what it's worth, all the failures happen when the call to "SEP command 23" is Above I theorized that these failures are the result of a deliberate policy decision by someone at Apple. I still think that. But now I believe I focused too much on the "SEP command 33" failures themselves. They may be just accidental. Now I suspect that the crucial decision was to not allow My logs above came from the output (on a VM) of Interestingly, I get no output from running |
|
I've found some potentially useful stuff. The AuxilaryStorage image appears to be an iBoot1 bootloader. According to https://github.com/AsahiLinux/docs/wiki/SW:Boot#stage-1-llbiboot1, it reads a local boot policy from According to that wiki page, it should be in "metadata keys", but although running I'm thinking of asking the how to get these metadata keys on the Asahi IRC, but I'm not sure which channel to use (as they don't have an offtopic channel). The closest I can think of is |
|
Yes, @ktprograms, it'd be interesting to know the contents of the local boot policy file in a VM that's been set up to support loading third party kexts (while booted into the Recovery Partition). I'd bet the I've got some more ideas about how to make They're based on two observations, one of which is a new discovery:
|
|
The page at https://github.com/AsahiLinux/docs/wiki/SW:Boot is very interesting. I notice that it mentions (and at least partially defines) "AP" and "kcOS" -- both of which show up in my research (in logs and in kexts' machine code). |
Anyone know of a utility that can do this? It will probably need explicit Apple Silicon support. A quick look on the web tells me I may need to write my own -- possibly as a "driver extension" (https://developer.apple.com/documentation/kernel/implementing_drivers_system_extensions_and_kexts). Edit: I already tried Edit: Edit: I also tried Edit: The Edit: The source code for Apple's |
|
WRT reading the local boot policy: I think |
|
Apple actually has pretty decent documentation on the local policy file: |
|
Here's a progress report ... or better yet a lack-of-progress report. I pretty quickly discovered that the Then I tried to force In principle you should be able to create a custom kext cache (using https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html But I couldn't get it to work. I tried adding I suspect Apple's I'm giving up on this, at least for the time being. |
|
Since there was "Ask Apple" this week, I did. Quinn confirmed it's not going to work yet, and i'd consider that definitive. |
I did this, and managed to get it to work. But yes, it took me down another rabbit hole. I did discover one useful bit of information, though: The failures have nothing to do with the kernel itself (custom or otherwise). In fact the kernel never runs at all when you try to use This may be some kind of permissions issue (though I loosened them as much as possible). But I think it's more likely that code somewhere in |
|
Could you test if the loop gets hit on physical hardware? |
|
It does get hit on physical hardware. To play it safer, I changed the loop to a So I didn't do a full test of my custom kernel (and its custom kext cache) on physical hardware. I don't want to risk harming my machine. The logical place to do this kind of test is in a virtual machine. Clearly Apple has gone to some trouble to prevent tests of custom kernels and third party kernel extensions from working in a Virtualization framework VM. It's rather hard to understand why :-( |
|
Thanks for testing that. I wonder if the VM's bootloader is somehow hardcoding "allowed" kernels. Do you think trying to boot a kernel with a different version that was used in another VM might work? (You could check with Also, is |
|
I doubt that
From Over the next few days (and possibly weeks), I'm going to dig into |
|
I'm still working on this, but very slowly. I've now ruled out On a hunch that interesting logs might be found on the host (instead of the guest), I looked there too, for entries matching My next direction will be using HookCase to debug the Virtualization framework on Intel Macs. It can't virtualize macOS, but it can virtualize Linux and Windows. The work is likely to be very involved, and will probably take a long time. I've already spent a lot of time using HookCase to debug |
|
Exactly the same nonsense still happens on the first macOS 14 beta (build 23A5257q). In a macOS 14 guest (running on a macOS 14 host), after using And once again, the same kernel extension loads just fine on the host. I tested using UTM 4.2.5 (the current version). I've made some progress reverse engineering the |
|
This is a problem with SEP (or, more exactly, with its s/w replacement made specifically for .vmapple image because there is no real SEP). Kernel manager and boot policy app tries to query some sort of permissions to build a manifest for auxiliary kernel cache (AuxKC) using a special SEP command and during that an error is thrown which makes AuxKC building process to fail, thus any kernel extension you have approved via "Privacy & Security" are not incorporated into auxiliary kernel cache during reboot process when kcgend daemon is started. |
|
Very interesting, @flatz, that you were able to patch a bunch of system files. How did you do it? I assume you had to lower the security settings, but which values did you pick? |
|
@steven-michaud Unfortunately you can't just lower security settings to do that. As I told earlier I've started from the Root of Trust, by patching ROM image (AVPBooter binary from Virtualization.fw) to skip signature checks of iBSS/iBEC, ramdisk, kernel, etc. |
|
So it looks like the only file you patched is What do you mean by "patch"? Did you change the file on disk? If so, how did you manage that? Did you boot into Recovery mode and change it there? Did you also need to re-codesign the Virtualization framework? Did you need to set Permissive Security in the Recover mode's Startup Security Utility? Or did you change the file's image after it was loaded into memory? If so, how did you do that? |
|
I've learned on my own how to change macOS system files: How to Defang macOS System Protections. I made a trivial change to I'll be playing with Thanks, @flatz, for pointing out that it's an interesting target for investigation. |
|
I'm glad to see that you got the point. But keep in mind, just patching of AVPBooter is not enough because it's just a single piece of boot-process, you need to patch iBoot too (which includes iBSS/iBEC and their duplicates in LLB/iBoot) to be able to make patches for kernel, etc. Your end goal is patching ACM check in AppleVPBootPolicy.kext (which itself is embedded into kernelcache), this check (see function called __validate_acm_context) prevents generation of manifest for AuxKC. |
|
I'm going to take a different approach. My theory (above and above) is that the Ultimately only Apple can really fix the AuxKC problem. But so far they don't seem the least bit interested in doing so. |
|
I've figured out how to use third-party kexts on Virtualization framework macOS clients! Please check it out. |
|
Above I talked about custom kext caches not working in macOS guest VMs. My workaround for third-party kexts doesn't directly address this problem. But if Stage 2 iBoot ( I used |
|
Apple still hasn't fixed this bug, but I've discovered a sign they might be working on it. |
Describe the issue
3rd party kexts, like macFUSE or the kext I am working on personally, won't load. After reboot I get a pop-up message that says "System Extension Error", "An error occurred with your system extensions during startup and they need to be rebuilt before they can be used. Please go to Security & Privacy System Preferences to re-enable them."
If I go to Preferences and do as it asks, after reboot the same message appears.
SIP is disabled, developer mode is on, etc.
The same kexts work fine on bare metal.
Configuration
Logs
Output of
log show --debug | grep -i macfuse:The text was updated successfully, but these errors were encountered: