macOS: implement device capture to force unload of KEXT drivers

[mcuee]

Pull request https://github.com/libusb/libusb/pull/893 has been closed but the idea seems to be pretty interesting.

From the author osy:
It's a well known issue on macOS that you cannot use libusb to attach to an interface already claimed by an KEXT for exclusive access. The common workaround is to do sudo kextunload or inject a codeless KEXT. Both require admin permissions, and does not work for all devices on latest macOS. Additionally, SIP has to be disabled to inject a codeless KEXT, making it a pain to work with USB devices.

This patch uses the kUSBReEnumerateCaptureDeviceBit introduced back in macOS 10.10 which will tell the kernel to unload all other drivers for you without needing any dirty hacks.

Unfortunately, a side effect is that it will reset (simulate unplug-replug) the device to do this and so I didn't want to make it the default behaviour of libusb_open as that may, for example, cause all USB devices to be reset if the application decides to read the product string of every device attached. I can't find a good way to change this without touching the API specs, so I introduced a new option LIBUSB_OPTION_FORCE_CAPTURE_OPEN that lets you toggle the "capture" behaviour.

Also, the way libusb_close works makes it a bit awkward to implement the "release" of the captured device because just closing the handle does not tell the KEXTs to start matching again and calling USBDeviceReEnumerate with kUSBReEnumerateReleaseDeviceBit will also cause the device to be reset which then makes the next call of libusb_open fetch a stale cached object and fail. However, calling darwin_reset_device to address the stale cache issue will cause the device to be opened again, which causes another "capture" and then the OS will never get the device back. So the workaround of this is as follows:

1. When darwin_close is called, if the device is captured, then darwin_reset_device will first be called.
2. darwin_reset_device_capture will call USBDeviceReEnumerate and release the capture.
3. darwin_restore_state calls darwin_open indicating that capture should NOT be used.
4. darwin_open succeeds and the device is re-opened without the capture bit.
5. Call returns to darwin_close who calls USBDeviceClose on the non-captured device handle.

[mcuee]

New pull request #911

[supercairos]

Looking forward to this feature.

[mcuee]

Close this now that #911 has been merged.