Weird issues with LUFA on macOS (sierra and yosemite)

[nekromant]

I'm using LUFA for my own multi-mcu/cross-platform bootloader that is using that HID trick to bypass windows signing once more and ran into a very weird issue:

• In macOS on a real mac with usb3.0 IOHIDDeviceSetReport seems to fail randomly, returning -536870163 or -536850432 when it fails.
• On a hackintosh laptop that is my usual testing rig for mac builds everything works with usb 3.0 ports (Those ports are using opensource, non-apple GenericUSBXHCI.kext driver), but when plugged into usb 2.0 ports that are handled by the apple driver same thing as on a real mac happens.
• The same bootloader (e.g. using the same spec and descriptors) when compiled for a different avr using vusb seems to work with apple's usb driver with no issues, so I really think it might be a lufa proplem.

The bootloader userspace tools are here: https://github.com/uISP/uHID-userspace
The bootloader itself (lufa and v-usb stacks): https://github.com/uISP/uHID-bootloader-avr

The same code works with no problems on the same hardware under windows, linux (hidraw and libusb alike) and freebsd, only mac is giving me pain (
Any idea where to start looking for the bug?

[abcminiuser]

Unfortunately I don't have any Mac equipment around for testing, so I can't debug this easily on my end. Do you have access to a USB analyzer?

Is there a canonical table of OSX error codes that includes those return values, so we can at least take a guess as to what might be the failure cause?

[nekromant]

I'm not a really big mac expert myself, just know a handful of tricks. The only thing I've found is this: http://krypted.com/lists/comprehensive-list-of-mac-os-x-error-codes/ and it doesn't list the numbers I got. Judging by the fact that the error code:

1. sometimes changes
2. is a really huge number
   it might well be some memory corruption, but I'll look into that.

I had some saleae-compatible 8ch logic analyzer somewhere around. I'm not sure it will be fast enough for usb fs, but I'll give it a try. Thanks for your quick reply.

[nekromant]

@abcminiuser Sorry for the delay, took me a long while to find my old logic analyzer capable of capturing and decoding usb fs. Here go the dumps:

https://cloud.ncrmnt.org/index.php/s/iYSCHlKwTc2uEW1

Idle is when no software (save for the mac HID driver, I guess) is talking to the device.
The other one is when my uhidtool is trying to read flash from the target MCU by continuously sending HID Get Feature report on the control endpoint and ultimately failing.

You can view the dumps using software from saleae.com (That, luckily, has a built-in usb protocol decoder).

P.S. I have my usb analyzing setup ready, so if you need any further dumps/experiments I'll be glad to help out.

EDIT: particulary interesting is the fact that there are error packets on the bus every now and then, when the device is supposed to be idle.