Windows: hid_enumerate() skips devices that have already been opened

[mrpippy]

I am trying to use the latest hidapi with the Qt-based HID Bootloader software written by Microchip (which is included in the latest Microchip Application Library and located at Microchip Solutions v2011-07-14/USB/Device - Bootloaders/HID/Software - Cross Platform/)

This app runs hid_enumerate on a timer (once a second by default), always searching for its given VID and PID. If the device is found, it opens it with hid_open() and starts communicating. However, the next time hid_enumerate() runs, it is unable to open the device (because of the '0' passed for the share value), thus unable to read the VID/PID, and returns a linked list without the device included.
To the app, this means that the device is now disconnected, and it calls hid_close(). Then the device is detected when hid_enumerate() runs, and the loop starts over.

I was able to fix this problem by removing the first (0 share value) CreateFileA call in open_device (so CreateFileA is always called with FILE_SHARE_READ|FILE_SHARE_WRITE).
Is there any better way to solve this problem though? Another way to detect device disconnection maybe, or (even better) a more lightweight or callback-based way to get device connection/disconnection events?

[signal11]

On 09/27/2011 09:35 PM, mrpippy wrote:

I am trying to use the latest hidapi with the Qt-based HID Bootloader software written by Microchip (which is included in the latest Microchip Application Library and located at Microchip Solutions v2011-07-14/USB/Device - Bootloaders/HID/Software - Cross Platform/)

Great!

This app runs hid_enumerate on a timer (once a second by default), always searching for its given VID and PID. If the device is found, it opens it with hid_open() and starts communicating. However, the next time hid_enumerate() runs, it is unable to open the device (because of the '0' passed for the share value), thus unable to read the VID/PID, and returns a linked list without the device included.
To the app, this means that the device is now disconnected, and it calls hid_close(). Then the device is detected when hid_enumerate() runs, and the loop starts over.

Unfortunately, that's the way Windows works. If a device is there and is
already opened exclusively (without the share flags), it won't show up
in the enumerate list (from Windows).

I communicated with the guy who wrote that software from Microchip, and
I thought he used a -1 return value from hid_read() (or hid_write()) to
determine that the device was disconnected.

I was able to fix this problem by removing the first (0 share value) CreateFileA call in open_device (so CreateFileA is always called with FILE_SHARE_READ|FILE_SHARE_WRITE).

Yeah, if you do that every time, there's no way to make a device
exclusively opened (which is what's desired). For some reason unknown to
me, some devices must be opened with sharing, and others do not require
this. It also varies I think with versions of Windows.

Is there any better way to solve this problem though? Another way to detect device disconnection maybe, or (even better) a more lightweight or callback-based way to get device connection/disconnection events?

Try the checking hid_read() for -1 return value to determine if the
device is disconnected.

Alan.

[mrpippy]

To the app, this means that the device is now disconnected, and it calls hid_close(). Then the device is detected when hid_enumerate() runs, and the loop starts over.

Unfortunately, that's the way Windows works. If a device is there and is already opened exclusively (without the share flags), it won't show up in the enumerate list (from Windows).

I communicated with the guy who wrote that software from Microchip, and I thought he used a -1 return value from hid_read() (or hid_write()) to determine that the device was disconnected.

The code does gracefully handle a -1 return value from hid_read/write() (presumably in case a device is disconnected in the middle of an operation), but device connection/disconnection status is done with hid_enumerate(). Just relying on hid_read() wouldn't work, since we want to know about a connection/disconnection even when there isn't I/O going on.
The hidapi included with the Microchip code is older (there's no open_device function), but it has FILE_SHARE_READ|FILE_SHARE_WRITE as the share mode for all the CreateFileA calls

[signal11]

On 09/28/2011 02:04 AM, Brendan Shanks wrote:

The code does gracefully handle a -1 return value from hid_read/write() (presumably in case a device is disconnected in the middle of an operation), but device connection/disconnection status is done with hid_enumerate(). Just relying on hid_read() wouldn't work, since we want to know about a connection/disconnection even when there isn't I/O going on.
The hidapi included with the Microchip code is older (there's no open_device function), but it has FILE_SHARE_READ|FILE_SHARE_WRITE as the share mode for all the CreateFileA calls

I would expect the code with the Microchip lib to be a bit older. The
read flags must be a local modification that they made.

HIDAPI doesn't currently support hotplug notifications.

Alan.

[signal11]

On 09/28/2011 02:04 AM, Brendan Shanks wrote:

The code does gracefully handle a -1 return value from hid_read/write() (presumably in case a device is disconnected in the middle of an operation), but device connection/disconnection status is done with hid_enumerate(). Just relying on hid_read() wouldn't work, since we want to know about a connection/disconnection even when there isn't I/O going on.

Why could you not call hid_read_timeout() at an interval to find out if
the device has been disconnected? The result would either be:
> 0, there's data, and it needs to be processed anyway,
0, there was no data, but the device is still present,
< 0, the device is disconnected.

[vanweric]

I think I found a solution - it is working well on my box here.

In CreateFileA in the enumeration step, the parameters are dwDesiredAccess = 0, dwShareMode = 0.

In CreateFileA in the open path step, the parameters are dwDesiredAccess = GENERIC_WRITE | GENERIC_READ, dwShareMode = 0.

This will allow enumeration to pull parameters from open devices, but after a device is opened subsequent calls will (correctly) fail.

Should I push this change back up?

[signal11]

Unfortunately, some devices must be opened in share mode. They will not open in exclusive mode. I'm not entirely sure what causes it. HIDAPI opens devices in exclusive mode if possible to be compatible with other OS's.

[vanweric]

Which devices must be opened in shared mode? I'd like to take a look into them. Is it just keyboards / mice?

What is the behaviour on Mac / Linux? I am away from those test boxes at the moment.

[signal11]

I'm not sure the actual criteria which determines whether shared mode is required. Keyboards and mice cannot be opened at all in Windows using the HID interface, and thus can't be opened using HIDAPI.

On Linux, it depends on the implementation you use. For Linux/hidraw, devices can be opened multiple times. On Linux/libusb, they can only be opened once. Actually, opening it a second time will steal the device from the first instance. On mac, it allows multiple opens.

[modtronix-com]

Hi, I am having the same issue. I wrote a modified version of the Microchip USB HID Bootloader. It uses an old version of HIDAPI. I updated with latest version of HIDAPI, and found the same issue. It goes into a connect/disconnect loop. As explained above, whenever hid_enumerate() is called when device is open, it returns NULL.

I could also solve the problem by specifying FILE_SHARE_READ|FILE_SHARE_WRITE as the share mode for all the CreateFileA calls. Does anyone know if this is the preferred method? My application will only be used with Microchip PIC USB HID devices.

[signal11]

hid_enumerate() returns NULL if there are no devices. Devices which cannot be opened (ie: they are already opened with no sharing), do not count on Windows. This is the way the WIndows native libraries are implemented.

Don't use hid_enumerate() to detect device disconnection. Use something like hid_read() returning -1. This will work on all platforms.

[mrpippy]

I'm just specifying FILE_SHARE_READ|FILE_SHARE_WRITE as the share mode for all CreateFileA calls, and using hid_enumerate() to detect disconnection.
I would have to change the code significantly to use hid_read() to detect disconnection, and I'm not totally convinced it would work.

[signal11]

I am :)

hid_read() will return -1 if the device has been disconnected on all platforms. If you can find a case where it doesn't, please report it, as it will be a bug. Doing it the way you do it will allow multiple programs to access the device at the same time, which can be less than desirable.

[modtronix-com]

Thanks for all replies! I have the same issue as mentioned above, I want to know when device is disconnected even when there is no I/O with device. The hid_read_timeout() suggestion from "signal11" might work, and I will try to implement that into my code.

[quote]
Why could you not call hid_read_timeout() at an interval to find out if the device has been disconnected? The result would either be:

0, there's data, and it needs to be processed anyway,
0, there was no data, but the device is still present,
< 0, the device is disconnected.
[/quote]

[mrpippy]

Oh yeah, I should have been clearer. I'm sure it works, just not sure how complex it'll be to make this code use it. I guess it would be something like:

if device is disconnected, run hid_enumerate() periodically
if device is connected but there's no I/O happening, run hid_read_timeout() periodically
if device is connected and I/O is happening, make sure hid_read() and hid_write() gracefully handle

[signal11]

Is there still any issue here about which anything can be done?

[vanweric]

I made the change I suggested in my October 26th post locally, and it is working well for me. Is there any way to incorporate this change without affecting devices that require shared mode? Perhaps open could include a parameter to determine whether or not it should be shared?

[signal11]

I misread that comment the first time. To be sure, you're saying that calling CreateFileA() with DesiredAccess and ShareMode both as 0 during hid_enumerate() will make it show devices that are present but opened exclusively by another process?

That's not working on my XP system here. What are you running, and can you post your code for that somewhere where I can see it? There's some piece of it that I'm missing.

[vanweric]

I fear I must eat my words a bit: The implementation I proposed in October can misfire in a few cases. This should be a bit more robust:

In enumeration, I open with dwDesiredAccess=0, dwShareMode = ReadWrite
In communication, I open with dwDesiredAccess = ReadWrite, dwShareMode = Read
In both cases, lpSecurityAttributes = null, dwCreationDisposition = Open existing, dwFlagsAndAttributes = Overlapped or Normal, hTemplateFile = null.

This allows only a single communication handle to be open, but multiple enumeration handles to be opened.

I develop on Windows 7 64 bit. If you need, I have access to a test grid that includes XP, Vista (32 + 64), and 7 (32 + 64). I do not have ready access to a 98 box - we deprecated support a while back.

[signal11]

MrPippy and djhosken, please grab the branch named enumerate_fix and check for 3 things:

1. That the device can be opened.
2. When a device is open, that hid_enumerate() in a separate process still sees the device.
3. When a device is open, that hid_enumerate() in the same process still sees the device.
4. When a device is open, that the device can (or cannot) be opened by another process (I'm curious :) ).

Vanweric, I'm not sure if you have one of these "must-open-in-shared-mode" devices. Even so, give it a try.

Thanks!

Alan.

[vanweric]

Thank you Alan

I just ran a quick test on the enumerate_fix branch, and it looks good! I am able to enumerate the device regardless of what process it is open in. Additionally, I am unable to re-open an opened device (which is exactly what I need).

I think the "must-open-in-shared-mode" devices just mean that another driver has already laid claim to them. My unjustified guess is that the method in enumerate_fix branch should work for all devices that are only opened by HidApi.

Let me know if you'd like testing on additional OSs.

• Eric

[mrpippy]

I just gave it a try, using my app alone #1 and #3 both seem to be true.

As for #2, while my app was running and holding a device open I ran hidtest. The output was the same as when I ran hidtest without my app open, so I guess it's safe to assume that having the device open in another process doesn't affect hid_enumerate.

As for #4, I modified hidtest with my device's VID/PID. When my app was holding the device open, hidtest couldn't open it. When the device wasn't opened by another process, hidtest opened it fine.

So I think #1-4 are all true!

[signal11]

MrPippy, just to be sure, this is the device which previously required FILE_SHARE_READ|FILE_SHARE_WRITE before?

[signal11]

Pushed e3ab121

[mrpippy]

Yeah this is the app that constantly runs hid_enumerate() on a loop to detect device disconnection, previously I had to use FILE_SHARE_READ|FILE_SHARE_WRITE as the share mode for all the CreateFileA calls for it to work correctly. I don't think the device requires it, just the disconnect detect method that the code uses.

[TheOnlyJoey]

So, this is still a issue with the current Github version or 0.8rc.
We are testing with HMD hid devices running hidtext.exe and when the device is opened by another process it still does not show up (Tested on Windows 7 SP1 and Windows 10 with latest versions).
This is odd since setupapi shouldn't require device access and devcon works reporting these devices as well.

@signal11 Any chance enumeration could be looked at again? as far as i understand opening the devices should not be necessary when using the enumeration features in setupapi on Windows.

[signal11]

Feel free to pick around at the code and figure out what's going on. Some devices behave strangely. Maybe the other process is opening it in exclusive mode.

[TheOnlyJoey]

Regardless of exclusive mode or not, enumeration should always be possible (enumeration functions of setupapi can always enumerate every device). Maybe rewrite hidapi to use the setupapi features directly?

[signal11]

HIDAPI does use setupapi features directly. Have a look at the code for yourself. If you know setupapi, the code isn't hard to follow. Step through it with a debugger and see if you can isolate where the issue is. There's not a lot I can do on my end because I don't have your device, and different devices do some different (and weird) things.

[TheOnlyJoey]

So, i spend 2 days debugging this, and it seems HIDAPI is doing some extra unnecessary steps after enumerating the devices.
This includes already opening handles and such which is one of the primary reasons why certain exclusive opened devices can not be enumerated.

I created a test application to show the enumeration functionality, i will also work on a PR with a new enumeration function using this method.
http://pastebin.com/xPspjM8Q if there is any feedback please let me know.

[aspalmer]

When I try to open devices exclusively it doesn't work until I disable/re-enable the device in the device manager. It then works for open open. Is that likely to be fixed by your change? When the problem is occurring I can see an extra symlink to the USB device in Process Explorer.

[signal11]

What are these "unnecessary steps" you speak of?

[aspalmer]

@signal11 I think his point is that you can get all the info you need in hid_enumerate with SetupDi calls without calling open_device or any of the functions that use write_handle.

[signal11]

Which versions of windows will that work back to?