Provide binary wheels with libusb dll for Windows

[bauerj]

Providing a binary wheel with libusb-1.0.dll would make the installation as simple as pip install libusb1.

Are there any licensing issues that prevent you from shipping a copy of that library?

[vpelletier]

libusb is licensed under the GPL (LGPL v2.1 actually), so there is no restrictions on redistribution.

I'm not very tempted about bundling it with python-libusb1, but not for licence reasons:

• I will inevitably fall behind with their releases, as I will likely have to reference whatever is the latest version and bump the wheel. I know I'm already lagging behind in API support around USB3-specific functions, I'd rather spend time to fix this.
• I will have to support a build toolchain I'm not too familiar with (even on linux: I just use the one packaged by my distribution) on an OS I rarely use (and even more rarely build software on), so I will do mistakes and accidentally prevent installation for extended periods of time.
• There is more needed on windows than just the libusb1 dll: there are the various kernel-level drivers (WinUSB vs. libusbK, and maybe some other), and the .inf generator which binds these to devices when plugged in so libusb can use them... So I would likely have to also bundle these somehow, and make a poorly informed/not always fitting choice of the one to bundle.

[bauerj]

Thanks for clarifying.

I could help with setting up automated builds using appveyor. This would solve your second point and maybe the first one too, since it would automatically pull in a new version with each release of python-libusb1.

I didn't know about the need to use special drivers though. I didn't have to install any drivers to use python-trezor (which uses this library) but I assume that's not true for all devices.

[vpelletier]

While I am not familiar with python-trezor, I see it supports several ways to access the device, including HID. HID is much more convenient to use on windows than "raw" USB, precisely because it does not require a specific driver (which in turn prompted many device makers to use the HID class, I believe, for example I have an NFC-ish dongle which uses it). Maybe this is what is used in your setup ?

python-libusb1 loads the library and pulls all entry points when the usb1 module is imported, so it would complain if it is missing even if it were not actually used. Or it could be used to list connected devices, find nothing (because no supported driver would be present) and containing app could carry on enumerating with other mechanisms. I believe libusb development team recommends using HID-specific libraries rather than implementing boiler-plate HID stuff over libusb on each use.

Then again, I see they are referring to WebUSB, which I do not know and which apparently handles the driver part. Looks like I'm lagging behind in libusb-related knowledge on windows.

[whitequark]

@vpelletier Would you be open to reconsidering this? I understand the reluctance, but I'd like to address all three points here:

• So far, upstream libusb shipped a release about once per 1.5 years, and python-libusb1 about 3-4 times faster than that.
• You don't have to build the libusb shared library yourself. I would actually find it quite strange if you did—that doesn't seem to solve any problems, but creates quite a few for everyone involved. Upstream libusb provides Windows binaries, which you can easily repackage in the binary wheels. You could do it just fine from your *nix machine, too.
• WinUSB, available since XP (realistically since Vista) does not require .inf generators. In fact, it doesn't require any intervention—as long as the device exports the right descriptors, the kernel mode driver is installed transparently the first time it's plugged in.

To add to the last point, I believe that even if we disregard WinUSB, packaging the libusb shared library would significantly improve the Windows experience. Please consider this from my perspective of an application developer trying to make installation process as painless as possible:

• Installing drivers through Zadig, a simple GUI application, is not the most straightforward or most foolproof way to install drivers (that would be WinUSB, which requires no interaction whatsoever), but it comes close. If I could not use WinUSB, I would only need to include three screenshots in my documentation, and that would be sufficient for virtually all Windows users. The installation process is exactly the same on every Windows system, it does not have any dependencies, and you typically only do it once.

• There is no procedure for installing the libusb1 shared library. Its releases are distributed in .7z or .tar.bz2 files, neither of which Windows can open out of the box. The archive contains 12 library files, of which only 2 would actually work on any given Python installation.

  Where do you install it? Well, you could put it in system32 (or syswow64, good luck explaining why the first one is for the 64-bit DLL and the second one is for the 32-bit one). But that's horrible, so you'll probably put it into the Python installation.

  Which Python installation? It could be the MSI installer (configured either as user-specific or system-wide), the Windows Store installer, something that uses Anaconda/Miniconda, or something worse. There could be many versions installed concurrently. All of those (except for the Store one) could be both 32-bit and 64-bit, and it is often not obvious which, yet this is critical because the 32-bit and 64-bit DLLs can't be co-installed. The official MSI installer in the recommended configuration installs Python to C:\Users\<user>\AppData\Local\Programs\Python\PythonXY, which I had to Google because that's about the last place you'd expect it to be. Also, AppData is hidden, and if you don't know that, you might be confused by its apparent absence!

  The Python installation has a directory temptingly called "DLLs", which it will not load libusb-1.0.dll from if you put it there—it has to be next to the .exe. The Windows Store installation is immutable (it uses reparse points), so you can't put the library there. You have to repeat the process (which you likely blissfully forgot) when you upgrade or reinstall Python. And there's no way I can guide people through something like this in my documentation.

  I've been doing some sort of systems programming for most of my life. I knew most of those things offhand. Still, I had to do both some minor searching and minor experimentation to actually get it to work. Most people would suffer far more.

Packaging the libusb1 shared library would eliminate the second step. If there is working pip, there is working libusb, that's it. What remains is, at worst, three clicks in Zadig, that's if the device isn't using WinUSB, and only until someone packages libwdi for Python to automate the .inf generation, too.

Let me know if you'd like me to send a PR implementing this—it would be only a few lines of code. Now that I've shipped YoWASP, python-libusb1 is the last thing that stops Glasgow from having a Windows installation procedure that consists of a single pip call on top of stock Python.

[vpelletier]

Thanks for the very good points and insight into the libusb windows install process.

Some not-too-structured thoughts:

• No problem with including dirname(__file__) (or so) to the ctype lookup path on windows (which, as I understand it, is where the dll will be found).
• When installed on OSes which package libusb (so *nix and maybe OSX), I would like to rely on the "system" library (for whatever ctype thinks "system" is).
• My OSX experience is getting very old, but as I remember it the libusb installation methods are not exactly uniform (each package manager maintaining its own file tree), maybe it could benefit from this approach as well ?
• Bringing back the license point brought by bauerj, there needs to be some pointer to source code of the packaged binary library, ideally to the same version. I think this is what I had in mind when I mentioned the build chain: if I package the binary I also have to keep track of the source, so maybe just pull the source and automate the build somehow, then the source is already there.
• I would like the library to be downloaded, extracted and put into the wheel automatically on wheel build. An extra (maybe shellout) bdist-time dependency to 7z seems fine to me. Pulling the libusb.info page and locating the windows download url is fine too. Having an expected hash (and hence wheel build failing on my side when a new version is out, so I can check it a bit) is also fine.

I'm not asking you to provide implementation or answer to each of these thoughts, but rather your opinion on whether these make sense.

Let me know if you'd like me to send a PR implementing this

Yes please. I do not have enough experience with python wheels, so every bit helps. And again, do not feel required to implement the points I just mentioned.

[whitequark]

• No problem with including dirname(__file__) (or so) to the ctype lookup path on windows (which, as I understand it, is where the dll will be found).

I lean towards always loading the bundled libusb-1.0.dll if it exists, for a few reasons.

• If it's inserted in the front of ctypes lookup path, that's just directly loading it but with more steps.
• If it's inserted in the back of ctypes lookup path, this could cause issues. If the system one is newer than the bundled one, then things are probably fine. If the system one is older (and who knows what kind of junk decided to put its private libusb-1.0.dll onto PATH?) then things will probably break in a confusing way.

In case someone really does want to use system libusb for some reason, it is easy enough to do: pip install --no-binary (this uses an sdist), or pip install --platform any (this uses an arch-independent bdist) would pull a package without the bundled library. The --no-binary option can even be used to force installing an sdist of python-libusb1 when it is being downloaded as a transitive dependency.

• When installed on OSes which package libusb (so *nix and maybe OSX), I would like to rely on the "system" library (for whatever ctype thinks "system" is).

Indeed. This isn't really viable anyway. It's not completely technically impossible to build such an universal wheel (on Linux at least), but that approach has too many flaws to seriously consider shipping it.

• My OSX experience is getting very old, but as I remember it the libusb installation methods are not exactly uniform (each package manager maintaining its own file tree), maybe it could benefit from this approach as well ?

I've heard that these days people mostly use brew and based on asking a small amount of macOS users I know, installing libusb1 from brew would not present any nontrivial obstacle. So this can probably be safely skipped, at least at first.

• Bringing back the license point brought by bauerj, there needs to be some pointer to source code of the packaged binary library, ideally to the same version.

libusb1 is released on GitHub, so each binary archive inherently has a corresponding source archive produced by GitHub. I'd personally expect that a link to the official release page would be enough (given that there are cryptographic hashes for both sources and binaries proving that you indeed are distributing the upstream version of libusb1).

• I would like the library to be downloaded, extracted and put into the wheel automatically on wheel build. An extra (maybe shellout) bdist-time dependency to 7z seems fine to me. Pulling the libusb.info page and locating the windows download url is fine too. Having an expected hash (and hence wheel build failing on my side when a new version is out, so I can check it a bit) is also fine.

Yup, sounds like what I'd do.

I do not have enough experience with python wheels, so every bit helps.

I use a 2 step build process:

1. Obtain the libusb binaries and place them into the source tree as package data. By "package data" I mean they should be declared in setup(), something similar to package_data={"libusb1": ["*.dll"]}.
2. Run python setup.py bdist_wheel --plat-name <plat> where <plat> is (for Windows) one of win32 and win_amd64. You are aiming to produce a compatibility tag of py3-none-win32 or py3-none-win_amd64. Such a tag is slightly unusual in that it is platform-specific but not ABI-specific (I believe distutils doesn't normally produce tags like that, but they are certainly supported by PyPI/pip and used in the wild).

[vpelletier]

After reading a bit more about the "DLL hell" and what is done in other bdist wheels, ideally the DLL should have a unique name to force windows to actually load the DLL even if the same process already loaded an homonym. I guess this is not really mandatory, as it should be unusual to mix multiple libusb dll in the same process, but this is something which should ideally be taken care of.

I'm also wondering how I should package 32 and 64bits versions: build-time copy of the right version to a common name, or run-time detection and loading of the correct version ?

So fer I have not found something which would handle these for me, but please let me know if this is just my google-fu being weak - I've already doubled my setup.py size and it is not working yet.

[whitequark]

ideally the DLL should have a unique name to force windows to actually load the DLL even if the same process already loaded an homonym

If I recall correctly, if you pass a full path to LoadLibrary (which you can do just fine with ctypes), Windows will actually load that specific DLL.

I'm also wondering how I should package 32 and 64bits versions: build-time copy of the right version to a common name

I'd say two different wheels, though one could argue for runtime detection too.

[whitequark]

So fer I have not found something which would handle these for me, but please let me know if this is just my google-fu being weak - I've already doubled my setup.py size and it is not working yet.

I think this would be fairly straightforward to me--once I find time I'll try to get something working. But that might take a while, unfortunately.

[whitequark]

Quick update: I expect to be able to implement this soon.

[whitequark]

As promised, #61. I verified that these work with Glasgow on Windows with INF-less WinUSB, with both 32-bit and 64-bit Python.

[vpelletier]

Commits applied, and released as 1.8.1 .