New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't package pyusb, 'Unable to find "libusb-1.0.so.0"' #2633

Closed
joelhoisko opened this Issue May 29, 2017 · 15 comments

Comments

Projects
None yet
6 participants
@joelhoisko

joelhoisko commented May 29, 2017

To reproduce

  1. virtualenv env
    source env/bin/activate
    pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
    pip install pyusb
    pip freeze > requirements.txt
  2. Create simple python file which imports pyusb
    test.py
import usb.core
print 'Success!
  1. Run the python file normally
$ python test.py
Success!
  1. Package it with pyinstaller
pyinstaller --log-level=DEBUG test.py &> log.txt

log.txt

14 INFO: PyInstaller: 3.3.dev0+5fd013bfe
14 INFO: Python: 2.7.12+
14 INFO: Platform: Linux-4.8.0-53-generic-x86_64-with-Ubuntu-16.10-yakkety
14 INFO: wrote /home/joel/Code/random/pyinstaller/test.spec
15 DEBUG: Testing for UPX ...
17 INFO: UPX is not available.
17 DEBUG: script: /home/joel/Code/random/pyinstaller/test.py
17 INFO: Extending PYTHONPATH with paths
['/home/joel/Code/random/pyinstaller', '/home/joel/Code/random/pyinstaller']
17 INFO: checking Analysis
17 INFO: Building Analysis because out00-Analysis.toc is non existent
17 INFO: Initializing module dependency graph...
18 INFO: Initializing module graph hooks...
19 DEBUG: Hidden import: codecs
65 DEBUG: Hidden import 'codecs' already found
65 INFO: running Analysis out00-Analysis.toc
65 DEBUG: Analyzing /home/joel/Code/random/pyinstaller/env/bin/python
71 DEBUG: Skipping libdl.so.2 dependency of python
71 DEBUG: Skipping libc.so.6 dependency of python
71 DEBUG: Skipping libutil.so.1 dependency of python
71 DEBUG: Skipping libpthread.so.0 dependency of python
71 DEBUG: Skipping libm.so.6 dependency of python
72 DEBUG: Adding libz.so.1 dependency of python from /lib/x86_64-linux-gnu/libz.so.1
72 DEBUG: Analyzing /lib/x86_64-linux-gnu/libz.so.1
78 DEBUG: Skipping libc.so.6 dependency of libz.so.1
78 INFO: Caching module hooks...
81 INFO: Analyzing /home/joel/Code/random/pyinstaller/test.py
1078 INFO: Loading module hooks...
1079 INFO: Loading module hook "hook-usb.py"...
Unable to find "libusb-1.0.so.0" when adding binary and data files.

This causes the packaging process to halt with an empty dist/ and build/test/ folders.

I dug around and found out, that hook-usb.py reports the binaries to be [('libusb-1.0.so.0', '')] and I think this causes that src_root_path_or_glob(utils.py, lines 450+) to just contain 'libusb1.0.so.0', which produces just []after it get's run through the glob.glob() and not the real path, which I think is/lib/x86_64-linux-gnu/libusb-1.0.so.0 on my computer.

Environment:

Python 2.7.12+
Ubuntu 16.10
Virtualenv 15.1.0

requirements.txt

appdirs==1.4.3
packaging==16.8
PyInstaller==3.3.dev0+5fd013bfe
pyparsing==2.2.0
pyusb==1.0.0
six==1.10.0

Also, this seems to be somewhat related to #1682 and commit 23901eb

@unwenliu

This comment has been minimized.

unwenliu commented Jul 17, 2017

I also have this problem

Can you tell me how to solve it?

@joelhoisko

This comment has been minimized.

joelhoisko commented Jul 18, 2017

I haven't tried to solve it/dig deeper into yet, as my project doesn't yet depend on pyusb.

@htgoebel htgoebel added the hooks label Jul 18, 2017

@htgoebel

This comment has been minimized.

Member

htgoebel commented Jul 18, 2017

Please help tracking this down:

  • Add import pdb; pdb.set_trace() near the top of hook-usb.py (just behind the import statements)
  • In the shell run export PYUSB_DEBUG=debug
  • run PyInstaller pyinstaller test.py (no output-redirection)
  • step through hook-usb.py to gain more insights.

Thanks.

@joelhoisko

This comment has been minimized.

joelhoisko commented Aug 7, 2017

Sorry for the delay, but I managed to find something to get me forward.

The custom mechanism of hook-usb.py as defined in here seems to find the right binaries for my system. The 'pyusb library locator' (lines 29-46) makes the binaries to be [('libusb-1.0.so.0', '')], meaning the branch if not binaries: on line 50 isn't executed even though we don't have a path to the binaries.

I changed line 50 to if binaries: for testing purposes and $ pyinstaller test.py runs fine and the usb module works as intended. I also tested it on a small script (writes to an ESC/POS printer with pyusb) and it worked fine.

So yeah, I don't know the underlying issue of why the usblib._name doesn't resolve into the full path (I guess it should?) but this 'hack' circumvents the problem.

themadinventor added a commit to themadinventor/pyinstaller that referenced this issue Sep 15, 2017

jxltom added a commit to jxltom/pyinstaller that referenced this issue Dec 15, 2017

Merge pull request #1 from themadinventor/bugfix/usb-lib-path
Fix pyinstaller#2633: Resolve library name reported by usb.backend in hook-usb
@joelhoisko

This comment has been minimized.

joelhoisko commented Dec 18, 2017

The fix by themadinventor (5de8a59) does indeed seem to work for me.

jxltom added a commit to jxltom/pyinstaller that referenced this issue Jan 15, 2018

@jxltom

This comment has been minimized.

jxltom commented Jan 16, 2018

In Linux 5de8a59 works fine. But in Windows 7 Enterprise, assert len(binaries[0]) == 3 fails when using Conda's python35 env with libusb...

@htgoebel

This comment has been minimized.

Member

htgoebel commented Jan 16, 2018

@jxltom Please open an issue providing all information required for analysis, esp. the traceback.

@charlesnicholson

This comment has been minimized.

Contributor

charlesnicholson commented Jan 31, 2018

We hit this issue as well. Here's a repository you can clone to see the failure:

Python 3.6, Ubuntu 16.04:
https://github.com/charlesnicholson/pyinstaller-linux-pyusb-bug-repro

As reported above, the discovered backend isn't anchored in any path, and _resolveCtypesImports is never called on it to compute the absolute path.

Note also that on Ubuntu 14.04 we see PyInstaller crashes coming from usb-hook.py:

usb-hook.py first attempts to get the backend by importing and using usb.backend:
https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/hook-usb.py#L31

usb gets unloaded when the module is finalized, and libusb_exit causes a segmentation fault inside of PyInstaller, which then crashes.

If detecting the usb shared library by way of the pyusb backend is required, it would be nice to be able to configure the hook to not attempt this method, and instead use the ctypes.util.find_library approach.

@charlesnicholson

This comment has been minimized.

Contributor

charlesnicholson commented Jan 31, 2018

BTW I'm happy to commit time to writing a patch if there's a maintainer-approved solution.

@charlesnicholson

This comment has been minimized.

Contributor

charlesnicholson commented Feb 1, 2018

I've added some print instrumentation to my repro repository that shows the exact problem. Check out the logs at the bottom of the README.md at
https://github.com/charlesnicholson/pyinstaller-linux-pyusb-bug-repro

The PyInstaller fork that I've instrumented is here:
https://github.com/charlesnicholson/pyinstaller/tree/usb-hook-linux-repro

The instrumented hook file is here:
charlesnicholson@f4297eb

I'm curious what the motivation is for the first half of the file that engages pyusb? It seems like if your libusb is somewhere that only pyusb can find and not in your LD_LIBRARY_PATH etc, maybe that's worth a custom entry in your spec file?

It seems like the entire hook might be simpler and just as full-featured if the bottom half of the file that uses ctypes.util.find_library was the only approach used?

In my hacked-up fork that unblocked my project, I simply wrapped the top half of the file in "if we're not on Linux" logic and it seems to work fine. That's probably not a great long-term solution though :)

@htgoebel

This comment has been minimized.

Member

htgoebel commented Feb 1, 2018

I'm happy to commit time to writing a patch if there's a maintainer-approved solution.

Great! Please provide a pull-request, so we can easily see, review and comment your changes. Thanks.

@charlesnicholson

This comment has been minimized.

Contributor

charlesnicholson commented Feb 1, 2018

Sure, happy to. My only initial question is that previously in this thread, this solution was posted:
jxltom@8789584

Which (correctly) calls _resolveCtypesImports on the basename of the discovered library.

I think that's the correct fix for the case where you /want/ to use the pyusb backend.

My open question is, what's the advantage of engaging the pyusb backend? It seems like it should always give you the same results that ctypes.util.find_library does, but is more heavyweight in that it actively loads various libusb implementations.

I'm happy to open a new PR that simply deletes the top half of the file, but I think that might be bulldozing past some use cases.

I think the least intrusive PR would be a combination of what's listed above, plus something like a check for an environment variable like PYINSTALLER_HOOK_USB_DONT_SEARCH_WITH_PYUSB so that the hook's default behavior doesn't change, but can be overridden.

Is this a reasonable approach?

charlesnicholson added a commit to charlesnicholson/pyinstaller that referenced this issue Feb 3, 2018

USB hook updates: Special-case Cygwin per bug pyinstaller#2633. Colle…
…ct basename library names from pyusb and normalize with _resolveCtypesImports. Unify code paths for library validation and normalization, however they were found. Fix bug pyinstaller#2633 for Linux. Delete unnecessary list re-initialization, delete redundant comments, little cleanups, etc
@charlesnicholson

This comment has been minimized.

Contributor

charlesnicholson commented Feb 3, 2018

Opened #3269 with fixes, feedback welcome!

charlesnicholson added a commit to charlesnicholson/pyinstaller that referenced this issue Feb 7, 2018

USB hook updates: Special-case Cygwin per bug pyinstaller#2633. Colle…
…ct basename library names from pyusb and normalize with _resolveCtypesImports. Unify code paths for library validation and normalization, however they were found. Fix bug pyinstaller#2633 for Linux. Delete unnecessary list re-initialization, delete redundant comments, little cleanups, etc
@mofahead

This comment has been minimized.

mofahead commented Feb 9, 2018

I just got hit with this problem too. Awesome that it's being worked on!

@htgoebel htgoebel closed this in 4218c19 Feb 25, 2018

@charlesnicholson

This comment has been minimized.

Contributor

charlesnicholson commented Feb 25, 2018

@htgoebel It's great that the build system is better and that the first fix went in!

Please also take a look at my work in #3269. It cleans up duplicated logic, simplifies the usb tests, and allows for the ubuntu 14.04 usb hook crash to be avoided by an environment variable.

It's a more thorough set of fixes and improvements to the usb hook module than what you just merged.

@htgoebel htgoebel added this to the PyInstaller 3.4 milestone Sep 2, 2018

cowo78 pushed a commit to cowo78/pyinstaller that referenced this issue Dec 7, 2018

Hooks: hook-usb: Resolve library name reported by usb.backend.
hook-usb.py "asks" pyusb which library to pack by inspecting
usb.backend. This only results in a library name (e.g libusb-1.0.so.0),
which is sufficient for ctypes to load the library, but not enough for
pyinstaller to find the file and pack it. This commit solves this problem
by using _resolveCtypesImports to figure out the full path to the file
pyusb loads.

Fixes pyinstaller#2633
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment