New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hook-zmq no longer works for PyZMQ 15.4.0+ #2147

Closed
codewarrior0 opened this Issue Aug 16, 2016 · 13 comments

Comments

Projects
None yet
7 participants
@codewarrior0
Member

codewarrior0 commented Aug 16, 2016

Previously, zmq.__init__ would search its package path for the library libzmq.pyd using a glob-pattern and then load it using a call to ctypes.CDLL.LoadLibrary. This allowed us to freeze the file as zmq/libzmq.pyd and also exclude the file zmq.libzmq.pyd (for reasons I will get into).

As of a recent update, zmq.__init__ now simply imports the library as a Python module using from . import libzmq. When frozen, this import now silently fails, but leads to a different error. When PyZMQ later imports the extension modules under zmq.backends, it yields an error "ImportError: DLL load failed: The specified module could not be find". Since the backend modules are actually present, this indicates they failed to load because one of their DLL dependencies was not found - this dependency being libzmq.pyd.

libzmq.pyd is not on the DLL search path. In the previous PyZMQ version, loading the DLL via LoadLibrary implicitly puts it on the search path as an already-loaded-DLL, so when the backends are imported, they can always find it. In the current version, it will only attempt to load this DLL through a Python import statement, which will cause our CExtensionImporter to try to load zmq.libzmq.pyd, and fail because it was excluded during the build process.

If zmq.libzmq.pyd is not excluded, we still get the same "ImportError: DLL load failed" while importing the backend modules: they link to the library by name, which is libzmq.pyd. Our CExtensionImporter will not import the module from zmq/libzmq.pyd.

As mentioned in another issue (#2005 (comment)), I discovered that adding our importer to sys.path_hooks prevents the Python import system from falling back to the filesystem for imports: After the sys.meta_path importers fail to import zmq.libzmq, Python finds a cached path_hooks importer for the module zmq and stops there.

The resolution to this might be to change the importer added to sys.path_hooks to also search the filesystem if it fails to find the module in the archive.

This issue may be specific to Windows and/or to PyZMQ installations that bundle libzmq (as opposed to allowing the system's package manager to install it).

@bjones1

This comment has been minimized.

Member

bjones1 commented Aug 17, 2016

A more radical proposal: What if we did away with the CExtensionImporter, placing all extensions in a directory rather than a dotted filename (i.e. zmq/libzmq.pyd instead of zmq.libzmq.pyd? This should make PyInstaller behave a bit more like Python, and hopefully avoid some of these corner cases in the future. The downside I see: this will produce a lot of subdirectories, cluttering the filesystem a bit. A fair trade, IMHO.

@bjones1

This comment has been minimized.

Member

bjones1 commented Aug 19, 2016

I'm seeing Appveyor failures of the zmq tests only on Python 2.7. Is this the same issue, or is something else going on?

bjones1 added a commit to bjones1/pyinstaller that referenced this issue Aug 19, 2016

bjones1 added a commit to bjones1/pyinstaller that referenced this issue Aug 19, 2016

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Aug 20, 2016

Is this the same issue?

Yep. The test failure looks exactly like the error I got locally.

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Aug 20, 2016

What if we did away with the CExtensionImporter, placing all extensions in a directory rather than a dotted filename (i.e. zmq/libzmq.dll instead of zmq.libzmq.dll?

Hmm. We will almost certainly run afoul of the sys.path_hooks behavior that prevents the filesystem from being searched if a path_hooks-returned importer claims that pathname. But that's already fouling us in a couple other ways.

Other than that, I get the feeling that CExtensionImporter (and the dotted filename pattern) is a legacy of the iu.py import hooking mechanism from the earliest versions of PyInstaller. I'd have to double check to be sure.

bjones1 added a commit that referenced this issue Aug 20, 2016

@TTimo

This comment has been minimized.

TTimo commented Jan 14, 2017

What is a good workaround for this? I think I'm running into the same issue.
I tried to go back as far as I could in old versions via PyPI and they all fail.

@TTimo

This comment has been minimized.

TTimo commented Jan 18, 2017

I decided to get latest from git and see if I could make some progress on this. It seems latest has a new problem in collect_submodules which wasn't happening in 3.2:

5463 DEBUG: Collecting submodules for zmq.backend
6103 DEBUG: collect_submodules - Found submodules: set(['zmq.backend.cffi', 'No', 'file:', 'zmq.backend.cython.socket', 'file', 'zmq.backend', 'zmq.backend.cython.message', 'open', 'zmq.backend.cython.constants', 'zmq.backend.cython.error', ':', 'zmq.backend.cython', 'zmq.backend.cython._poll', 'Cannot', 'zmq.backend.cython._device', 'fatal', 'C:\\Python27\\lib\\site-packages\\zmq\\backend\\cffi\\__pycache__\\_cffi_ext.c(209)', 'C1083:', 'such', 'zmq.backend.cython.utils', 'zmq.backend.cython._version', 'include', "'sys/un.h':", 'zmq.backend.cython.context', 'zmq.backend.select', 'error', 'directory', '_cffi_ext.c', 'or'])

The full log for 3.3-dev is at: https://gist.github.com/TTimo/1614752c59579e099e4f4106d85c14fd

For reference the log for 3.2 is here: https://gist.github.com/TTimo/d91d43c4c73d6269021f52db376d3c40

@andrewjaykeller

This comment has been minimized.

andrewjaykeller commented Jan 22, 2017

I'm running into the same issue

Edit: Back dated to 15.3.0 and it worked +1 for the well named issue

@TTimo

This comment has been minimized.

TTimo commented Jan 26, 2017

I tried backing to older revisions but could never get it working. I will try again.

Unfortunately I have not been able to get any feedback from core developers on this. My posts to pyinstaller google group are not going through, and the irc channel is utterly dead.

I'm posting a bounty through BountySource to encourage a fix though, it's not much but maybe that will encourage some activity. I would really like to see zmq as an officially supported package in pyinstaller.

https://www.bountysource.com/issues/37007741-hook-zmq-no-longer-works-for-pyzmq-15-4-0

@ghost

This comment has been minimized.

ghost commented Jan 27, 2017

@TTimo If you're willing to live with a bit larger executable, subzero can package this correctly if you disable optimize_imports. You have to forgive the core developers because they are very busy people.

@albertogomcas

This comment has been minimized.

albertogomcas commented Jul 28, 2017

I saw a similar issue mentioned in zeromq/pyzmq#999 but the suggested "fix" did not work for me. (Python3.6, pyzmq 16.0.2)

I would love any hints about how to workaround this issue!

@carlosperate

This comment has been minimized.

carlosperate commented Sep 24, 2017

I had the same problem with pyzmq 16.0.2 on Windows, I had to revert back to version 15.3.0 as suggested by @aj-ptw .

bjones1 added a commit to bjones1/pyinstaller that referenced this issue May 25, 2018

bjones1 added a commit to bjones1/pyinstaller that referenced this issue May 26, 2018

@bjones1 bjones1 closed this in 2b5a3b2 May 26, 2018

@TTimo

This comment has been minimized.

TTimo commented May 27, 2018

tested successfully - thanks @bjones1 !
please feel free to claim the bounty

@bjones1

This comment has been minimized.

Member

bjones1 commented May 31, 2018

Thanks! Will do. I've also submitted #3544, which makes the hooks easier to maintain in the future.

@htgoebel htgoebel added this to the PyInstaller 3.4 milestone Sep 2, 2018

cowo78 pushed a commit to cowo78/pyinstaller that referenced this issue Dec 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment