Windows: dependencies not filled in shared library manifests

[RussellTaylor]

We've been trying to add the ability to launch an IPython qt console from our C++ application. On Windows, things are failing because the import of (py)zmq fails (i.e. trying an "import zmq" in our python terminal fails). It turns out that this is because initthreads.pyd depends on the Visual Studio 2008 runtime (MSVCR90.DLL) but it does not know where to find it because the embedded manifest is lacking that information. So whereas in libzmq.dll the manifest contains the section:

<dependency>
    <dependentAssembly>
      <assemblyIdentity type="win32" name="Microsoft.VC90.CRT" version="9.0.21022.8" processorArchitecture="amd64" publicKeyToken="1fc8b3b9a1e18e3b"></assemblyIdentity>
    </dependentAssembly>
</dependency>

the corresponding section in initthreads.pyd is empty (as it is for the other pyzmq libraries - this is just the first to get loaded).

Would it be possible to fix the install kits to point to the runtime correctly?
I assume that the import works on running a regular python terminal because it already has the runtime loaded - our application is built with VS2010, but we still use the stock (VS2008) python underneath.

[minrk]

I really don't know much about building and linking on Windows. All I do is run setup.py bdist_egg [or bdist_msi], which seems to work on every machine I've tried, but I don't know what the variables are. If there are further arguments that should be passed to distutils, I would be happy to know them.

Are you using one of the bdists, or building pyzmq yourself?

[minrk]

Also, I seem recall something about 'debug' builds of libzmq needing the CRT, and 'release' builds not needing it, so maybe that's the issue. The bdists should all be using 'release' builds, but it's possible some of them slipped up.

[RussellTaylor]

Unfortunately, I'm not an expert on the Windows manifest, and even less so on distutils. I've been using the bdists and see this on all varieties (easy_install, msi, 32 & 64 bit). I don't see any evidence of debug library dependencies.

I managed to get things to work by deleting the embedded manifest from all the .pyd files (this can easily be done by opening each library in Visual Studio if you rename the file to a .dll extension first). [I now have the same problem with readline, but at least that's optional so I can push on for a while without it.]

I think that the manifest can be set up correctly by passing some kind of flag (e.g. /MANIFESTDEPENDENCY) to the linker via msvc9compiler, but I'm not sure exactly what, nor whether the correct thing to do is to have the runtime dependency fully specified or not to have a manifest at all. If I can figure it out I'll let you know!

A couple of links that talk about this issue:
http://kb.froglogic.com/display/KB/Errors+with+%28third+party%29+Python+modules
http://bugs.python.org/issue4120 (and other python issues such as 4566 & 7833)

[minrk]

Okay, thanks for the information. It seems like one link is arguing that there should either be a CRT specified or no manifest, and the other argues that it should have a manifest, but not specify crt (the current situation). If there's an easy distutils flag or something that prevents there being any embedded manifest, that seems most sensible based on my ill-informed reading.

I don't understand why it would do a better job having no manifest than having a manifest with no information in it, but that's good to know.

I think I'm going to go with a 'pull requests welcome' on this one, because I don't think I have any relevant knowledge or experience for figuring this one out.

[arkottke]

This is also an issue when trying to load pyzmq into a vim session. The vim command :python import zmq fails because MSVCR90.DLL cannot be located when initthreads.pyd is referenced. However, within a regular python session import zmq works just fine.

[minrk]

I don't know if you guys will see this, but does pyzmq with libzmq built as an extension have this issue in vim?

[ascotan]

This was also an issue with pyodbc. See here: http://code.google.com/p/pyodbc/issues/detail?id=214. This issue prevents pyzmq from being used with python 2.7 for embedded use or with WSGI through a webserver. Currently you get WindowsError: [Error 126] The specified module could not be found until you fix the manifests in the .pyd libraries.

[tmr232]

Just spent the best part of 2 days trying to work this out. Here are my results.
First, it seems like distutils intentionally prevents the existence of the MSVCR90.DLL reference in extensions:

def _remove_visual_c_ref(self, manifest_file):
        try:
            # Remove references to the Visual C runtime, so they will
            # fall through to the Visual C dependency of Python.exe.
            # This way, when installed for a restricted user (e.g.
            # runtimes are not in WinSxS folder, but in Python's own
            # folder), the runtimes do not need to be in every folder
            # with .pyd's.
            # Returns either the filename of the modified manifest or
            # None if no manifest should be embedded.
            # Rest of the function removed for brevity.

This is an excerpt from mscv9compiler.py in distutils. As you can see, the manifest is removed for a reason, so I would not suggest going around it in pyzmqs buildscript, as it would cause issues for restricted users.

I've managed to get the following solutions to work for me:

1. Manually add the manifests to the *.pyd files after installtion. This can be done automatically using mt.exe and a batch file (I've done this using the CLI, minor changed might be required for running from a file):

   :: Extract the manifest from python.exe
   mt.exe -inputmanifest:C:\python27\python.exe;#1 -out:C:\python27\python.manifest
   
   :: Loop over all *pyd files and add the manifest
   FOR /R C:\python27\Lib\site-packages\zmq %i IN (*.pyd) DO (
   mt.exe -manifest C:\python27\python.manifest -outputresource:%i;#2
   )

2. Add the manifest to your own application. This seems like the obvious choice, but did not always work for me.

3. (bad idea) modifying distutils to do my bidding. That way you can either add a manifest or make it build against your current runtime (worked out of the box for me, tested against IPython).

I hope this helps.

[Usonaki]

This issue has already been discussed here #283. Here are the fixed MSIs https://www.dropbox.com/sh/8wpsm3h7y49y1pd/lHggJ5pQhY. These versions of pyzmq might not properly work with newer IPython versions. What I did is, I overwrote the files from pyzmq\zmq\eventloop\ with the ones from a later version.

[minrk]

Closing stale issues: I think this shouldn't be an issue anymore with the new Windows builds, but new issues can be opened if it comes up.