modulegraph seams to not handle extended __path__ set in a hook (hook-win32com)

[Torxed]

http://stackoverflow.com/questions/31203580/pyinstaller-not-reading-my-hook-file-and-doesnt-work-with-win32com-shell

In short, doing from win32com.shell import shell doesn't get shell included, it also doesn't help to create a hook-win32com.shell.py file. That never gets read. At first I thought it might get picked up by win32com.py which is the main module hook/file.. But modifying that doesn't help the end result.

Here's the log:

C:\Users\x\Desktop>pyinstaller filetegrity.py
78 INFO: wrote C:\Users\x\Desktop\filetegrity.spec
79 INFO: Testing for ability to set icons, version resources...
105 INFO: ... resource update available
106 INFO: UPX is not available.
109 INFO: Extending PYTHONPATH with C:\Users\x\Desktop
109 INFO: checking Analysis
109 INFO: Building because filetegrity.py changed
109 INFO: Creating base_library.zip for Python 3
5815 INFO: running Analysis out00-Analysis.toc
5978 INFO: Analyzing filetegrity.py
10951 INFO: Analyzing C:\Python34\lib\site-packages\pyinstaller-3.0.dev0-py3.4.egg\PyInstaller\loader\_pyi_bootstrap.py
11096 INFO: Analyzing C:\Python34\lib\site-packages\pyinstaller-3.0.dev0-py3.4.egg\PyInstaller\loader\pyi_importers.py
11115 INFO: Analyzing C:\Python34\lib\site-packages\pyinstaller-3.0.dev0-py3.4.egg\PyInstaller\loader\pyi_archive.py
11138 INFO: Analyzing C:\Python34\lib\site-packages\pyinstaller-3.0.dev0-py3.4.egg\PyInstaller\loader\pyi_carchive.py
11161 INFO: Analyzing C:\Python34\lib\site-packages\pyinstaller-3.0.dev0-py3.4.egg\PyInstaller\loader\pyi_os_path.py
11165 INFO: Hidden import 'codecs' has been found otherwise
11165 INFO: Hidden import 'encodings' has been found otherwise
11165 INFO: Looking for import hooks ...
11180 INFO: Processing hook   hook-encodings.py
11784 INFO: Processing hook   hook-pydoc.py
11786 INFO: Processing hook   hook-distutils.py
11789 INFO: Processing hook   hook-xml.py
12197 INFO: Processing hook   hook-xml.sax.py
12197 INFO: Processing hook   hook-site.py
12200 INFO: Processing hook   hook-sysconfig.py
12201 INFO: Processing hook   hook-win32com.py
13006 INFO: Processing hook   hook-win32com.client.py
13007 INFO: Processing hook   hook-win32ui.py
13010 WARNING: Hidden import 'cStringIO' not found (probably old hook)
13011 INFO: Processing hook   hook-pywintypes.py
13126 INFO: Processing hook   hook-pythoncom.py
13263 INFO: Analyzing run-time hooks ...
13266 INFO: Including run-time hook 'pyi_rth_win32comgenpy.py'
13273 INFO: Looking for dynamic libraries
14207 INFO: Looking for eggs - TODO
14228 INFO: Using Python library C:\windows\system32\python34.dll
14272 INFO: Warnings written to C:\Users\x\Desktop\build\filetegrity\warnfiletegrity.txt
14273 INFO: checking PYZ
14277 INFO: Building because toc changed
14277 INFO: Building PYZ (ZlibArchive) out00-PYZ.toc
14845 INFO: checking PKG
14846 INFO: Building because C:\Users\x\Desktop\build\filetegrity\out00-PYZ.pyz changed
14846 INFO: Building PKG (CArchive) out00-PKG.pkg
14878 INFO: checking EXE
14882 INFO: Rebuilding out00-EXE.toc because pkg is more recent
14882 INFO: Building EXE from out00-EXE.toc
14885 INFO: Bootloader C:\Python34\lib\site-packages\pyinstaller-3.0.dev0-py3.4.egg\PyInstaller\bootloader\Windows-32bit\run
14888 INFO: Appending archive to EXE C:\Users\x\Desktop\build\filetegrity\filetegrity.exe
14898 INFO: checking COLLECT
WARNING: The output directory "C:\Users\x\Desktop\dist\filetegrity" and ALL ITS CONTENTS will be REMOVED! Continue? (y/n)y
26914 INFO: Removing dir C:\Users\x\Desktop\dist\filetegrity
26925 INFO: Building COLLECT out00-COLLECT.toc

warnfiletegrity.txt

...
no module named pythoncom.DISPID_COLLECT
...
no module named win32com.shell - imported by C:\Users\x\Desktop\filetegrity.py

Problem:

C:\Users\x\Desktop>dist\filetegrity\filetegrity.exe
Traceback (most recent call last):
  File "<string>", line 3, in <module>
ImportError: No module named 'win32com.shell'

[Torxed]

Also followed all the instructions and parameters under: https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged

[bjones1]

Anton,

Perhaps:

import pythoncom
import win32com.shell # _New line_*
from win32com.shell import shell
from win32com import storagecon

might work? Doesn't makes sense, but does help in one of my hooks.

[htgoebel]

Seams as if the module is actually called win32comext.shell.shell:

$wine python
Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from win32com.shell import shell
>>> shell
<module 'win32com.shell.shell' from 'C:\Python27\lib\site-packages\win32comext\shell\shell.pyd'>
>>> 

[htgoebel]

Looks as if win32com/__init__.py does some path-hacking (in win32com/__init__.py)

>>> win32com.__path__
['win32com', 'C:\\Python27\\Lib\\site-packages\\win32comext']

So this comes down do be yet another way for namespace-packages :-(

[Update:]
For both develop- and python3 branch, hook-win32com.py includes this path-extension. On the develop-branch everything works fine. So this is a problem related to modulegraph.

[Torxed]

I'll verify that this statement is correct on my laptop in two days time, would be awesome if the dev branch solved this (I just used the prepackaged installer for a quick fix on a project) :)

[Torxed]

@htgoebel not sure how you got it to run or how your hook-win32com.py looks like because I still get:
(Tried cloning the Python3 branch here on github)

C:\Users\x\Desktop>dist\filetegrity\filetegrity.exe
Traceback (most recent call last):
  File "<string>", line 3, in <module>
ImportError: No module named 'win32com.shell'

[htgoebel]

@Torxed I only tested it using Python2.7 and the develop branch. You are using Python 3.x and the python3 branch, which uses a very different mechanism for finding dependencies (called modulegraph), which seams cause the bug.

[htgoebel]

We need to handle this in PyiModuleGraph.import_hook resp. we need to make our hooks work on modulegraph.modulegraph.Nodes. Maybe some wrapper using properties to map teh the Node's attributes.

[htgoebel]

I had a look at this. As far as I understand, modulegraph first collects all imports recursively and then PyInstaller applies the hooks. This is too late for any hook changing the path or retrageting :-(

Related: #1367, #1393

[htgoebel]

This has basically been solved by implementing post-create package hooks. Since as of today we have no failing test for this and esp. win32com is working, I'm closing this issue. I've opened #1532 as a follow up for checking all hooks if they need to be converted.

[r0x0r]

What is a solution to this one? I am using version 3.0 and still getting this issue with win32com.shell.

[codewarrior0]

The problem is that the folder win32comext is added to the __path__ of package win32com, but ModuleGraph can't follow this path addition because it is done at runtime. I can't think of a quick-fix that you can use right now.

[codewarrior0]

win32comext modules are not covered by our test suite. I'm introducing some new functional tests for these modules.

[r0x0r]

This is how py2exe handles this module if this is of any help
http://www.py2exe.org/index.cgi/win32com.shell

[matysek]

@htgoebel Does this look like a new candidate for pre_find_module_path hook?
@codewarrior0 Are you going to work on this issue?

[htgoebel]

This should become a pre_save_import hook, as we need to extend the path after the package has been found, but before processing it's imports.

[matysek]

Ok, I'm going to try that.

[matysek]

Closing. This should be fixed.
@Torxed or @r0x0r could you verify that it works for you?

[Torxed]

@matysek Sorry for getting back on this so late.
I have confirmed that the master branch here from git now works, and develop branch also works but as it's unstable at times I haven't tried it out that much.

But all appears to be well. Thank you for looking in to this!

Best regards,
Anton