importing tifffile causes RecursionError

[radjkarl]

Pyinstaller 3.3 and 3.4currentdev
Anaconda 4.3.27
Python 3.6
tifffile 0.12.1

minimalTest.py
from tifffile import TiffFile

Make exe:
pyinstaller minimalTest.py

Workaround
In tifffile.py:
Replace

try:
    if __package__:
        from . import _tifffile  # noqa
    else:
        import _tifffile  # noqa
except ImportError:
    warnings.warn(
        "ImportError: No module named '_tifffile'. "
        "Loading of some compressed images will be very slow. "
        "Tifffile.c can be obtained at http://www.lfd.uci.edu/~gohlke/")

with
from . import _tifffile

Error message:
231 INFO: PyInstaller: 3.4.dev0+133d18156
231 INFO: Python: 3.6.1
232 INFO: Platform: Windows-10-10.0.10240-SP0
237 INFO: wrote D:\Programming\eclipse_workspace\scripts\AAA\test.spec
10504 INFO: UPX is not available.
10508 INFO: Extending PYTHONPATH with paths
['D:\Programming\eclipse_workspace\scripts',
'D:\Programming\eclipse_workspace\scripts\AAA']
10508 INFO: checking Analysis
10508 INFO: Building Analysis because out00-Analysis.toc is non existent
10508 INFO: Initializing module dependency graph...
10514 INFO: Initializing module graph hooks...
10521 INFO: Analyzing base_library.zip ...
21421 INFO: running Analysis out00-Analysis.toc
21426 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
required by C:\ProgramData\Anaconda3\python.exe
22475 INFO: Caching module hooks...
22482 INFO: Analyzing D:\Programming\eclipse_workspace\scripts\AAA\test.py
27173 INFO: Processing pre-find module path hook distutils
33490 INFO: Processing pre-find module path hook site
33493 INFO: site: retargeting to fake-dir 'C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\fake-modules'
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\Scripts\pyinstaller-script.py", line 11, in
load_entry_point('PyInstaller==3.4.dev0+133d18156', 'console_scripts', 'pyinstaller')()
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller_main_.py", line 92, in run
run_build(pyi_config, spec_file, **vars(args))
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller_main_.py", line 46, in run_build
PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\building\build_main.py", line 791, in main
build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\building\build_main.py", line 737, in build
exec(text, spec_namespace)
File "", line 16, in
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\building\build_main.py", line 213, in init
self.postinit()
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\building\datastruct.py", line 161, in postinit
self.assemble()
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\building\build_main.py", line 415, in assemble
priority_scripts.append(self.graph.run_script(script))
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\depend\analysis.py", line 201, in run_script
self._top_script_node = super(PyiModuleGraph, self).run_script(pathname)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 1343, in run_script
self._scan_code(m, co, co_ast)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2528, in _scan_code
self._process_imports(module)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2727, in _process_imports
target_module = self._safe_import_hook(*import_info, **kwargs)[0]
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2202, in _safe_import_hook
target_attr_names=None, level=level, edge_attr=edge_attr)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 1416, in import_hook
source_package, target_module_partname, level)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 1568, in _find_head_package
target_module_headname, target_package_name, source_package)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\depend\analysis.py", line 271, in _safe_import_module
module_basename, module_name, parent_package)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 1985, in _safe_import_module
module_name, file_handle, pathname, metadata)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2023, in _load_module
m = self._load_package(fqname, pathname, packagepath)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2797, in _load_package
self._load_module(fqname, fp, buf, stuff)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2077, in _load_module
self._scan_code(m, co, co_ast)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2528, in _scan_code
self._process_imports(module)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2727, in _process_imports
target_module = self._safe_import_hook(*import_info, **kwargs)[0]
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2202, in _safe_import_hook
target_attr_names=None, level=level, edge_attr=edge_attr)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 1416, in import_hook
source_package, target_module_partname, level)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 1568, in _find_head_package
target_module_headname, target_package_name, source_package)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\depend\analysis.py", line 271, in _safe_import_module
module_basename, module_name, parent_package)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 1985, in _safe_import_module
module_name, file_handle, pathname, metadata)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2077, in _load_module
self._scan_code(m, co, co_ast)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2528, in _scan_code
self._process_imports(module)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2727, in _process_imports
target_module = self._safe_import_hook(*import_info, **kwargs)[0]
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2312, in _safe_import_hook
target_attr_names=None, level=level, edge_attr=edge_attr)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2312, in _safe_import_hook
target_attr_names=None, level=level, edge_attr=edge_attr)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2312, in _safe_import_hook
target_attr_names=None, level=level, edge_attr=edge_attr)
[Previous line repeated 943 more times]
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2202, in _safe_import_hook
target_attr_names=None, level=level, edge_attr=edge_attr)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 1416, in import_hook
source_package, target_module_partname, level)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 1568, in _find_head_package
target_module_headname, target_package_name, source_package)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\depend\analysis.py", line 271, in _safe_import_module
module_basename, module_name, parent_package)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 1979, in _safe_import_module
module_partname, search_dirs, parent_module)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2853, in _find_module
return self._find_module_path(fullname, name, path)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\depend\analysis.py", line 315, in _find_module_path
fullname, module_name, search_dirs)
File "C:\ProgramData\Anaconda3\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 2934, in _find_module_path
module_name)
File "", line 1223, in find_loader
File "", line 1268, in find_spec
File "", line 60, in _path_join
RecursionError: maximum recursion depth exceeded

[dlejarazu]

Same problem here.

Gets stuck in an infinite recursive loop involving is_swig_candidate() and _safe_import_hook() being called over and over again on _tifffile

in my case, this is using scikit-image which includes tifffile. This only occurs on scikit-image 0.13. Downgrading to 0.12.3 solves the problem

Linking to what @radjkarl mentioned as workaround, the import at 0.12.3 is:

try:
    from . import _tifffile
except ImportError:
    warnings.warn(
        "failed to import the optional _tifffile C extension module.\n"
        "Loading of some compressed images will be slow.\n"
        "Tifffile.c can be obtained at http://www.lfd.uci.edu/~gohlke/")

while in 0.13 is:

try:
    if __package__:
        from . import _tifffile
    else:
        import _tifffile
except ImportError:
    warnings.warn(
        "ImportError: No module named '_tifffile'. "
        "Loading of some compressed images will be very slow. "
        "Tifffile.c can be obtained at http://www.lfd.uci.edu/~gohlke/")

[dlejarazu]

I believe the pain point is here:

pyinstaller/PyInstaller/lib/modulegraph/modulegraph.py

Lines 2277 to 2288 in 133d181

	if target_modules is None:
	self.msg(2, "ImportError:", str(msg))
	# Add this module as a MissingModule node.
	target_module = self.createNode(
	MissingModule,
	_path_from_importerror(msg, target_module_partname))
	self._updateReference(
	source_module, target_module, edge_data=edge_attr)
	# Initialize this list to this node.
	target_modules = [target_module]

when tries to import _tifffile as a SWIG module and fails, ceates a MissingModule. then at

pyinstaller/PyInstaller/lib/modulegraph/modulegraph.py

Lines 2298 to 2315 in 133d181

	if isinstance(target_module, MissingModule) and is_swig_candidate():
	# if this possible swig C module was previously imported from
	# a python module other than its corresponding swig python
	# module, then it may have been considered a MissingModule.
	# Try to reimport it now. For details see pull-request #2578
	# and issue #1522.
	# Remove the MissingModule node from the graph so that we can
	# attempt a reimport and avoid collisions. This node should be
	# fine to remove because the proper module will be imported and
	# added to the graph in the next line (call to _safe_import_hook).
	self.removeNode(target_module)
	# Reimport the SWIG C module
	target_modules = self._safe_import_hook(
	target_module_partname, source_module,
	target_attr_names=None, level=level, edge_attr=edge_attr)
	# return the output regardless because it would just be
	# duplicating the processing below
	return target_modules

It checks if it is a potential SWIG module and if it is a MissingModule and calls _safe_hook_import again, going through the same process over and over again

[htgoebel]

Thanks for you analysis. To verify this, please revert commit 711e9e7 (which is the one introducing this behaviour).

[yspark61]

I have the following problems.

• importing tifffile causes RecursionError

Is there a solution?

The development environment is as follows.
Pyinstaller 3.3
Python 3.6
tifffile 0.12.1

[htgoebel]

See #2911 (comment). I still did not get an answer.

[htgoebel]

@radjkarl @dlejarazu I'm still waiting for an answer. Please revert commit 711e9e7 and report back whether this solves the issue. Otherwise I'm going to close the issue.

[radjkarl]

@htgoebel sorry . dont have time for checking our the code reverting etc.
My workaround resolved the issue.

[htgoebel]

@radjkarl :-(( This is a workaround and not a solution. Reverting the commit and testing is a matter of a few minutes and wold help narrowing the issue and fixing the bug.

Anyway I need more information:

• Which platform you are using?
• Does the module site-packages/tifffile/_tifffile.so (or respective on your platform) exist?

@dlejarazu @yspark61 This question also go to you. If you want this bug fixed, I need the questions answered.

[djhoese]

@htgoebel FYI I was able to reproduce this on OSX 10.12 with Anaconda Python 3.6 environment with pyinstaller installed from an older github commit. The .so does exist /Users/davidh/anaconda/envs/sift/lib/python3.6/site-packages/tifffile/_tifffile.cpython-36m-darwin.so.

Also if I had to guess the issue with my PR (referenced above) is due to a change in SWIG and the import if statement mentioned by the OP.

[djhoese]

@htgoebel I see two problems here that I'd like your opinion on:

1. My changes make it so any _x module imported but not existing will cause an infinite recursive loop. This has to be fixed.
2. I don't think pyinstaller properly handles package relative imports which are perfectly allowed in any modern implementation of python (from what I understand). https://www.python.org/dev/peps/pep-0366/

I have to look at it further, but it seems like when level == ABSOLUTE_IMPORT_LEVEL (python 3) that there has to be a check in the current package first. Here is a test I just ran:

mypkg/
    __init__.py
    mymod1.py
    mymod2.py

# mymod1.py
def test():
    print("Test mymod1")

# mymod2.py
from . import mymod1
print("mymod2")
mymod1.test()

Then run:

pyinstaller mypkg/mymod2.py

Running ./dist/mymod2/mymod2 gives me an import error on mymod1. Shouldn't this work @htgoebel?

[djhoese]

Actually, nevermind. This would work with python -m mypkg.mymod2 but not python mypkg/mymod2.py so maybe this is expected.

[dlejarazu]

sorry for the delay. The latest develop doesn't show this problem. I guess that is @davidh-ssec fix.

Thanks

[djhoese]

@dlejarazu Really? My PR hasn't been merged yet so this should still be a problem. Make sure you delete your build and dist directories before trying it.

[dlejarazu]

@davidh-ssec you are absolutely right. I was using the wrong test. I get the original error now.

@htgoebel reverting 711e9e7 fixes the issue and produces the executable

• Platform: Windows 10 64-bits, Anaconda 32 bits. python 3.5
• site-packages\tifffile_tifffile.cp35-win32.pyd exists

[htgoebel]

@dlejarazu Thanks. A solution is in the works.

[htgoebel]

This should be solved now. Please try the current development version. If the problem still persists, please drop a note and reopen the ticket. Thanks.