setuptools 45.0.0 may cause PyInstaller 3.3 packaged executable fail to launch

[frederickfung]

In my specific example where we use PyInstaller 3.3 to package our application into an executable for deployment, the resulting executable cannot be launched after setuptools was updated from 44.0.0 to 45.0.0. This happened right at 9:00PM Pacific Time yesterday.

The error signature is (when launching the executable generated by PyInstaller):

[1072] Failed to execute script pyi_rth_pkgres
Traceback (most recent call last):
  File "site-packages/PyInstaller/loader/rthooks/pyi_rth_pkgres.py", line 11, in <module>
  File "/<virtualenv-path>/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 631, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages/pkg_resources/__init__.py", line 86, in <module>
ModuleNotFoundError: No module named 'pkg_resources.py2_warn'

PyInstaller version is 3.3, setuptools is 45.0.0
Working just fine with setuptools 44.0.0

For certain limitations in our Python application, we are stuck with Pyinstaller 3.3, but we are definitely looking into using later versions of PyInstaller.

[yoctozepto]

And virtualenv is broken: pypa/virtualenv#1493

It seems the impact is wider than imagined.

[pganssle]

Hm, maybe I'm wrong here, but I'm not sure why this particular failure mode would happen. Presumably if setuptools==45.0.0 is installed, it should have a pkg_resources.py2_warn, and if it's not installed, there should be no references to that module in the code. Is there some sort of cross-compile happening? Can you provide a minimal working example?

Also, you mention trying different versions of setuptools, but have you tried the latest version of PyInstaller? Whether or not you can use it yourself, it would be good to know whether this will also break with the latest version of PyInstaller.

[tardyp]

FWIW, I got this error also on the buildbot CI.

it happen that I had pip pinned to < 19 because of pypa/pip#6163 (which has ben fixed since), but pyinstaller was latest

pinning setuptools < 45 fixes the issue as per buildbot/buildbot@1ab0916

unpinning everything keeps the error. buildbot/buildbot@340bcf3

[1337] Failed to execute script pyi_rth_pkgres
Traceback (most recent call last):
  File "PyInstaller/loader/rthooks/pyi_rth_pkgres.py", line 11, in <module>
  File "/home/buildbot/repo/.venv/lib/python3.5/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "pkg_resources/__init__.py", line 86, in <module>
ImportError: No module named 'pkg_resources.py2_warn'

[dploeger]

On another note, the Puppet package provider for pip packages also has problems with this warning. IMO it should have the option to disable the warning.

[Cory-Kramer]

We ran into the same issue this morning, using PyInstaller 3.6

[priitlatt]

Observed the same behaviour on Python 3.7.5 with PyInstaller 3.5 and PyInstaller 3.6. I was able to build again successfully after reverting to previous release of setuptools:

pip install --upgrade 'setuptools<45.0.0'

[christianbioinf]

I am having the same issue with Python 3.6.8 and Pyinstaller 3.6. I suspect that it is caused by the recent commit 79f1694 in setuptools since it added the line __import__('pkg_resources.py2_warn') but not a regular import line like import pkg_resources.py2_warn (as done for other __import__ statements in pkg_resources.__init__.py) and afaik Pyinstaller does not scan for __import__ statements and hence does not freeze the file pkg_resources.py2_warn.py.

[gsemet]

Fixed by forcing in a hook file the following submodule collection:

...
hiddenimports.extend(collect_submodules('pkg_resources'))

[ifalex]

Shall we get a new update today or what is the plan?

[wedesoft]

Fixed by adding a hidden import:

a = Analysis(...,
             hiddenimports=['pkg_resources.py2_warn'],
             ...)

[jaraco]

It looks to me like Pyinstaller has a fix for this issue. I would simply update the import to be from . import py2_warn, but that adds an unwanted name to the namespace.

Is it possible for setuptools to provide a hook that would enable older versions of pyinstaller to work with newer versions of setuptools? How is that done?

[calebwoofenden]

@jaraco what fix does Pyinstaller have for this?

[jaraco]

I was mistaken. I misread the references above and thought they were applied to Pyinstaller.

[jaraco]

How is it that other similar imports in pkg_resources aren't a problem for pyinstaller?

Is it possible for setuptools to provide a hook that would enable pyinstaller to work with newer versions of setuptools? How is that done?

Here's how keyring adds a hook for hidden imports.

Could someone with Pyinstaller expertise make a recommendation on what's the recommended approach here?

[gsemet]

Fixed by forcing in a hook file the following submodule collection:

...
hiddenimports.extend(collect_submodules('pkg_resources'))

My hack works whatever the version of setuptools. I’ll see if I have time to do a clean PR in here.

[rkeerdo93]

Can also confirm this happens with later versions of PyInstaller (3.6)
Traceback (most recent call last): File "PyInstaller/loader/rthooks/pyi_rth_pkgres.py", line 13, in <module> File "/root/.local/share/virtualenvs/HIDDEN-2XAjamty/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module File "pkg_resources/__init__.py", line 86, in <module> ModuleNotFoundError: No module named 'pkg_resources.py2_warn' [24514] Failed to execute script pyi_rth_pkgres

[ifalex]

If you're using pyinstaller from bash you can use "--hidden-import='pkg_resources.py2_warn'" to get rid of this issue. Eg: pyinstaller --hidden-import='pkg_resources.py2_warn' ${package}

[anchalsingh298358]

I am getting his error while trying to convert my project exe file directroy with Auto-py-to-exe

Running auto-py-to-exe v2.9.0

Building directory: C:\Users\Dell\AppData\Local\Temp\tmpan03u43h

Provided command: pyinstaller --noconfirm --onedir --console --add-data "E:/Projects in Python/Drowsiness detection/alarm.wav;." --add-data "E:/Projects in Python/Drowsiness detection/model.py;." --add-data "E:/Projects in Python/Drowsiness detection/tkinter event.py;." --add-data "E:/Projects in Python/Drowsiness detection/models;models/" --add-data "E:/Projects in Python/Drowsiness detection/haar cascade files;haar cascade files/" "E:/Projects in Python/Drowsiness detection/drowsiness detection.py"

Recursion Limit is set to 5000

Executing: pyinstaller --noconfirm --onedir --console --add-data E:/Projects in Python/Drowsiness detection/alarm.wav;. --add-data E:/Projects in Python/Drowsiness detection/model.py;. --add-data E:/Projects in Python/Drowsiness detection/tkinter event.py;. --add-data E:/Projects in Python/Drowsiness detection/models;models/ --add-data E:/Projects in Python/Drowsiness detection/haar cascade files;haar cascade files/ E:/Projects in Python/Drowsiness detection/drowsiness detection.py --distpath C:\Users\Dell\AppData\Local\Temp\tmpan03u43h\application --workpath C:\Users\Dell\AppData\Local\Temp\tmpan03u43h\build --specpath C:\Users\Dell\AppData\Local\Temp\tmpan03u43h

736467 INFO: PyInstaller: 4.3

736489 INFO: Python: 3.6.8rc1

736498 INFO: Platform: Windows-10-10.0.19041-SP0

736520 INFO: wrote C:\Users\Dell\AppData\Local\Temp\tmpan03u43h\drowsiness detection.spec

736653 INFO: UPX is not available.

736783 INFO: Extending PYTHONPATH with paths
['E:\Projects in Python\Drowsiness detection',
'C:\Users\Dell\AppData\Local\Temp\tmpan03u43h']

744094 INFO: checking Analysis

744123 INFO: Building Analysis because Analysis-00.toc is non existent

744152 INFO: Initializing module dependency graph...

744198 INFO: Caching module graph hooks...

744302 WARNING: Several hooks defined for module 'win32ctypes.core'. Please take care they do not conflict.

744620 INFO: Analyzing base_library.zip ...

763780 INFO: Caching module dependency graph...

764287 INFO: running Analysis Analysis-00.toc

764398 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
required by c:\users\dell\appdata\local\programs\python\python36\python.exe

764756 INFO: Analyzing E:\Projects in Python\Drowsiness detection\drowsiness detection.py

766483 INFO: Processing pre-find module path hook distutils from 'c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\hooks\pre_find_module_path\hook-distutils.py'.

766558 INFO: distutils: retargeting to non-venv dir 'c:\users\dell\appdata\local\programs\python\python36\lib'

766839 INFO: Processing pre-find module path hook site from 'c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\hooks\pre_find_module_path\hook-site.py'.

766861 INFO: site: retargeting to fake-dir 'c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\fake-modules'

769115 INFO: Processing pre-safe import module hook six.moves from 'c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\hooks\pre_safe_import_module\hook-six.moves.py'.

893728 INFO: Processing pre-safe import module hook urllib3.packages.six.moves from 'c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\hooks\pre_safe_import_module\hook-urllib3.packages.six.moves.py'.

1408036 INFO: Processing module hooks...

1408057 INFO: Loading module hook 'hook-certifi.py' from 'c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\_pyinstaller_hooks_contrib\hooks\stdhooks'...

1408204 INFO: Loading module hook 'hook-cryptography.py' from 'c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\_pyinstaller_hooks_contrib\hooks\stdhooks'...

An error occurred while packaging
Traceback (most recent call last):
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\auto_py_to_exe\packaging.py", line 131, in package
run_pyinstaller()
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller_main_.py", line 114, in run
run_build(pyi_config, spec_file, **vars(args))
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller_main_.py", line 65, in run_build
PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\building\build_main.py", line 737, in main
build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\building\build_main.py", line 684, in build
exec(code, spec_namespace)
File "C:\Users\Dell\AppData\Local\Temp\tmpan03u43h\drowsiness detection.spec", line 18, in
noarchive=False)
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\building\build_main.py", line 242, in init
self.postinit()
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\building\datastruct.py", line 160, in postinit
self.assemble()
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\building\build_main.py", line 420, in assemble
self.graph.process_post_graph_hooks()
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\depend\analysis.py", line 367, in process_post_graph_hooks
module_hook.post_graph()
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\depend\imphook.py", line 447, in post_graph
self._load_hook_module()
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\depend\imphook.py", line 409, in _load_hook_module
self.hook_module_name, self.hook_filename)
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\compat.py", line 605, in importlib_load_source
return mod_loader.load_module()
File "", line 399, in check_name_wrapper
File "", line 823, in load_module
File "", line 682, in load_module
File "", line 265, in load_module_shim
File "", line 684, in load
File "", line 665, in load_unlocked
File "", line 678, in exec_module
File "", line 219, in call_with_frames_removed
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages_pyinstaller_hooks_contrib\hooks\stdhooks\hook-cryptography.py", line 26, in
datas = copy_metadata('cryptography')
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\utils\hooks_init.py", line 944, in copy_metadata
dist = pkg_resources.get_distribution(package_name)
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\pkg_resources_init.py", line 466, in get_distribution
dist = get_provider(dist)
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\pkg_resources_init.py", line 342, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\pkg_resources_init.py", line 886, in require
needed = self.resolve(parse_requirements(requirements))
File "c:\users\dell\appdata\local\programs\python\python36\lib\site-packages\pkg_resources_init.py", line 772, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'cryptography' distribution was not found and is required by the application

Project output will not be moved to output folder

Complete.