Unable to build executable using embedded zip Python

[ruudkassing]

Description of the issue

The official Python embeddable zip file distributions for Windows from Python.org contain a zip file (pythonXX.zip) that contains the pre-compiled and optimized .pyc files to run Python isolated on a machine. I've installed the embedded Python on a local agent that we use in our azure devops pipeline, following the Microsoft docs.

However, building on this agent using PyInstaller fails, see Stacktrace below. This failure occurs because the requirements for base_library.zip are located inside a zip. Thus, PyInstaller tries to do an os.stat() on a path inside a zip folder, which fails.

Context information (for bug reports)

• Output of pyinstaller --version: 3.6
• Version of Python: 3.7.7
• Platform: Windows 10
• Same failure occurs on newest development version of PyInstaller
• Followed all the instructions in the "If Things Go Wrong" Guide and checked that everything is packaged correctly. The error occurs regardless of PyInstaller settings or spec file content.

A minimal example program which shows the error

• Download embedded Python from Python.org
• Add pip using get-pip
• Install PyInstaller
• Create helloworld.py

  print("Hello World!")
  

• Try to run PyInstaller on helloworld.py using any settings.

Stacktrace / full error message

==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.164.2
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
Script contents:
python -m PyInstaller --distpath "%cd%\exe" SPECFILENAME.spec
========================== Starting Command Output ===========================
"C:\WINDOWS\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "agent2\_work\_temp\44e0b0b7-f2f9-4990-a3ee-31bbec9a0c70.cmd""
285 INFO: PyInstaller: 3.6
285 INFO: Python: 3.7.7
285 INFO: Platform: Windows-10-10.0.18362-SP0
291 INFO: UPX is not available.
303 INFO: checking Analysis
307 INFO: Building Analysis because Analysis-00.toc is non existent
307 INFO: Initializing module dependency graph...
313 INFO: Caching module graph hooks...
326 INFO: Analyzing base_library.zip ...
1791 INFO: Processing pre-find module path hook   distutils
1804 INFO: distutils: retargeting to non-venv dir 'agent2\\_work\\_tool\\Python\\3.7.7\\x64\\python37.zip'
2579 INFO: Caching module dependency graph...
2629 ERROR: base_library.zip could not be created!
Traceback (most recent call last):
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\runpy.py", line 193, in _run_module_as_main
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\runpy.py", line 85, in _run_code
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\__main__.py", line 121, in <module>
    run()
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\__main__.py", line 114, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\building\build_main.py", line 734, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\building\build_main.py", line 681, in build
    exec(code, spec_namespace)
  File "SPECFILENAME.spec", line 24, in <module>
    noarchive=False)
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\building\build_main.py", line 244, in __init__
    self.__postinit__()
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\building\datastruct.py", line 160, in __postinit__
    self.assemble()
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\building\build_main.py", line 352, in assemble
    create_py3_base_library(libzip_filename, graph=self.graph)
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\depend\utils.py", line 74, in create_py3_base_library
    st = os.stat(mod.filename)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'agent2\\_work\\_tool\\Python\\3.7.7\\x64\\python37.zip\\encodings\\__init__.pyc'
##[error]Cmd.exe exited with code '1'.

[ruudkassing]

I manually adjusted depends\utils.py in the PyInstaller package to handle files inside a zip folder. However, this did not solve the issue as PyInstaller would then continue to load the hook-distutils.py and attempt to get Include\pyconfig.h which does not exist for embedded zip Python:

169 INFO: PyInstaller: 3.6
169 INFO: Python: 3.7.7
169 INFO: Platform: Windows-10-10.0.18362-SP0
184 INFO: UPX is not available.
184 INFO: checking Analysis
200 INFO: Building Analysis because Analysis-00.toc is non existent
200 INFO: Initializing module dependency graph...
200 INFO: Caching module graph hooks...
222 INFO: Analyzing base_library.zip ...
1140 INFO: Processing pre-find module path hook   distutils
1156 INFO: distutils: retargeting to non-venv dir 'Python\\3.7.7\\x64\\python37.zip'
1689 INFO: Caching module dependency graph...
1742 INFO: running Analysis Analysis-00.toc
1742 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
  required by Python\3.7.7\x64\python.exe
1990 INFO: Analyzing FILE_TO_BUILD_TO_EXECUTABLE.py
3078 INFO: Processing pre-safe import module hook   setuptools.extern.six.moves
3627 INFO: Processing pre-find module path hook   site
3633 INFO: site: retargeting to fake-dir 'Python\\3.7.7\\x64\\lib\\site-packages\\PyInstaller\\fake-modules'
7645 INFO: Processing module hooks...
7645 INFO: Loading module hook "hook-distutils.py"...
Unable to find "Python\3.7.7\x64\Include\pyconfig.h" when adding binary and data files.This would mean your Python installation doesn't come with proper library files. This usually happens by missing development package, or unsuitable build parameters of Python installation.
* On Debian/Ubuntu, you would need to install Python development packages
  * apt-get install python3-dev
  * apt-get install python-dev
* If you're building Python by yourself, please rebuild your Python with
`--enable-shared` (or, `--enable-framework` on Darwin)

I found that it's related to PR #4182 and issue #4181. As mentioned in the PR, this issue was not actually fixed as filesystem operations are still used, even if Python is embedded in a zip file. Furthermore, as shown in this reply, it's unfortunately not as simple as only addressing the filesystem operations.

[Legorooj]

@Ruud-K this is indeed a bug. I shall fix this in the next few days before the next release.

[menteora]

Hi, is there any temporary o manual workaround to bypass error using embedded python?

[Legorooj]

@menteora nope - sorry this got buried. I'll do this next - it's now top of my to-do list.

[bwoodsend]

@Ruud-K Do you still have the changes you made to get PyInstaller to handle the zipped libs? The pyconfig.h issue should be fixed by #5218 once it goes through. So if we apply your changes afterwards, this issue will hopefully be fixed.

[shouldsee]

I ran into similar issues when using pyinstaller with embedded python3.7.9. The solution was to use the --onedir option and then copy from PYTHONHOME the python37.zip into dist\{appname}\base_library.zip

[shouldsee]

This seems to be a long-running issue and the content of base_library.zip has been updated several times

• Missing warnings module in base_library.zip #3397, f101f6f, March 2018, added warnings module
• Python 3.5 code compiled with PyInstaller 3.0 crashes running under code page 65001 on Windows #1705, e7fc427, Dec 2015, added functools and a bunch, and holds an informative discussion

I have a naive suggestion here: maybe we can consider using embeddable python.zip directly as base_library.zip? Otherwise we might ends up updating PY3_BASE_MODULES once anyone runs into a missing module problem. And because --onefile mode exe erase its temporary directory, it's even harder for the user to find out the actual reason of a Cannot import ... Errorr .

Also configuring embeddable python alone is already a demanding task and I need to thank the creator of this post https://dev.to/fpim/setting-up-python-s-windows-embeddable-distribution-properly-1081

[parinavcodes]

I'll work on this to fix the bug right away

[wsc0615]

I successfully generated the exe file through an ad hoc solution like this: in the embeddable python, there is a pythonxx .zip of the package, extract the package and rename it to the pyhonxx.zip folder, and run the pyinstaller again. Witness the miracle.
Of course, I still hope to completely solve this bug and facilitate the use of embeddable Python.

[listenerri]

still

[SimoLRepo]

No definitive solution?

In my case it appears:
FileNotFoundError: [Errno 2] No such file or directory: 'Python_amd64\python311.zip\struct.pyc'