upx causes --onefile exe to unpack too long

[Zenadix]

I originally posted this on Stack Overflow.

I'm trying to create a single-file executable for Windows from a Python application, using pyinstaller.

I downloaded the experimental Python 3 branch of pyinstaller from here (scroll down and you'll find the download link, the file is py3.zip). And I installed it using python setup.py install. UPDATE: I tried it with the Python 2 version too, and ran into the same problem.

Then I created a test python script called test.py, with the following content:

print('Hello, World!')

Afterwards, I ran the following command to create a single-file executable:

pyinstaller --onefile test.py

The command succeeded, and I verified that the file dist/test.exe had been generated. However, when I try to run it, all I get is an empty cmd window. Nothing ever appears, and the program never terminates. It just hangs there forever, until I force close it.

Calling pyinstaller test.py (without the --onefile option) works fine.

UPDATE: I just tested it under Python 2 (using the normal PyInstaller version), and I ran into the same problem. So, this is not just a Python 3 problem.

[codewarrior0]

Could you try running pyinstaller --debug --onefile test.py? You'll get some output about the progress of the loader when you run it.

[Zenadix]

Here is the output of pyinstaller --debug --onefile test.py.

C:\mac\des\ide\test\cxora>pyinstaller --debug --onefile test.py
181 INFO: wrote C:\mac\des\ide\test\cxora\test.spec
183 INFO: Testing for ability to set icons, version resources...
240 INFO: ... resource update available
305 INFO: UPX is available.
321 INFO: Extending PYTHONPATH with C:\mac\des\ide\test\cxora
321 INFO: checking Analysis
322 INFO: building Analysis because out00-Analysis.toc non existent
322 INFO: Creating base_library.zip for Python 3
3075 INFO: running Analysis out00-Analysis.toc
3282 INFO: Analyzing test.py
3283 INFO: Analyzing C:\mac\des\WinPython-32bit-3.3.5.0\python-3.3.5\lib\site-packages\py
installer-3.0dev-py3.3.egg\PyInstaller\loader\_pyi_bootstrap.py
5267 INFO: Analyzing C:\mac\des\WinPython-32bit-3.3.5.0\python-3.3.5\lib\site-packages\py
installer-3.0dev-py3.3.egg\PyInstaller\loader\pyi_importers.py
5272 INFO: Analyzing C:\mac\des\WinPython-32bit-3.3.5.0\python-3.3.5\lib\site-packages\py
installer-3.0dev-py3.3.egg\PyInstaller\loader\pyi_archive.py
5276 INFO: Analyzing C:\mac\des\WinPython-32bit-3.3.5.0\python-3.3.5\lib\site-packages\py
installer-3.0dev-py3.3.egg\PyInstaller\loader\pyi_carchive.py
5281 INFO: Analyzing C:\mac\des\WinPython-32bit-3.3.5.0\python-3.3.5\lib\site-packages\py
installer-3.0dev-py3.3.egg\PyInstaller\loader\pyi_os_path.py
5284 INFO: Hidden import 'codecs' has been found otherwise
5285 INFO: Hidden import 'encodings' has been found otherwise
5285 INFO: Looking for import hooks ...
5295 INFO: Processing hook   hook-encodings.py
5586 INFO: Processing hook   hook-pydoc.py
5590 INFO: Processing hook   hook-distutils.py
5595 INFO: Processing hook   hook-xml.py
5730 INFO: Processing hook   hook-xml.sax.py
5739 INFO: Processing hook   hook-site.py
5743 INFO: Processing hook   hook-sysconfig.py
5746 INFO: Analyzing run-time hooks ...
5762 INFO: Looking for dynamic libraries
6385 INFO: Looking for eggs - TODO
6408 INFO: Using Python library C:\mac\des\WinPython-32bit-3.3.5.0\python-3.3.5\python33.
dll
6465 INFO: Warnings written to C:\mac\des\ide\test\cxora\build\test\warntest.txt
6469 INFO: checking PYZ
6469 INFO: rebuilding out00-PYZ.toc because out00-PYZ.pyz is missing
6469 INFO: building PYZ (ZlibArchive) out00-PYZ.toc
7455 INFO: checking PKG
7456 INFO: rebuilding out00-PKG.toc because out00-PKG.pkg is missing
7456 INFO: building PKG (CArchive) out00-PKG.pkg
9136 INFO: checking EXE
9137 INFO: rebuilding out00-EXE.toc because test.exe missing
9137 INFO: building EXE from out00-EXE.toc
9141 INFO: Appending archive to EXE C:\mac\des\ide\test\cxora\dist\test.exe

Then, when I run test.exe, this is the output.

C:\mac\des\ide\test\cxora\dist>test.exe
LOADER: executable is C:\mac\des\ide\test\cxora\dist\test.exe
LOADER: homepath is C:\mac\des\ide\test\cxora\dist
LOADER: _MEIPASS2 is NULL
LOADER: archivename is C:\mac\des\ide\test\cxora\dist\test.exe
LOADER: Extracting binaries

And then it hangs for several minutes. Then it continues.

LOADER: Executing self as child
LOADER: set _MEIPASS2 to C:\Users\s39546\AppData\Local\Temp\_MEI42322
LOADER: Setting up to run child
LOADER: Creating child process
LOADER: Waiting for child process to finish...
LOADER: executable is C:\mac\des\ide\test\cxora\dist\test.exe
LOADER: homepath is C:\mac\des\ide\test\cxora\dist
LOADER: _MEIPASS2 is C:\Users\s39546\AppData\Local\Temp\_MEI42322
LOADER: archivename is C:\mac\des\ide\test\cxora\dist\test.exe
LOADER: Already in the child - running user's code.
LOADER: manifestpath: C:\Users\s39546\AppData\Local\Temp\_MEI42322\test.exe.manifest
LOADER: Activation context created
LOADER: Activation context activated
LOADER: Python library: C:\Users\s39546\AppData\Local\Temp\_MEI42322\python33.dll
LOADER: Loaded functions from Python library.
LOADER: Manipulating evironment (PYTHONPATH, PYTHONHOME)
LOADER: PYTHONPATH is C:\Users\s39546\AppData\Local\Temp\_MEI42322\base_library.zip;C:\Users\s39546\AppData\Local\Temp\_MEI42322
LOADER: PYTHONHOME is C:\Users\s39546\AppData\Local\Temp\_MEI42322
LOADER: Setting runtime options
LOADER: Initializing python
LOADER: Setting sys.argv
LOADER: importing modules from CArchive
LOADER: extracted struct
LOADER: callfunction returned...
LOADER: extracted pyi_os_path
LOADER: callfunction returned...
LOADER: extracted pyi_archive
LOADER: callfunction returned...
LOADER: extracted pyi_importers
LOADER: callfunction returned...
LOADER: Installing import hooks
LOADER: out00-PYZ.pyz
LOADER: import sys; sys.path.append(r"C:\mac\des\ide\test\cxora\dist\test.exe?94208")

LOADER: Running scripts
Hello, World!
LOADER: OK.
LOADER: Deactivating activation context
LOADER: Releasing activation context
LOADER: Done
LOADER: Cleaning up Python interpreter.
LOADER: Back to parent
LOADER: Doing cleanup
LOADER: Freeing archive status for C:\mac\des\ide\test\cxora\dist\test.exe

So I realize that the program does indeed run, but it hangs for several minutes before starting.

[matysek]

Please try with option --noupx

[Zenadix]

The --noupx option fixed it! Now the executable takes barely one second to start. I would never have guessed that upx was responsible for this issue.

Why does upx cause such long startup times? Is this a bug?

[matysek]

upx might add some other overhad. Not sure how exactly.

@Zenadix Do you think the fix could be to not use upx compression for the exe created by pyinstaller but use upx only for dlls?

[Zenadix]

@matysek I really couldn't tell. I think the only way to know is to try it.

For now, disabling upx completely does the trick for me.

[AndCycle]

does anyone with this problem tried without antivirus software enable?
most antivirus software will block upx compressed file first before moving on which cause significant delay.

[den-run-ai]

maybe using 7-zip or IExpress can help with extraction & running time:

https://en.wikipedia.org/wiki/IExpress
http://stackoverflow.com/a/30896241/2230844

@htgoebel ?

[htgoebel]

If somebody wants to implement support for a different packer, please step forward and provide a pull-request.

[Legorooj]

Closing this. UPX compresses exe's; therefore it has to uncompress them before runtime.