Staticx fails to recognize a PyInstaller archive

[ole-de]

Hi Jonathon,

finally I got PyInstaller working natively on Gentoo (amd64). But now staticx fails to recognize the single file binary as a PyInstaller archive. The reason apparently is, Gentoo creates the PyInstaller bootloader as

ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0

The cause for staticx failing is the "pie", position independent executable. If I force the PyInstaller bootloader to be compiled with "-no-pie" it works.

Does this make any sense to you? If staticx just requires the PyInstaller bootloader to be compiled/linked like this, I am fine with that. Just wanted to check back with you, as this may bother someone else as well.

Let me know, if you need any additional information.

Thanks, Ole

[JonathonReinhart]

Hi Ole,

Staticx doesn't know a lot about PyInstaller archives.

When building an archive, generate_archive calls run_hooks. This calls process_pyinstaller_archive which uses CArchiveReader from PyInstaller itself to determine if there are additional dependencies.

So I'm not sure why being a position-independent executable would cause CArchiveReader to fail to read the archive.

You could confirm this by running pyi-archive_viewer (installed with PyInstaller) on your single-file executable and confirm that it behaves the same.

You could also post a non-working executable from Gentoo and I can take a look.

[ole-de]

I am attaching a simple test case. It is a PyInstaller single file archive generated from

#! /usr/bin/env python3

print ("Hello World!")

I can dump the contents using

$ python3 /usr/bin/pyi-archive_viewer -b test
['struct',
 'pyimod01_os_path',
 'pyimod02_archive',
 'pyimod03_importers',
 'pyiboot01_bootstrap',
 'test',
 '_bisect.cpython-36m-x86_64-linux-gnu.so',
 '_blake2.cpython-36m-x86_64-linux-gnu.so',
 '_bz2.cpython-36m-x86_64-linux-gnu.so',
 '_codecs_cn.cpython-36m-x86_64-linux-gnu.so',
 '_codecs_hk.cpython-36m-x86_64-linux-gnu.so',
 '_codecs_iso2022.cpython-36m-x86_64-linux-gnu.so',
 '_codecs_jp.cpython-36m-x86_64-linux-gnu.so',
 '_codecs_kr.cpython-36m-x86_64-linux-gnu.so',
 '_codecs_tw.cpython-36m-x86_64-linux-gnu.so',
 '_datetime.cpython-36m-x86_64-linux-gnu.so',
 '_hashlib.cpython-36m-x86_64-linux-gnu.so',
 '_heapq.cpython-36m-x86_64-linux-gnu.so',
 '_lzma.cpython-36m-x86_64-linux-gnu.so',
 '_md5.cpython-36m-x86_64-linux-gnu.so',
 '_multibytecodec.cpython-36m-x86_64-linux-gnu.so',
 '_opcode.cpython-36m-x86_64-linux-gnu.so',
 '_pickle.cpython-36m-x86_64-linux-gnu.so',
 '_posixsubprocess.cpython-36m-x86_64-linux-gnu.so',
 '_random.cpython-36m-x86_64-linux-gnu.so',
 '_sha1.cpython-36m-x86_64-linux-gnu.so',
 '_sha256.cpython-36m-x86_64-linux-gnu.so',
 '_sha3.cpython-36m-x86_64-linux-gnu.so',
 '_sha512.cpython-36m-x86_64-linux-gnu.so',
 '_socket.cpython-36m-x86_64-linux-gnu.so',
 '_ssl.cpython-36m-x86_64-linux-gnu.so',
 '_struct.cpython-36m-x86_64-linux-gnu.so',
 'binascii.cpython-36m-x86_64-linux-gnu.so',
 'grp.cpython-36m-x86_64-linux-gnu.so',
 'libbz2.so.1',
 'libcrypto.so.1.0.0',
 'libexpat.so.1',
 'liblzma.so.5',
 'libncurses.so.6',
 'libncursesw.so.6',
 'libpython3.6m.so.1.0',
 'libreadline.so.7',
 'libssl.so.1.0.0',
 'libz.so.1',
 'math.cpython-36m-x86_64-linux-gnu.so',
 'pyexpat.cpython-36m-x86_64-linux-gnu.so',
 'readline.cpython-36m-x86_64-linux-gnu.so',
 'resource.cpython-36m-x86_64-linux-gnu.so',
 'select.cpython-36m-x86_64-linux-gnu.so',
 'termios.cpython-36m-x86_64-linux-gnu.so',
 'unicodedata.cpython-36m-x86_64-linux-gnu.so',
 'zlib.cpython-36m-x86_64-linux-gnu.so',
 'base_library.zip',
 'PYZ-00.pyz']

Anyway, after running staticx on it, the resulting binary gives:

$ ./test.static 
[23752] Cannot open self /tmp/staticx-ddQUST/.staticx.prog or archive /tmp/staticx-ddQUST/.staticx.prog.pkg

This is, since staticx fails to recognize "test" as a PyInstaller generated archive. For whatever reason...

test.zip

[JonathonReinhart]

Hi Ole,

I'm trying to play with the binary you provided, but it is crashing on this instruction:

C4E248F2C0        andn eax,esi,eax

ANDN is part of the BMI1 instruction set extension added in Haswell, so it looks like my CPU is too old 🙁 I could run it under full QEMU emulation, but this is becoming a bigger task than I have time for.

So, I'm working just off of the information you've provided above.

First, can you confirm that the application built with PyInstaller works without staticx?

This error:

$ ./test.static 
[23752] Cannot open self /tmp/staticx-ddQUST/.staticx.prog or archive /tmp/staticx-ddQUST/.staticx.prog.pkg

...is actually printed by the PyInstaller bootloader (in pyi_main()) and not by staticx. So it appears that the staticx bootloader was at least able to laucn .staticx.prog, the patched user binary (your PyInstaller app).

To debug this further, I think we need the debug output of the PyInstaller bootloader. Can you first build your PyInstaller application with pyinstaller -F --debug? Then when you run your application, you should see messages like this:

[18308] LOADER: Extracting binaries
[18308] LOADER: Executing self as child

Hopefully this output will tell us why PyInstaller doesn't like the application that we dropped out.

I'm adding a similar --debug command to staticx as well (see #87).

[sbneto]

I am having the same issue. I have used the --debug flag from #87 and --debug all and -F from pyinstaller to generate the final binary. Upon execution, it gives me:

STATICX [4051]: Version 0.7.0+g2dc7ce2
STATICX [4051]: Temporary bundle dir: /tmp/staticx-Z4lbSO
STATICX [4051]: Found archive at offset 0xD76A7 (20172560 bytes)
STATICX [4051]: Archive is XZ-compressed
STATICX [4051]: Extracting tar archive to /tmp/staticx-Z4lbSO
-rwxr-xr-x root     root      19831176 May 17  3:27 2019 .staticx.prog
lrw-r--r-- root     root             0 Jan  1  0:00 1970 .staticx.interp -> ld-linux-x86-64.so.2
lrw-r--r-- root     root             0 Jan  1  0:00 1970 libdl.so.2 -> libdl-2.24.so
-rw-r--r-- root     root         14640 Feb  6 21:17 2019 libdl-2.24.so
lrw-r--r-- root     root             0 Jan  1  0:00 1970 libz.so.1 -> libz.so.1.2.8
-rw-r--r-- root     root        105088 Jan 29 17:22 2017 libz.so.1.2.8
lrw-r--r-- root     root             0 Jan  1  0:00 1970 libc.so.6 -> libc-2.24.so
-rwxr-xr-x root     root       1689360 Feb  6 21:17 2019 libc-2.24.so
lrw-r--r-- root     root             0 Jan  1  0:00 1970 ld-linux-x86-64.so.2 -> ld-2.24.so
-rwxr-xr-x root     root        153288 Feb  6 21:17 2019 ld-2.24.so
lrw-r--r-- root     root             0 Jan  1  0:00 1970 libpthread.so.0 -> libpthread-2.24.so
-rwxr-xr-x root     root        135440 Feb  6 21:17 2019 libpthread-2.24.so
lrw-r--r-- root     root             0 Jan  1  0:00 1970 libutil.so.1 -> libutil-2.24.so
-rw-r--r-- root     root         10688 Feb  6 21:17 2019 libutil-2.24.so
lrw-r--r-- root     root             0 Jan  1  0:00 1970 libm.so.6 -> libm-2.24.so
-rw-r--r-- root     root       1063328 Feb  6 21:17 2019 libm-2.24.so
STATICX [4051]: Successfully extracted archive to /tmp/staticx-Z4lbSO
STATICX [4051]: Patching user program /tmp/staticx-Z4lbSO/.staticx.prog
STATICX [4051]: Current program interpreter: "i...(256 times)"
STATICX [4051]: Set new interpreter: "/tmp/staticx-Z4lbSO/.staticx.interp"
STATICX [4051]: Current RPATH (0x33E): "r...(256 times)"
STATICX [4051]: Set new RPATH: "/tmp/staticx-Z4lbSO"
STATICX [4051]: Setting env var: STATICX_BUNDLE_DIR=/tmp/staticx-Z4lbSO
STATICX [4051]: Setting env var: STATICX_PROG_PATH=/app/dist/s3conf
STATICX [4051]: Ready to run child process with new argv:
STATICX [4051]:   [0] = "/tmp/staticx-Z4lbSO/.staticx.prog"
STATICX [4052]: Child process started; ready to call execv()
[4052] PyInstaller Bootloader 3.x
[4052] LOADER: executable is /tmp/staticx-Z4lbSO/.staticx.prog
[4052] LOADER: homepath is /tmp/staticx-Z4lbSO
[4052] LOADER: _MEIPASS2 is NULL
[4052] LOADER: archivename is /tmp/staticx-Z4lbSO/.staticx.prog
[4052] Loader: Cannot find cookie[4052] LOADER: archivename is /tmp/staticx-Z4lbSO/.staticx.prog.pkg
[4052] LOADER: Cannot open archive: /tmp/staticx-Z4lbSO/.staticx.prog.pkg
[4052] Cannot open self /tmp/staticx-Z4lbSO/.staticx.prog or archive /tmp/staticx-Z4lbSO/.staticx.prog.pkg
STATICX [4051]: Child process terminated with wstatus=65280
STATICX [4051]: Removing temporary bundle dir /tmp/staticx-Z4lbSO
STATICX [4051]: Child exited with status 255

When running the pyinstaller generated binary, it succeeds with:

[4053] PyInstaller Bootloader 3.x
[4053] LOADER: executable is /app/dist/s3conf-Linux-x86_64
[4053] LOADER: homepath is /app/dist
[4053] LOADER: _MEIPASS2 is NULL
[4053] LOADER: archivename is /app/dist/s3conf-Linux-x86_64
[4053] LOADER: Extracting binaries
[4053] LOADER: Executing self as child
[4053] LOADER: set _MEIPASS2 to /tmp/_MEIUr8RNw
[4053] LOADER: LD_LIBRARY_PATH=/tmp/_MEIUr8RNw
[4053] LOADER: Registering signal handlers
[4054] PyInstaller Bootloader 3.x
[4054] LOADER: executable is /app/dist/s3conf-Linux-x86_64
[4054] LOADER: homepath is /app/dist
[4054] LOADER: _MEIPASS2 is /tmp/_MEIUr8RNw
[4054] LOADER: archivename is /app/dist/s3conf-Linux-x86_64
[4054] LOADER: Already in the child - running user's code.
[4054] LOADER: Python library: /tmp/_MEIUr8RNw/libpython3.6m.so.1.0
[4054] LOADER: Loaded functions from Python library.
[4054] LOADER: Manipulating environment (sys.path, sys.prefix)
[4054] LOADER: sys.prefix is /tmp/_MEIUr8RNw
[4054] LOADER: Pre-init sys.path is /tmp/_MEIUr8RNw/base_library.zip:/tmp/_MEIUr8RNw
[4054] LOADER: Setting runtime options
[4054] LOADER: Initializing python
[4054] LOADER: Overriding Python's sys.path
[4054] LOADER: Post-init sys.path is /tmp/_MEIUr8RNw/base_library.zip:/tmp/_MEIUr8RNw
[4054] LOADER: Setting sys.argv
[4054] LOADER: setting sys._MEIPASS
[4054] LOADER: importing modules from CArchive
[4054] LOADER: extracted struct
[4054] LOADER: callfunction returned...
[4054] LOADER: extracted pyimod01_os_path
[4054] LOADER: callfunction returned...
[4054] LOADER: extracted pyimod02_archive
[4054] LOADER: callfunction returned...
[4054] LOADER: extracted pyimod03_importers
[4054] LOADER: callfunction returned...
[4054] LOADER: Installing PYZ archive with Python modules.
[4054] LOADER: PYZ archive: PYZ-00.pyz
[4054] LOADER: Running pyiboot01_bootstrap.py
[4054] LOADER: Running pyi_rth_multiprocessing.py
[4054] LOADER: Running pyi_rth_pkgres.py
[4054] LOADER: Running pyinstaller.py
.
.
.

Not really sure what else to try.

[ole-de]

Hi Jonathon,

first of all, sorry for getting back to this VERY late.

Luckily this is not an issue for me any more. I am compiling the pyinstaller loader using -no-pie and then everything works like a charm.

Interestingly it also looks like the latest default on Gentoo (at least using gcc8) seems to be non-pie executables as well.

Anyway, when I am using -pie in my compile of the pyinstaller loader, I am still able to reproduce the issue. Unfortunately the loader for --debug, which is

/usr/lib64/python2.7/site-packages/PyInstaller/bootloader/Linux-64bit/run_d

for me. is compiled as non-pie even when using -pie in CFLAGS. Don't know why.

So, I am unsure how to help any further. As the issue can be perfectly worked around with using a non-pie loader of pyinstaller, I think a note in the docs could easily do it.

@sbneto: You should check, if this also works for you.

BTW: The 0.7.0 does not seem to have the --debug option yet. So there is no debug output for pyinstaller (as the loader is always no-pie) and no debug output for staticx as well with my installation. Sorry about that.

I am anyway attaching a tarball just as a reference how I was using pyinstaller and staticx for testing.

Thanks a lot for this great tool. Helped me out of some trouble!

Cheers, Ole

staticx-issue71.zip

[bwoodsend]

This is fixed in pyinstaller/pyinstaller#5511. You can close this.

[JonathonReinhart]

Great! Thanks for letting me know, @bwoodsend.

[ssuresh-10]

Hi
I am using Python 3.7 with Pyinstaller 3.5 and stacticx-0.7.0 versions and it was working without any issue in ldap environment, now we moved to Active Directory and I am encountering - 'Cannot open self /tmp/staticx-hAhhdA/.staticx.prog or archive /tmp/staticx-hAhhdA/.staticx.prog.pkg' . Recent pull of pyinstaller 4.2 and staticx also produces the same error. Would like to know how to address this issue, any pointers would be really helpful.
Thanks

[bwoodsend]

The fix hasn't been released yet. You'd need to install the development version of PyInstaller.

pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

Then rebuild but add the --clean option.

[ssuresh-10]

Thank you for the pointer, will pull and do the build I am new to python and pyinstaller , I am kind of curious to understand though why the previously working python-3.7 pyinstaller-3.5 staticx-0.7.0 versions in the ldap are not working now. Is it anything to do with Active Directory environment? Thanks,

…

________________________________ From: Brénainn Woodsend ***@***.***> Sent: Friday, April 2, 2021 5:17 PM To: JonathonReinhart/staticx ***@***.***> Cc: Subashree Suresh ***@***.***>; Comment ***@***.***> Subject: [External] Re: [JonathonReinhart/staticx] Staticx fails to recognize a PyInstaller archive (#71) The fix hasn't been released yet. You'd need to install the development version of PyInstaller. pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip Then rebuild but add the --clean option. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#71 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATQKAO3IHNNDERI7YNAP4Q3TGYX73ANCNFSM4GE3HLAA>.

[JonathonReinhart]

@ssuresh-10 Given that the issue is caused by the particular PyInstaller build options, I don't see how Active Directory is involved.

However, please read these caveats regarding NSS.