Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Binary tools spawned from exe made with PyInstaller and StaticX pick up wrong libraries #239

Open
quite68 opened this issue Jun 5, 2023 · 2 comments
Open

Binary tools spawned from exe made with PyInstaller and StaticX pick up wrong libraries #239

quite68 opened this issue Jun 5, 2023 · 2 comments

Comments

@quite68
Copy link

quite68 commented Jun 5, 2023 •
edited by JonathonReinhart

We have a standalone exe we've created with PyInstaller and StaticX. It spawns some binary tools such as adb and our own tools that are bundled in by PyInstaller. These do not pick up libraries correctly.

E.g. running an exe we created on Ubuntu 22.04 on Ubuntu 16.04 gives the following:

Shell failed. command: adb start-server
Shell failed. output:  adb: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.35' not found (required by /tmp/_MEIs3cXJk/libgcc_s.so.1)
adb: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/_MEIs3cXJk/libgcc_s.so.1)

This is running a pre-installed adb which runs fine when run from the command line. However when run from the exe created by StaticX it seems to pick up a libgcc_s.so from the tmp directory created by StaticX. This then needs GLIBC_2.34 which isn't available. (Ubuntu 16.04 only has 2.23).

I've tried asking StaticX to include a libc which -l /usr/lib/x86_64-linux-gnu/libc.so.6 but this doesn't seem to help.
@JonathonReinhart
Copy link
Owner

JonathonReinhart commented Jun 22, 2023 •
edited

Hi, sorry I'm just getting to this.

TL;DR

Read LD_LIBRARY_PATH / LIBPATH considerations and follow their advice to unset LD_LIBRARY_PATH when invoking target system apps from a Py-installed program.

Background

So, this is complicated. First, you should understand how PyInstaller and Staticx work under the covers.

PyInstaller

After the PyInstaller bootloader extracts the embedded archive, it launches the Python runtime and tweaks a few path-related things to make sure that the files from the unpacked archive are used. This involves Python sys.path for loading modules as well as LD_LIBRARY_PATH environment variable for telling ld.so where to find shared libraries.

This works great for basic scripts which may include extension modules (.so). However, the LD_LIBRARY_PATH environment variable persists across fork+exec, which means it applies to all child processes too. This includes when you're trying to run external programs -- both those from the host system, as well as ones you bring along in the PyInstaller archive! This is not what you want for system executables, but may be what you want for bundled applications..... however it is probably not good enough for those.... enter staticx.

See also:

Staticx

Staticx works differently. (And I need to document this #241). It does not use LD_LIBRARY_PATH.

At build-time, when given an app to staticx-ify:

  • Makes a copy of the app, and modifies it slightly, before adding it to a new staticx archive
    • Modifies the INTERP and RPATH fields to long dummy values which will be fixed up by the bootloader at runtime
  • Examines the app to determine all of its dynamic library dependencies (via ldd), and adds them to the staticx archive
    • This includes things that PyInstaller explicitly excludes, like ld.so and libc.so
  • If the app is a PyInstaller onefile app (see hooks/pyinstaller.py):
    • Iterates the contents of the embedded archive, and extracts all binary files for analysis
    • Examines all extracted shared objects, and adds all of their dependencies to the staticx archive
    • (As I write this, I'm actually second-guessing the correctness of this approach, but I digress)
  • Creates a copy of the staticx bootloader, and attaches the generated archive to it (via new ELF section)

At runtime, the bootloader:

  • Creates a temporary dir (the "bundle dir") and extracts the embedded archive (just like PyInstaller)
  • But now we need to run your embedded app, making sure to only reference files from the bundle dir!
    • Patch your embedded app, replacing:
      • RPATH: The absolute path of the bundle dir
      • INTERP: The absolute path of ld.so in the bundle dir
  • The bootloader does set a couple environment variables, but these are purely informative and do not affect the loading of shared libraries, etc.
    • STATICX_BUNDLE_DIR
    • STATICX_PROG_PATH
  • Forks + execs the modified app
    • ld.so (interp) gets control. It sees RPATH (and NODEFLIB) and restricts its search for shared objects to that directory
      • (I need to double-check how this interacts with LD_LIBRARY_PATH which will be subsequently set by the PyInstaller bootloader, which is 'the app')
    • NOTE: The RPATH trick does not apply to child processes!

Problem

Now, back to your question.

It spawns some binary tools such as adb and our own tools that are bundled in by PyInstaller. These do not pick up libraries correctly.

However when run from the exe created by StaticX it seems to pick up a libgcc_s.so from the tmp directory created by StaticX.

Correction: /tmp/_MEIxxxxxx is a temporary directory created by PyInstaller. The staticx tempdir is named /tmp/staticx-XXXXXX.

I think you said adb is expected to be installed on the target system. But let's discuss. both cases:

Invoking system apps

If you're trying to invoke an app installed on the target system, the problem is exactly as as described in the PyInstaller docs, and includes a solution: LD_LIBRARY_PATH / LIBPATH considerations.

I think this explains your specific error:

Shell failed. output:  adb: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.35' not found (required by /tmp/_MEIs3cXJk/libgcc_s.so.1)
adb: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/_MEIs3cXJk/libgcc_s.so.1)

So:

  • Staticx did its job and launched your PyInstalled app
  • PyInstaller did its job and loaded all of the required libraries
  • Your python code ran adb (from the system), but left LD_LIBRARY_PATH set
  • system ld.so interpreter loaded it
  • system libc.so (old) was loaded
    • I'm not 100% sure why your attempt to bring libc.so didn't work; it's possible that PyInstaller ignored it, or that the target system ld.so config ignored LD_LIBRARY_PATH for libc. That code is complex.
  • PyInstaller-bundled libgcc_s.so (new) was loaded
    • BUT it depends on newer libc

Try their advice.

Invoking bundled apps

This is where it gets tricky. But it basically devolves into the common staticx use case!

If you do nothing special, PyIntaller's LD_LIBRARY_PATH will apply; if you take the PyInstaller advice from the previous section, it will not.

  • Either way, the system dynamic linker (e.g. /lib64/ld-linux-x86-64.so.2) will load the application. There's nothing you can do about this. (Other than also staticx-ify those apps before bundling them into your pyinstaller app, and then staticx again. Russian dolls!)
  • Either way, libc.so will come from the system, because PyInstaller does not bundle it
  • Other libraries will either come from the PyInstaller bundle (if LD_LIBRARY_PATH is still set and they were included), or the target system if not.

The only real solution here today is nested-staticx.

Hope this helps! Please let me know if I can clarify anything.

@JonathonReinhart JonathonReinhart mentioned this issue Jun 22, 2023
Open
@quite68
Copy link
Author

quite68 commented Jun 22, 2023 via email

@brettdh brettdh mentioned this issue Jul 12, 2023
Open
@bashonly bashonly mentioned this issue May 1, 2024
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JonathonReinhart @quite68