Code compiled with 5.6+ doesn't run

[aefruswg]

Description of the issue

Using PyInstaller to compile a Flask app to run in Docker. We're doing a multi-stage build with the compilation happening on the first stage in the official Python 3.10 Debian image, and the binary is run on a second Debian (MiniDeb) image. Manually setting the version of PyInstaller to 5.4 resolves the issue. Did not test with 5.5, issue is present on 5.6 and 5.6.1

Context information (for bug reports)

• Output of pyinstaller --version: Not applicable in a non-functional Docker container, but each build pulls the most recent version

• Version of Python: 3.10

• Platform: Debian (MiniDeb) Bullseye

• How you installed Python: Official Python docker image

• Did you also try this on another platform? Does it work there?: N/A

• try the latest development version, using the following command:

pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

• follow all the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

Make sure everything is packaged correctly

• start with clean installation
• use the latest development version
• Run your frozen program from a command window (shell) — instead of double-clicking on it
• Package your program in --onedir mode
• Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
• Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.

A minimal example program which shows the error

Stacktrace / full error message

Traceback (most recent call last):
  File "PyInstaller/hooks/rthooks/pyi_rth_pkgres.py", line 16, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 499, in exec_module
  File "pkg_resources/__init__.py", line 32, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 499, in exec_module
  File "plistlib.py", line 61, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 499, in exec_module
  File "xml/parsers/expat.py", line 4, in <module>
ImportError: libexpat.so.1: cannot open shared object file: No such file or directory
[11] Failed to execute script 'pyi_rth_pkgres' due to unhandled exception!

Please also see https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs
for more about what would use to solve the issue.

[bwoodsend]

Can you give us the names of this images? e.g. is it python:3.10 or python:3.10-slim and bullseye or buster. I'm guessing that minideb means this?

[aefruswg]

Correct, we're using the Bullseye image of bitnami minideb for the resulting container. And the first stage of the build uses python:3.10-bullseye

[rokm]

Hmm, yeah... 5.4 (and also 5.5) seems to collect expat library as libexpat.so.1, while 5.6 collects the fully-versioned libexpat.so.1.6.12 instead. And pyexpat extension is linked against the .so.1 variant, hence the error.

[bwoodsend]

Caused by af2cd69 according to bisect.

[bwoodsend]

Although I can't see why.

[rokm]

Yeah, I also just bisected it to af2cd69...

[rokm]

There's probalby a symlink resolution that slipped in somewhere...

[rokm]

Ah, it's because it now constructs destination name based on resolved npth, which resolves the symlink.

[bwoodsend]

Yeah. Previously, src_path was set to the resolved variant but not used unless this conditional was true. Now src_path sets the name either way. That means that both versions are broken really - the old version would have had the same issue if the linked library was in a sys.path prefix?

[bwoodsend]

(Also, yet again I find I'm cursing myself for not refactoring away that npth variable name 🤮 )

[rokm]

That means that both versions are broken really - the old version would have had the same issue if the linked library was in a sys.path prefix?

Yeah, the design was broken, but it had no effect earlier because we populate preservation paths only on Windows, so the former else branch was always taken on linux and macOS. But after the refactoring, the symlink resolution became effectively active on linux and macOS - which would become a problem once we tried to turn on shared lib directory structure preservation on those OSes.

I think the fix is to simply remove the .resolve() call when turning npth into pathlib.Path here. This way, the filename should stay the same, regardless if the file is collected into sub-directory or into top-level directory.

[rokm]

And, once we start supporting symlinks, we should detect symlinks and collect both the link and its target file.

[bwoodsend]

I think the fix is to simply remove the .resolve() call when turning npth into pathlib.Path here. This way, the filename should stay the same, regardless if the file is collected into sub-directory or into top-level directory.

Sounds good to me. Looks like another patch release then... Don't think PyInstaller has ever reached an x.y.2 version before.