Add support for pip-installed console_scripts execution through subprocess.run

[bheadmaster]

Is your feature request related to a problem? Please describe.
PyInstaller does not currently support calling console_scripts through subprocess.run().

Describe the solution you'd like
The solution would be to include all console_scripts in the application bundle and add their location to $PATH.

Describe alternatives you've considered
I have managed to work around the issue by using --add-binary $(which $CONSOLESCRIPT):. and executing the binary in code with subprocess.run([os.path.join(os.path.dirname(__file__), $CONSOLESCRIPT]), but it is hacky and error-prone.

For example, uwsgi does not work properly with such setup, but that might be because it uses some kind of custom-built python executable for running, so maybe that use case is more complex to implement. But basic pip-installed console_scripts should work the same as they work in a python environment.

Additional context
Here's a minimal reproducible example (Ubuntu 20.04):

Let's define a package hello_world with the following file structure:

hello_world
├── hello_world
│   └── __init__.py
└── setup.py

Such that the contents of files are:

# hello_world/hello_world/__init__.py
def hello_world():
    print('Hello World')

# hello_world/setup.py
import setuptools

setuptools.setup(
    name='hello_world',
    packages=setuptools.find_packages(),
    entry_points={
        'console_scripts': [
            'hello_world = hello_world:hello_world',
        ],
    },
)

Let's now define an application main.py in which we use the hello_world script:

# main.py
import subprocess

subprocess.run(["hello_world"])

Assuming that we are in a venv, and we have the above defined hello_world package installed, running pyinstaller main.py creates an application bundle in a dist/main directory.
Deactivating venv and running dist/main/main now raises the following error:

Traceback (most recent call last):
  File "main.py", line 6, in <module>
    subprocess.run(['hello_world'])
  File "subprocess.py", line 493, in run
  File "subprocess.py", line 858, in __init__
  File "subprocess.py", line 1704, in _execute_child
FileNotFoundError: [Errno 2] No such file or directory: 'hello_world'
[31726] Failed to execute script 'main' due to unhandled exception

If we were to properly bundle all console_scripts with the application, the previous example would print:

Hello World

and exit without error.

[bwoodsend]

Ughh, this request again! At a distance it looks like a really neat feature until you start trying to figure out how it's supposed to be implemented and start unpicking the misassumptions around console scripts themselves and how PyInstaller works. Bear with me...

---

Let's start with collecting the executables:

I have managed to work around the issue by using --add-binary $(which $CONSOLESCRIPT):. and executing the binary in code with subprocess.run([os.path.join(os.path.dirname(file), $CONSOLESCRIPT]), but it is hacky and error-prone.

Those shim executables which setuptools uses to launch console scripts have absolute paths hardcoded and are therefore not relocatable. Take for example my Linux shim for pytest:

#!/home/brenainn/.pyenv/versions/3.9.6/bin/python3.9
# -*- coding: utf-8 -*-
import re
import sys
from pytest import console_main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(console_main())

The shebang at the top hardcodes a) that this is to be ran with Python and b) exactly where Python is installed. If I run this file on my machine it will launch the entrypoint from source using the Python I have installed and ignoring anything that PyInstaller built. If I ran this file on any other machine or if I uninstalled that particular version of Python I would get a file not found error. For the Windows EXE shims, it's the same story except that they're compiled so they're even harder to do any kind of use relative paths conversion on.

But I'm leading you down a bit of a rabbit hole here by talking about that absolute path to Python because, as PyInstaller does not ship the python entry point executable (e.g. /usr/bin/python or python.exe) that this hardcoded path is supposed to point to, no matter what we do with this path, it can never work because the file it needs to point to does not exist. This is why PyInstaller requires its bootloader and the correct way to add extra entry points to an application is through PyInstaller's multipackage bundles which will use a bootloader for each entry point.

---

Now to invoke them:

 # main.py
 import subprocess

 subprocess.run(["hello_world"])

This is not safe in regular Python. There is no guarantee that the current Python environment's bin or scripts directory is in PATH or that it's the first one found. This would break on Windows if you didn't tick the Add scripts directory to PATH option (which, for reasons that escape me, is the default for python.org's installers), or on any Unix system that installs Python somewhere private then symlinks only the python entry point into /usr/bin, and you're pretty much flipping a coin if you have more than one Python installed. The correct way would be to make your entry points their own submodules (and optionally have a __main__py which serves as the top level entry point):

# hello_world/hello_world/run_hello_world.py
from hello_world import hello_world
hello_world()

Then invoke them using:

subprocess.run([sys.executable, "-m", "hello_word.run_hello_world"])

This method is completely portable in that it doesn't matter how your Python environment is set up W.R.T PATH because it never uses PATH.

I'd also like to point out that adding a PyInstaller bundle's root directory to PATH is dangerous on Windows because PATH also serves as Window's shared library (DLLs) lookup path which has some nasty consequences (there is an issue showing this but I can't find it anywhere).

---

So what should you be doing instead?

1. Make all your entry points available as python -m invokable submodules and change all your subprocess.run(["entry_point_foo"]) usages to subprocess.run([sys.executable, "-m", "hello_word.entry_point_foo"]).
2. Extend those subprocess.run() calls to:

if getattr(sys, 'frozen', False):
    subprocess.run([os.path.join(sys._MEIPASS, "entry_point_foo")])
else:
    subprocess.run([sys.executable, "-m", "hello_word.entry_point_foo"])

3. Run PyInstaller on each entry point submodule (e.g. hello_world/hello_world/run_hello_world.py) and make use of Merge() to put them together in one bundle.

[rokm]

At the end of the day, even without all other issues and considerations listed above, the feature as proposed would require us to automatically freeze any entry-point script we encounter in collected packages, on the off chance that one (or some) of them will end up being used by the frozen application via subprocess... So I'm against going down that route.

[eulersIDcrisis]

Thank you for this solution here.

I have a few scripts that run in subprocesses and correctly use: python -m <module> to invoke them. However, the docs are sort of hard to follow on this point (see here ), specifically about running code that uses: python -m <module>.

I guess it isn't that hard, but the fact that os.path.join(sys._MEIPATH, <module>) actually works in this setting is not at all intuitive, IMO. This particular answer was somewhat hard to find, and there is a lot of noise about related issues that are confusing for this use-case.

I did correctly use: python -m <module> in code to invoke subprocesses directly, but can we include this particular example in the docs of pyinstaller itself? (As in, can we have a section/example/snippet that shows how python -m <module> can be made to work with a bundle?)

[bwoodsend]

I have a few scripts that run in subprocesses and correctly use: python -m to invoke them.

Do you actually need them to be subprocesses? Why not just import the scripts (wrapping their contents into function bodies if need be)?

As in, can we have a section/example/snippet that shows how python -m can be made to work with a bundle?

I would have no objection to an example being added to make doing this clearer if I had ever seen a genuine case where the subprocesses shouldn't have been removed in favour of just import-ing the top level function/class from the subprocess's entry point script or using multiprocessing. Everyone seems to want this feature for the wrong reasons.