Recipe subprocess

FkinGuy edited this page Aug 26, 2017 · 10 revisions

Calling subprocess.Popen or one its its variants requires care when using Pyinstaller; issue 1339 gives an example of these difficulties. The following function provides workarounds for several common problems in Windows:

  • subprocess.Popen will pop up a command window by default when run from Pyinstaller with the --noconsole option.
  • Windows doesn't search the path by default.
  • Running this from the binary produced by Pyinstaller with the --noconsole option requires redirecting everything (stdin, stdout, stderr) to avoid an OSError exception: "[Error 6] the handle is invalid."

This code was based on an Enki library.

import subprocess
import os, os.path
import sys

# Create a set of arguments which make a ``subprocess.Popen`` (and
# variants) call work with or without Pyinstaller, ``--noconsole`` or
# not, on Windows and Linux. Typical use::
#
#   subprocess.call(['program_to_run', 'arg_1'], **subprocess_args())
#
# When calling ``check_output``::
#
#   subprocess.check_output(['program_to_run', 'arg_1'],
#                           **subprocess_args(False))
def subprocess_args(include_stdout=True):
    # The following is true only on Windows.
    if hasattr(subprocess, 'STARTUPINFO'):
        # On Windows, subprocess calls will pop up a command window by default
        # when run from Pyinstaller with the ``--noconsole`` option. Avoid this
        # distraction.
        si = subprocess.STARTUPINFO()
        si.dwFlags |= subprocess.STARTF_USESHOWWINDOW
        # Windows doesn't search the path by default. Pass it an environment so
        # it will.
        env = os.environ
    else:
        si = None
        env = None

    # ``subprocess.check_output`` doesn't allow specifying ``stdout``::
    #
    #   Traceback (most recent call last):
    #     File "test_subprocess.py", line 58, in <module>
    #       **subprocess_args(stdout=None))
    #     File "C:\Python27\lib\subprocess.py", line 567, in check_output
    #       raise ValueError('stdout argument not allowed, it will be overridden.')
    #   ValueError: stdout argument not allowed, it will be overridden.
    #
    # So, add it only if it's needed.
    if include_stdout:
        ret = {'stdout': subprocess.PIPE}
    else:
        ret = {}

    # On Windows, running this from the binary produced by Pyinstaller
    # with the ``--noconsole`` option requires redirecting everything
    # (stdin, stdout, stderr) to avoid an OSError exception
    # "[Error 6] the handle is invalid."
    ret.update({'stdin': subprocess.PIPE,
                'stderr': subprocess.PIPE,
                'startupinfo': si,
                'env': env })
    return ret

# A simple test routine. Compare this output when run by Python, Pyinstaller,
# and Pyinstaller ``--noconsole``.
if __name__ == '__main__':
    # Save the output from invoking Python to a text file. This is the only
    # option when running with ``--noconsole``.
    with open('out.txt', 'w') as f:
        try:
            txt = subprocess.check_output(['python', '--help'],
                                          **subprocess_args(False))
            f.write(txt)
        except OSError as e:
            f.write('Failed: ' + str(e))

Exes created by PyInstaller on Windows let a lock lingering when terminating after creating a child process

There a behavior of exes produced with PyInstaller, specifically on Windows: the subprocess launched by subprocess.Popen inherits open filehandles from the parent, and this includes the filehandle opened for the parent's own executable file.

What it means, is that if in your python code you want to create a new process and then terminate the parent, the parent doesn't disappear completely: there is still a lock remaining on the exe of the parent. One of the consequences is that you can't delete/modify the parent exe from the child's process (like, you can't use a child exe to update the first one).

On Linux and Darwin you can get away with a simple call to subprocess.Popen. On windows you need to add the option close_fds=True in your subprocess.Popen call:

import subprocess

subprocess.Popen("myexecutable.exe", close_fds=True)
""" Now my python code can exit, and no nothing of itself will remain lingering """
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.