Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems

[maggyero]

BPO	46720
Nosy	@maggyero
PRs	bpo-46720: Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems #31279

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-02-11.14:57:09.587>
labels = ['type-feature', 'library', '3.9', '3.10', '3.11']
title = 'Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems'
updated_at = <Date 2022-02-11.22:25:00.201>
user = 'https://github.com/maggyero'

bugs.python.org fields:

activity = <Date 2022-02-11.22:25:00.201>
actor = 'maggyero'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-02-11.14:57:09.587>
creator = 'maggyero'
dependencies = []
files = []
hgrepos = []
issue_num = 46720
keywords = ['patch']
message_count = 1.0
messages = ['413073']
nosy_count = 1.0
nosy_names = ['maggyero']
pr_nums = ['31279']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue46720'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-90876: Restore the ability to import multiprocessing when sys.executable is None #106464
• [3.12] gh-90876: Restore the ability to import multiprocessing when sys.executable is None (GH-106464) #106494
• [3.11] gh-90876: Restore the ability to import multiprocessing when sys.executable is None (GH-106464) #106495

[maggyero]

Any path-like object can be passed to multiprocessing.set_executable, i.e. objects with str, bytes, or os.PathLike type.

For instance these work (tested on MacOS with all start methods: ‘spawn’, ‘fork’, and ‘forkserver’):

• multiprocessing.set_executable(sys.executable) (str);
• multiprocessing.set_executable(sys.executable.encode()) (bytes);
• multiprocessing.set_executable(pathlib.Path(sys.executable)) (os.PathLike).

This is because the ‘fork’ start method does not exec any program in the subprocess, the ‘spawn’ start method converts its path argument to bytes with os.fsencode before passing to _posixsubprocess.fork_exec, and the ‘forkserver’ start method spawns a server process (like with the ‘spawn’ start method) which then forks itself at each request (like the ‘fork’ start method):

        return _posixsubprocess.fork_exec(
            args, [os.fsencode(path)], True, passfds, None, None,
            -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
            False, False, None, None, None, -1, None)

Linux (and other Unix-like systems) uses the same code than MacOS for the three start methods so it should work for it too.

However I have not tested this on Windows which uses the function _winapi.CreateProcess for the ‘spawn’ start method (the only start method available on this OS) but I noticed that no conversion to str (not to bytes this time, since the function expects str) of the path argument with os.fsdecode (not os.fsencode this time) is performed before passing it to the function:

                hp, ht, pid, tid = _winapi.CreateProcess(
                    python_exe, cmd,
                    None, None, False, 0, env, None, None)

So on Windows only str path can be passed to multiprocessing.set_executable. This PR fixes this to be on a par with Unix-like systems which accept any path-like objects.

[brettcannon]

Closed via #31279 .