Windows Multiprocessing in Virtualenv: sys.prefix is incorrect

[ameily]

BPO	39439
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba, @ameily, @miss-islington
PRs	[3.7] bpo-38092: Reduce overhead when using multiprocessing in a Windows virtual environment (GH-16098) #18157 bpo-39439: Fix Multiprocessing Python Path in Windows Virtualenv #18158 [3.8] bpo-39439: Fix Multiprocessing Python Path in Windows Virtualenv (GH-18158) #18159 [3.7] bpo-39439: Fix multiprocessing spawn path in a venv on Windows (GH-18158) #18237 [3.8] bpo-39439: Fix multiprocessing spawn path in a venv on Windows (GH-18158) #18238
Superseder	bpo-38092: environment variables not passed correctly using new virtualenv launching in windows and python3.7+
Files	multiproc_venv_prefix.py: Example script that demonstrates the behavior

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-01-23.22:39:03.702>
created_at = <Date 2020-01-23.21:04:50.207>
labels = ['3.8', 'type-bug', '3.7', '3.9', 'OS-windows']
title = 'Windows Multiprocessing in Virtualenv: sys.prefix is incorrect'
updated_at = <Date 2020-01-28.10:52:50.803>
user = 'https://github.com/ameily'

bugs.python.org fields:

activity = <Date 2020-01-28.10:52:50.803>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2020-01-23.22:39:03.702>
closer = 'eryksun'
components = ['Windows']
creation = <Date 2020-01-23.21:04:50.207>
creator = 'meilyadam'
dependencies = []
files = ['48862']
hgrepos = []
issue_num = 39439
keywords = []
message_count = 6.0
messages = ['360581', '360582', '360589', '360590', '360862', '360864']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'meilyadam', 'miss-islington']
pr_nums = ['18157', '18158', '18159', '18237', '18238']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '38092'
type = 'behavior'
url = 'https://bugs.python.org/issue39439'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[ameily]

I upgraded from Python 3.7.1 to 3.7.6 and began noticing a behavior that was breaking my code. My code detects if it's running in a virtualenv. This check worked in 3.7.1 but is broken in 3.7.6.

From the documentation, sys.prefix and sys.exec_prefix should point to the virtualenv when one is active. However, I'm seeing that both of these constants are pointing to the system installation directory and not my virtualenv when I am in a multiprocessing child. Here is an example output of a test application running in 3.7.6 (I've attached the test script to this ticket):

Parent process
=============================================
sys.prefix: C:\Users\user\project\venv
sys.exec_prefix: C:\Users\user\project\venv
sys.base_prefix: C:\Program Files\Python37
sys.base_exec_prefix: C:\Program Files\Python37
=============================================

Subprocess
=============================================
sys.prefix: C:\Program Files\Python37
sys.exec_prefix: C:\Program Files\Python37
sys.base_prefix: C:\Program Files\Python37
sys.base_exec_prefix: C:\Program Files\Python37
=============================================

I would expect that sys.prefix and sys.exec_prefix to be identical in the parent and child process.

I verified that this behavior is present in 3.7.5, 3.7.6, and 3.8.1. I am on a Windows 10 x64 system.

Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)] on win32

[eryksun]

This is a duplicate of bpo-38092. The required change to fix this is very simple, if you want to work on a PR.

[ameily]

OK, that makes sense.

For 3.7, I can create a PR for that corrects the order of arguments passed into _winapi.CreateProcess

For 3.8 / master, the problem appears to be that the check in popen_spawn_win32.py to set the subprocess env is failing because sys.executable != spawn.get_executable() -- spawn.get_executable() is returning sys._base_executable. So, can you confirm that the fix is to just change spawn.get_executable() to return sys.executable, like it was prior to the PR mentioned in the other ticket?

[eryksun]

can you confirm that the fix is to just change spawn.get_executable()
to return sys.executable, like it was prior to the PR mentioned in
the other ticket?

Yes, in spawn.py, change _python_exe = sys._base_executable to _python_exe = sys.executable. This is the value that gets returned by spawn.get_executable().

This will restore the operation of the following code in Popen.__init__ in popen_spawn_win32.py:

        python_exe = spawn.get_executable()

        # bpo-35797: When running in a venv, we bypass the redirect
        # executor and launch our base Python.
        if WINENV and _path_eq(python_exe, sys.executable):
            python_exe = sys._base_executable
            env = os.environ.copy()
            env["__PYVENV_LAUNCHER__"] = sys.executable

This code switches to running the base executable in a virtual environment, which is required in order to support duplicating handles to the child or stealing handles from the parent. And the startup code in the child process is made aware of virtual environment via the "__PYVENV_LAUNCHER__" environment variable.

[zooba]

New changeset 0be3246 by Steve Dower (Adam Meily) in branch 'master':
bpo-39439: Fix multiprocessing spawn path in a venv on Windows (GH-18158)
0be3246

[miss-islington]

New changeset dbb37aa by Miss Islington (bot) in branch '3.8':
bpo-39439: Fix multiprocessing spawn path in a venv on Windows (GH-18158)
dbb37aa