Controlling venv from venv no longer works in 3.7.2

[schlamar]

BPO	35873
Nosy	@pfmoore, @tjguk, @schlamar, @zware, @zooba
PRs	bpo-35872 and bpo-35873: Clears __PYVENV_LAUNCHER__ variable #11745 bpo-35872 and bpo-35873: Clears __PYVENV_LAUNCHER__ variable #11745 [3.7] bpo-35872 and bpo-35873: Clears __PYVENV_LAUNCHER__ variable (GH-11745) #11753

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-02-04.15:23:13.377>
created_at = <Date 2019-02-01.09:03:04.786>
labels = ['3.7', 'type-bug', 'library', 'OS-windows']
title = 'Controlling venv from venv no longer works in 3.7.2'
updated_at = <Date 2019-02-04.15:23:13.377>
user = 'https://github.com/schlamar'

bugs.python.org fields:

activity = <Date 2019-02-04.15:23:13.377>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2019-02-04.15:23:13.377>
closer = 'steve.dower'
components = ['Library (Lib)', 'Windows']
creation = <Date 2019-02-01.09:03:04.786>
creator = 'schlamar'
dependencies = []
files = []
hgrepos = []
issue_num = 35873
keywords = ['patch', 'patch']
message_count = 4.0
messages = ['334659', '334774', '334813', '334829']
nosy_count = 5.0
nosy_names = ['paul.moore', 'tim.golden', 'schlamar', 'zach.ware', 'steve.dower']
pr_nums = ['11745', '11745', '11753']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35873'
versions = ['Python 3.7']

[schlamar]

Controlling a venv from the python.exe from another venv does not work since 3.7.2 on Windows. This is probably related to the change

bpo-34977: venv on Windows will now use a python.exe redirector rather than copying the actual binaries from the base environment.

This is obviously related to bpo-35872, but this could be a different bug.

When a Python script in a venv wants to control another venv by running commands like another-venv\python.exe -m pip with subprocess, python.exe is not referencing the other venv. It is referencing to the venv the script currently running from. This is probably because os.environ contains a 'PYVENV_LAUNCHER' which is pointing to the venv from the script.

This can be reproduced with pipx (https://github.com/pipxproject/pipx-app) by running

pipx install --python "C:\Program Files (x86)\Python37-32\python.exe" --verbose tox

This results in pip installing to venvs\pipx-app and not in venvs\tox.

I assume a simpler reproduction might be (but I cannot check this anymore as I'm back on 3.7.1 right now):

C:\Program Files (x86)\Python37-32\python.exe -m venv C:\Users\$USER\.local\pipx\venvs\tox
c:\users\$USER\.local\pipx\venvs\pipx-app\scripts\python.exe -c "import subprocess; subprocess.run(['C:\Users\$USER\.local\pipx\venvs\tox\Scripts\python.exe', '-m', 'pip', 'install', 'tox'])"

Downstream bugreport in pipx is pypa/pipx#81.

[zooba]

So with the fix for multiprocessing, we currently rely on __PYVENV_LAUNCHER__ remaining set throughout the process.

However, it may be better to add a "sys.base_executable" property instead and clear the __PYVENV_LAUNCHER__ variable once we've read it. Then we can set it again in multiprocessing and launch the base executable, and otherwise default to launching the redirector.

[zooba]

New changeset a8474d0 by Steve Dower in branch 'master':
bpo-35872 and bpo-35873: Clears __PYVENV_LAUNCHER__ variable (GH-11745)
a8474d0

[zooba]

New changeset 44467e8 by Steve Dower in branch '3.7':
bpo-35872 and bpo-35873: Clears __PYVENV_LAUNCHER__ variable (GH-11745)
44467e8