Python launcher on Windows does not detect active venv

[AlexandrosKarypidis]

BPO	38999
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba, @RossBoylan

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-12-08.18:49:03.610>
labels = ['type-bug', '3.10', 'OS-windows']
title = 'Python launcher on Windows does not detect active venv'
updated_at = <Date 2021-03-27.04:15:08.040>
user = 'https://bugs.python.org/AlexandrosKarypidis'

bugs.python.org fields:

activity = <Date 2021-03-27.04:15:08.040>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2019-12-08.18:49:03.610>
creator = 'Alexandros Karypidis'
dependencies = []
files = []
hgrepos = []
issue_num = 38999
keywords = []
message_count = 12.0
messages = ['358017', '358018', '358020', '358061', '358155', '358186', '358277', '358282', '358312', '358313', '358856', '389588']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'Alexandros Karypidis', 'rossb']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38999'
versions = ['Python 3.10']

Linked PRs

• gh-83180: Made launcher treat shebang 'python' tags as low priority so that active virtual environments are preferred #108101

[AlexandrosKarypidis]

When you activate a venv on Windows and use a shebang with a major verion qualifier, the python launcer does not properly detect that a venv is active and uses the system installation instead.

The incorrect behavior is documented in this SO question where another user has confirmed and suggested it is a bug: https://stackoverflow.com/questions/59238326

Steps to reproduce (needs script.py attached below):

1. Install Python 3.7 on Windows 10 (64 bit)

2. Run script.py you should see:
   PS C:\pytest> .\script.py
   EXECUTABLE: C:\Program Files\Python37\python.exe
   PREFIX: C:\Program Files\Python37
   BASE PREFIX: C:\Program Files\Python37

3. Create and activate a virtual environment with:
   PS C:\pytest> python -m venv .venv
   PS C:\pytest> . .\.venv\Scripts\Activate.ps1

4. Run script.py you should see it ignore the active virtual environment:
   (.venv) PS C:\pytest> .\script.py
   EXECUTABLE: C:\Program Files\Python37\python.exe
   PREFIX: C:\Program Files\Python37
   BASE PREFIX: C:\Program Files\Python37

I am using Windows 10 64-bit, update 1903 and Python 3.7.5-64

[AlexandrosKarypidis]

Forgot the simple script:

#!/usr/bin/env python3

import os, sys, platform

print('EXECUTABLE: ' + sys.executable)
print('PREFIX: ' + sys.prefix)
print('BASE PREFIX: ' + sys.base_prefix)

[AlexandrosKarypidis]

As confirmed by debug information when setting PYLAUNCH_DEBUG=1, the shebang seems to be ignored when using '#!//usr/bin/env python3' but works fine when using '#!//usr/bin/env python'.

This is with '#!//usr/bin/env python' (proper):
------------------------------------------------
(.venv) PS C:\pytest> .\script.py
launcher build: 32bit
launcher executable: Console
File 'C:\Users\karypid\AppData\Local\py.ini' non-existent
File 'C:\WINDOWS\py.ini' non-existent
Called with command line: "C:\pytest\script.py"
maybe_handle_shebang: read 167 bytes
maybe_handle_shebang: BOM not found, using UTF-8
parse_shebang: found command: python
searching PATH for python executable
Python on path: C:\pytest\.venv\Scripts\python.exe
...

This is with '#!//usr/bin/env python3' (wrong):
------------------------------------------------
(.venv) PS C:\pytest> .\script.py
launcher build: 32bit
launcher executable: Console
File 'C:\Users\karypid\AppData\Local\py.ini' non-existent
File 'C:\WINDOWS\py.ini' non-existent
Called with command line: "C:\pytest\script.py"
maybe_handle_shebang: read 168 bytes
maybe_handle_shebang: BOM not found, using UTF-8
parse_shebang: found command: python3
locating Pythons in 64bit registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
...

As you can see in the second case even though it regognises the python3 command, it makes no attempt to find it in the path.

Note that it appears that Windows installations only carry 'python.exe' and do not have a 'python3.exe' (neither in the native installation folder, nor in the virtual environment folder) so searching on the path would only 'work' if the user has copied python.exe to python3.exe in their <venv>\Scripts folder. (I actually tried that and it did not work).

A proper solution would probably need to search for 'python.exe' even for the '#!//usr/bin/env python3' shebang to detect if a virtual environment is present, possibly even confirming that the virtual environment is of the appropriate version.

[pfmoore]

See PEP-486 (https://www.python.org/dev/peps/pep-0486/). This is intended behaviour, as it is assumed that explicitly specifying a Python version means that the user had a specific Python interpreter in mind.

There are also technical issues - if the shebang says #!/usr/bin/python2, and a virtual environment is active, how do we know if it's a Python 2 or 3 environment (without launching the interpreter, which is a significant extra cost for the launcher)? It would definitely be wrong to launch a Python 3 interpreter in that case.

[AlexandrosKarypidis]

I think this situation is not ideal and the limitation seems artificial.

If I understand correctly, there are two parameters that govern the behaviour of the python launcher:

1. Is the shebang using /usr/bin/env or not?
2. IS the shebang specifying a version qualifier or not?

There are 4 combinations available:

a) env with version qualifier
b) env without version qualifier
c) non-env with version qualifier
d) non-env without version qualifier

Of the above, the python launcher supports (b) and (d).

For case (c) the launcher is configurable with py.ini

For case (a) it was decided to treat it as case (b) and search for "python.exe" in the path. To quote:

In the case of (1) I read:

"The launcher also looks for the specific shebang line #!/usr/bin/env python. On Unix, the env program searches for a command on $PATH and runs the command so located. Similarly, with this shebang line, the launcher will look for a copy of python.exe on the user's current %PATH% and will run that copy.

As activating a virtualenv means that it is added to PATH, no special handling is needed to run scripts with the active virtualenv - they just need to use the #!/usr/bin/env python shebang line, exactly as on Unix. (If there is no activated virtualenv, and no python.exe on PATH, the launcher will look for a default Python exactly as if the shebang line had said #!python)."

Why does the launcher search for "python.exe" specifically when using a version qualifier with env? This is very different to what happens on UNIX and causes the issues I reported. I see no reason why using #!/usr/bin/evn pythonX where X is a major version should not check the PATH for that specific version (e.g. #!/usr/bin/evn python3 should check for python3.exe in the PATH) and only fall back to plain "python.exe" if no match is found. This would be much closer to UNIX behaviour and allow for "common" shebangs across Windows/Linux/Darwin etc.

I would also note that if I create a python2 virtual environment and use a #!/usr/bin/env python3 shebang, a search on the path for python.exe would yield the version 2 executable from the virtual environment folder. So the argument about there being a technical issue is anyway true (the same wrong thing can happen in the current implementation).

Do you have any suggestion on how to work around this limitation using the current implementation of python launcher? Is there any way to use #!/usr/bin/env python3 across both Windows and non-Windows?

[eryksun]

I don't think the "/usr/bin/env" case needs to limit qualified names like "python3[.x]" to registered installations. This is a choice made to simplify the implementation. If it finds a generic "python.exe" executable in PATH, for 3.5+ it is possible to query version information directly from it. Older versions lack this metadata, but it's also possible to inspect the PE import table (i.e. IMAGE_DIRECTORY_ENTRY_IMPORT) for a dependency on "python2x.dll" or "python3x.dll".

[AlexandrosKarypidis]

My suggestion to gradually fall back from most specific to least specific version, was an attempt to reconcile the input from various viewpoints.

I think that ideally the behaviour should be as close to UNIX as possible. This means that it should just search the path for that binary (which is the whole point of using /usr/bin/env) and use it (even if it is the wrong version).

While I understand that one can do more (verify the version in whatever executable is located, etc) there is a lot in favor of consistency across platforms (versus even better behavior).

So my vote if the community decides to address this would be to just "search the path for <whatever>.exe when encountering #!/usr/bin/env <whatever>" and then launch it and hope for the best.

This makes it simple for users to intervene via the PATH (which is the whole point of /usr/bin/env) and launch whatever they want.

Currently, creatign a venv on windows only places "python.exe" in the Scripts folder (another difference to Linux). I would argue this is "correct" though because I think this was a decision of DISTROS (to use python for version 2 and python3 for version 3, while undertransition). Most distros still do that to this day and it will likely stick around. One day there will be only "python" and it will mean "version 3" but until then, it's better for the py launcher to do the same as on Linux and say "could not find python3 in the path" where the user can intervene (e.g. copy python.exe as python3.exe) and "fix" it manually.

Of course, doing "smarter" things on Windows is always an option but I would say that at least the "minimal" should be done to support consistency. (As I said, I would also suggest that no more than this minimal be done, but not in order to keep implementation simple, but to keep it consistent with other platforms).

Then again, that's merely one point of view from someone who is a user and does not even contribute to the project...

[zooba]

This makes it simple for users to intervene via the PATH (which is the whole point of /usr/bin/env) and launch whatever they want.

That's one point of view, but the other is that the whole point of the shebang line is for users to be able to run the script without having to use a terminal at all. And once you're there, you no longer have a customisable PATH to work with.

"Simple for users to intervene via the PATH" is not true on Windows. Modifying PATH globally can break applications or (some) system components, and modifying it temporarily requires becoming a terminal user, which is not the majority.

Personally, I think supporting the shebang line in py.exe is a misfeature and would prefer we'd never done it (though it predates my involvement). But if someone wants to implement support for detecting a venv and matching it to the shebang, I won't actively block it going in.

[eryksun]

I think supporting the shebang line in py.exe is a misfeature and
would prefer we'd never done it (though it predates my involvement).

Do you mean all shebangs, including those with a native file path?
Or do you mean just virtual shebangs, and in particular those that search PATH via "/usr/bin/env"? It's not without controversy even in Unix. For example, see [1] and the rebuttal to it in the comments.

In the case of this issue, "env" is used to run a script in an active virtual environment. This doesn't rely on the user or system PATH, assuming the command can be found in the active scripts directory.

As an alternative to modifying the launcher, pip (via distlib) could be updated to transform "#!/usr/bin/env pythonX[.Y]" shebangs into "#!/usr/bin/env python" when installing scripts in Windows. pip and distlib are third-party tools, however, so that suggestion is beyond the scope of this issue tracker.

[1] https://jmmv.dev/2016/09/env-considered-harmful.html

[zooba]

Do you mean all shebangs, including those with a native file path?

I like the idea as a whole, but ultimately end up disliking all the available options (in the context of py.exe, that is).

• Unix/virtual shebangs don't work reliably on Windows (as we can see :) )
• Native file path shebangs on Windows don't work on Unix

The scripts installed by pip &co. are their own executable wrappers, so the shebang shouldn't matter.

But, the python.org Windows installer associates .py files with py.exe, so it _does_ matter for double-click and PATHEXT scenarios.

Perhaps it is reasonable to redefine "/usr/bin/env python*" as "use %VIRTUALENV% python.exe if set" regardless of the specific version? I really don't want to add checks that require running Python - I'd rather bail out and recommend using "py" directly.

[RossBoylan]

As someone who finds the current behavior surprising, and having lost significant time because of it, I have a couple of comments.

1. If the venv created a python3 (or 2, as appropriate) file, would the expected behavior with !#/usr/bin/env python3 be restored? I think the previous analysis said that python3 doesn't match the venv because there is no python3 in the venv path. So putting one there would fix it. Of course, it would only help for newly created venvs. The pythonN file would need to be in addition to the python file, not instead of it, so that invocations of python without a number continued to work as expected.

2. There appear to be two separate sections of the documentation which govern this case. 3.7.1.2 of Python Setup and Usage says "If the launcher is run with no explicit Python version specification, and a virtual environment (created with the standard library venv module or the external virtualenv tool) active, the launcher will run the virtual environment’s interpreter rather than the global one. "
   If py myscript.py is invoked that seems to fit this case, but obviously that's not what's happening. I realize one could interpret the python3 in the shebang as an "explicit version specification", but a) that's not on the command line, which seems the more natural reading and b) it's only a partial specification, since it doesn't say which python3 to use.

Then section 3.7.2 is about !# lines. So if you invoke py without a version specifier, but the invoked file has a !#, does 3.7.1.2 or 3.2 apply?

[eryksun]

I realize one could interpret the python3 in the shebang as an
"explicit version specification", but a) that's not on the
command line, which seems the more natural reading and b) it's
only a partial specification, since it doesn't say which python3
to use.

The wanted version is specified by the command line, else by the script shebang. The default version to use for py -3 or a "python3" virtual command in a shebang (e.g. "#!/usr/bin/python3") is determined by the PY_PYTHON3 environment variable or the "python3" setting in "py.ini". If a default 3.x version isn't configured, the launcher uses the latest-version installation of 3.x, of which the 64-bit version is preferred (e.g. 3.10 is preferred over 3.10-32, which is preferred over 3.9).

For just py or a "python" virtual command in a shebang -- i.e. when no version is specified -- an active virtual environment is preferred. Else it uses the default that's set in PY_PYTHON environment variable or the "python" setting in py.ini. If no default is set, and there's no shebang, the launcher prefers the latest-version 3.x that's installed. If there's a shebang, the launcher prefers the latest-version 2.x that's installed.

Originally, the non-versioned "/usr/bin/env python" virtual command was handled the same as all other non-versioned "python" virtual commands. But later on the "env" form was changed to first try a path search for "python.exe" via WinAPI SearchPathW() before falling back on the common behavior. In most cases this search will find an active virtual environment. But it could be that another directory with "python.exe" is added to the front of PATH after a virtual environment is activated.

---

How about generalizing "/usr/bin/env python*" to support virtual environments by getting the "version" (venv) or "version_info" (virtualenv) value from "%VIRTUAL_ENV%\pyvenv.cfg"? I'd prefer to get that value in locate_venv_python(). It can still set the version string to "venv", but extend the INSTALLED_PYTHON record with a venv_version string that can be checked in the result of find_python_by_version(L"venv").

Extend locate_python() with a search parameter. If search is true, then do a path search [1] for wanted_ver, with the searched "python*.exe" executable name set depending on the launcher build (i.e. py[w][_d].exe). If searching is skipped or doesn't find a match, then run an active virtual environment if either wanted_ver is empty or search is true and wanted_ver is compatible with the virtual environment. Otherwise locate a registered Python installation based on wanted_ver and from_shebang.

---

For GUI and debug builds, the PYTHON_EXECUTABLE macro could be split into a base "python[w]" name and an extension "[_d].exe" name, which makes it easy to construct the name to search for given a valid version string. For example, with a pyw_d.exe debug build, "python3" would be searched for as "pythonw3_d.exe".

---

[1] I'd prefer to expand the path search to also check the user and machine "App Paths", which are like "$HOME/.local/bin" and "/usr/bin" for the Windows shell.

[Baljak]

This issue is not specific to virtual environments, because according to the Windows Launcher documentation, the first Python executable found in PATH should be used by the launcher instead of the default one:

The /usr/bin/env form of shebang line has one further special property. Before looking for installed Python interpreters, this form will search the executable PATH for a Python executable matching the name provided as the first argument. This corresponds to the behaviour of the Unix env program, which performs a PATH search. If an executable matching the first argument after the env command cannot be found, but the argument starts with python, it will be handled as described for the other virtual commands. The environment variable PYLAUNCHER_NO_SEARCH_PATH may be set (to any value) to skip this search of PATH.

And in fact, this is perfectly valid when putting the shebang line #!/usr/bin/env python, but not with #!/usr/bin/env python3, while it clearly should work too.

As a side note, I work on a cross-platform code that heavily relies on Python scripts for its execution and we bundle a specific version of Python with it. Because of this, we also tweak the PATH environment variable to ensure that this version is used. Now, the current issue is quite a problem since our scripts are hosted on a Git repository with the #!/usr/bin/env python3 shebang line, to make them work on Linux, but this requires any Windows user to patch these scripts manually to make them work on their OS, by replacing the shebang line with #!/usr/bin/env python. And this is not great, actually.

[zooba]

It's probably easier to copy/rename the python.exe to python3.exe than to modify the scripts.

What that section of documentation means is that it'll find an executable matching the name (python or python3 in your examples) and treat it as a Python executable. If it doesn't find anything, it falls back to the well-known templates, where python3 gets treated as py -3 which overrides the VIRTUAL_ENV environment variable.

So that either makes this a request for python3 to match files name python.exe when searching PATH (and the *w.exe and *_d.exe variations we also may have installed), or a request for py -3 to prefer VIRTUAL_ENV rather than its regular search. Either of these would be technically a breaking change, so we have to make the case for it, but it's possible that we could.

[Baljak]

It's probably easier to copy/rename the python.exe to python3.exe than to modify the scripts.

Renaming python.exe to python3.exe does not work with the Launcher provided with Python 3.10.