Version numbers not supported for shebang line virtual command /usr/bin/env python on windows

[jacacthomas]

Documentation

In section 4.8.2 of the docs (Shebang Lines), it is stated that: "Any of the above virtual commands can be suffixed with an explicit version (either just the major version, or the major and minor version)."

This sentence refers to the 4 virtual commands: /usr/bin/env, /usr/bin/python, /usr/local/bin/python, python.

This is correct for the last 3 virtual commands (e.g. /usr/bin/python3.12 works), but not for /usr/bin/env.

Specifically, using either the major version (e.g. "/usr/bin/env python3") or the major and minor version (e.g. "/usr/bin/env python3.12") in the shebang line of a .py file, then calling the .py with the windows launcher, results in the following error: "Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases."

Simple solution: change the relevant sentence in the docs to exclude /usr/bin/env: "Any of the above virtual commands (apart from /usr/bin/env) can be suffixed with an explicit version (either just the major version, or the major and minor version)."

Linked PRs

• gh-119070: Fix py.exe handling of /usr/bin/env commands missing extension #119426
• [3.13] gh-119070: Fix py.exe handling of /usr/bin/env commands missing extension (GH-119426) #119739
• [3.12] gh-119070: Fix py.exe handling of /usr/bin/env commands missing extension (GH-119426) #119741
• [3.12] gh-119070: Avoid test crash due to Unicode in stderr output #119747
• gh-119070: Update test_shebang_executable_extension to always use non-installed version #119846
• [3.13] gh-119070: Update test_shebang_executable_extension to always use non-installed version (GH-119846) #120015
• [3.12] gh-119070: Update test_shebang_executable_extension to always use non-installed version (GH-119846) #120016

[zooba]

It looks like it decided to launch the launcher stub rather than a regular Python install, which probably means you haven't got the latest version of the launcher (which will avoid doing this).

Disabling the stub from Settings > Manage App Execution Aliases should work, as then the /usr/bin/env python... format won't find a python3.exe on your PATH, or if you install Python 3.12.3 or 3.13.0b1 you will also get the newest launcher that deliberately ignores the one that Windows includes by default.

[eryksun]

In the original implementation of the launcher, "python*" commands in "/usr/bin/env" shebangs were special cased to only search registered Python installations. The new implementation of the launcher always searches PATH first, even for "python*" commands.

Generally that's the right behavior, but it does need to skip Microsoft's generic "python*" app links that run "Microsoft.DesktopAppInstaller". This was fixed back in February for Python 3.11.8+ and 3.12.2+. In PowerShell, you can check the version of the launcher using (py --help)[0]. In CMD, use py --help | more and check the first line of the output.

[jacacthomas]

Update: this problem no longer occurs on my machine (i.e. it works as the docs described).

However I'm confused as to why, since I didn't disable the stub as @zooba suggests, and the launcher version is 3.12.1. All I've done since posting earlier is clone the Python repo and install Visual Studio (I was intending to implement the doc change). It's a mystery.

Thanks both for your comments - I'm happy for this issue to be closed now.

[eryksun]

Specifically, using either the major version (e.g. "/usr/bin/env python3") or the major and minor version (e.g. "/usr/bin/env python3.12")

There is no "python3.12" appexec link for the app installer. Microsoft only creates links by default for "python.exe" and "python3.exe". The reason it's finding "python3.exe" is a bug in the launcher's searchPath() function that handles the ".12" file 'extension' in the shebang command. It gets the base name "python3"; compares the ".12" extension to ".exe"; and since it's not ".exe", appends ".exe" to the base name to search for "python3.exe". If the extension isn't ".exe", it should append ".exe" to the original name, not the base name, in order to search for "python3.12.exe".

[zooba]

The reason it's finding "python3.exe" is a bug in the launcher's searchPath() function that handles the ".12" file 'extension' in the shebang command.

Not us:

    const wchar_t *ext = L".exe";
    // If the command already has an extension, we do not want to add it again
    if (!lastDot || _comparePath(&filename[lastDot], -1, ext, -1)) {
        if (wcscat_s(filename, MAXLEN, L".exe")) {
            ....

I could imagine SearchPathW doing this though, but we should have already added .exe to the end.

My guess is that it's working now with /usr/bin/env python3.12, which has no stub to be distracted by, and if that one failed earlier it was a different reason from /usr/bin/env python3 (which will have been distracted by the stub).

[eryksun]

I just stepped through the code in the debugger before writing that post. It's a bug in the launcher. The contents of filename are set by wcsncpy_s(filename, MAXLEN, command, lastDot), where lastDot is the count up to the last dot in the command. So filename does not include the ".12" extension when _comparePath() is called, and filename[lastDot] is the null character. It ends up appending ".exe" to the base name "python3".

[zooba]

The contents of filename are set by wcsncpy_s(filename, MAXLEN, command, lastDot), where lastDot is the count up to the last dot in the command.

Okay, yeah, that's a bug. That means the _comparePath() call has never done anything useful.

Have I ever mentioned how much I wish we didn't try to support these shebang templates 🙃

[encukou]

test_launcher now fails on Windows buildbots, e.g. here & here:

======================================================================
FAIL: test_shebang_executable_extension (test.test_launcher.TestLauncher.test_shebang_executable_extension)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\test_launcher.py", line 770, in test_shebang_executable_extension
    data = self.run_py([script])
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\test_launcher.py", line 254, in run_py
    self.assertEqual(expect_returncode, p.returncode)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 0 != 103
Stdout:
++ COMMAND ++
[WindowsPath('D:/buildarea/3.x.bolen-windows10/build/PCbuild/amd64/py_d.exe'), WindowsPath('D:/buildarea/3.x.bolen-windows10/build/build/test_python_1300�/tmpn00o3m8y.py')]
++ STDOUT ++
++ STDERR ++
argv0: D:\buildarea\3.x.bolen-windows10\build\PCbuild\amd64\py_d.exe
version: 3.14.0a0
# Read 26 bytes from D:\buildarea\3.x.bolen-windows10\build\build\test_python_1300?\tmpn00o3m8y.py to find shebang line
Shebang: /usr/bin/env python3.12
# Search PATH for python3.12.exe
# Did not find python3.12.exe on PATH
# Reading from C:\Users\db3l\AppData\Local\py.ini for commands/python3.12
# Did not find file C:\Users\db3l\AppData\Local\py.ini
# Reading from D:\buildarea\3.x.bolen-windows10\build\PCbuild\amd64\py.ini for commands/python3.12
# Did not find file D:\buildarea\3.x.bolen-windows10\build\PCbuild\amd64\py.ini
# Treating shebang command '3.12' as 'py -3.12'
# Cannot select defaults for tag '3.12'
SearchInfo.originalCmdLine: D:\buildarea\3.x.bolen-windows10\build\PCbuild\amd64\py_d.exe D:\buildarea\3.x.bolen-windows10\build\build\test_python_1300?\tmpn00o3m8y.py
SearchInfo.restOfCmdLine:  D:\buildarea\3.x.bolen-windows10\build\build\test_python_1300?\tmpn00o3m8y.py
SearchInfo.executablePath: (null)
SearchInfo.scriptFile: D:\buildarea\3.x.bolen-windows10\build\build\test_python_1300?\tmpn00o3m8y.py
SearchInfo.executable: python_d.exe
SearchInfo.executableArgs: 
SearchInfo.company: (null)
SearchInfo.tag: 3.12
SearchInfo.oldStyleTag: True
SearchInfo.lowPriorityTag: True
SearchInfo.allowDefaults: True
SearchInfo.allowExecutableOverride: True
SearchInfo.windowed: False
SearchInfo.list: False
SearchInfo.listPaths: False
SearchInfo.help: False
SearchInfo.limitToCompany: (null)
# PythonTestSuite/ignored has no executable path
 -V:3.11          C:\Python\3.11\python.exe
 -V:2.7           C:\Python\2.7\python_d.exe
 -V:PythonTestSuite/3.100 X.Y.exe
 -V:PythonTestSuite/3.100-32 X.Y-32.exe
 -V:PythonTestSuite/3.100-arm64 X.Y-arm64.exe -X fake_arg_for_test
 -V:PythonTestSuite1/3.100 D:\buildarea\3.x.bolen-windows10\build\PCbuild\amd64\python_d.exe
No suitable Python runtime found
Pass --list (-0) to see all detected environments on your machine
or set environment variable PYLAUNCHER_ALLOW_INSTALL to use winget
or open the Microsoft Store to the requested version.

[zooba]

On it. We should ignore the exit code for this test - there's a single debug message we're looking for.

[zooba]

See #119846

[encukou]

Thanks for the fix!