Python does not expand env["PATH"] properly in the subprocess module when using the env keyword.

[swaldhoer]

Bug report

Python does not expand env["PATH"] properly in the subprocess module when using the env keyword.

Consider the following examples:

import sys
import subprocess

subprocess.run([sys.executable, "-c", "import os; print(os.environ['PATH'])"])
# echos PATH

which will print your current PATH.

Now, we add an environment variable BLA=FOO and run it again

import sys
import subprocess
import os

env = os.environ.copy()
env["BLA"] = "FOO"

subprocess.run([sys.executable, "-c", "import os; print(os.environ['BLA'])"], env=env)
#                                                                             ^^^^^^^
# prints FOO

Ok, so let's update PATH to include some additional directory.

import sys
import subprocess
import os

env = os.environ.copy()
env["PATH"] = env["PATH"] + r"C:\Program Files\WindowsApps\Microsoft.PowerShell_7.3.3.0_x64__8wekyb3d8bbwe"

subprocess.run([sys.executable, "-c", "import os; print(os.environ['PATH'])"], env=env)
#                                                                             ^^^^^^^
# prints old PATH plus the path to PowerShell.

Ok, so let's now let's check shutil.which()s opinion on that.

import sys
import subprocess
import os

subprocess.run([sys.executable, "-c", "import shutil; print(shutil.which('pwsh'))"])
# prints None

env = os.environ.copy()
env["PATH"] = env["PATH"] + r"C:\Program Files\WindowsApps\Microsoft.PowerShell_7.3.3.0_x64__8wekyb3d8bbwe"

subprocess.run([sys.executable, "-c", "import shutil; print(shutil.which('pwsh'))"], env=env)
#                                                                                    ^^^^^^^
# prints old PATH plus the path to PowerShell.

So far so good. Let's create a non-Python subprocess with an updated env, and see what happens:

import sys
import subprocess
import os

env = os.environ.copy()
env["PATH"] = env["PATH"] + r"C:\Program Files\WindowsApps\Microsoft.PowerShell_7.3.3.0_x64__8wekyb3d8bbwe"

subprocess.run(["pwsh", "--version"], env=env)
#                                     ^^^^^^^
# FileNotFoundError: [WinError 2] The system cannot find the file specified

The created subprocess does not respect the provided key PATH in the env dictionary for the env argument in the last example, but respects it in the second and third last example.
This behavior is not documented. So this seems to me like a bug.

Of course, if setting shell=True then the last example also works, as the environment variables are then expanded. But the last example should work without the need for shell=True.

Your environment

• CPython versions tested on: Python 3.10.10
• Operating system and architecture: Windows, AMD64

[eryksun]

On Windows, subprocess.Popen lets WinAPI CreateProcessW() search for the executable that's passed in the command line. The system implements this internally by calling WinAPI SearchPathW() with a search path that includes the application directory, possibly the current working directory(i.e. "."), system directories, and the directories in the PATH environment variable. The value of the current directory and PATH come from the current process, not from the lpCurrentDirectory and lpEnvironment parameters that are passed to CreateProcessW(). In the documentation of the Popen constructor, there's the following warning regarding the difference in behavior between POSIX and Windows.

For maximum reliability, use a fully qualified path for the executable. To search for an unqualified name on PATH, use shutil.which(). On all platforms, passing sys.executable is the recommended way to launch the current Python interpreter again, and use the -m command-line format to launch an installed module.

Resolving the path of executable (or the first item of args) is platform dependent. For POSIX, see os.execvpe(), and note that when resolving or searching for the executable path, cwd overrides the current working directory and env can override the PATH environment variable. For Windows, see the documentation of the lpApplicationName and lpCommandLine parameters of WinAPI CreateProcess, and note that when resolving or searching for the executable path with shell=False, cwd does not override the current working directory and env cannot override the PATH environment variable. Using a full path avoids all of these variations.

[eryksun]

Note that the value of PATH usually doesn't end in os.pathsep (semicolon). A separator should be appended before each appended directory. For example:

env = os.environ.copy()
pwsh_path = r"C:\Program Files\WindowsApps\Microsoft.PowerShell_7.3.3.0_x64__8wekyb3d8bbwe"
env["PATH"] = fr"{env['PATH']}{os.pathsep}{pwsh_path}"

---

Off topic comment

While directly executing r"C:\Program Files\WindowsApps\Microsoft.PowerShell_7.3.3.0_x64__8wekyb3d8bbwe\pwsh.exe" happens to work in this case, even without enabling the "pwsh.exe" app execution alias, it's an unusual way to execute an app. The app files under "%ProgramFiles%\WindowsApps" aren't directly executable by standard users or even administrators (i.e. internally the initial NtCreateUserProcess() system call fails with access denied), so CreateProcessW() has to create a new access token that's allowed to execute the file. However, first the system has to verify that the current user is licensed to use the app. In my experience, it doesn't always reliably determine the app package information from just the path of the binary, in which case the execution over all will fail with access denied. I guess this could be just the case for desktop bridge apps. In my experience, getting the app package data is always reliable if the system can read the information from the same-named app execution alias in "%LocalAppData%\Microsoft\WindowsApps". Of course, if the appexec alias is enabled already, there's really no reason to execute the real path of the binary under "%ProgramFiles%\WindowsApps". Just execute the alias.

[swaldhoer]

First, thanks for the fast, detailed and good answer!

On Windows, subprocess.Popen lets WinAPI CreateProcessW() search for the executable that's passed in the command line. The system implements this internally by calling WinAPI SearchPathW() with a search path that includes the application directory, possibly the current working directory(i.e. "."), system directories, and the directories in the PATH environment variable. The value of the current directory and PATH come from the current process, not from the lpCurrentDirectory and lpEnvironment parameters that are passed to CreateProcessW(). In the documentation of the Popen constructor, there's the following warning regarding the difference in behavior between POSIX and Windows.

To make sure I fully got your answer:

What I want to achieve (i.e., the subprocess knowing the updated value for PATH) is not possible without passing shell=True (obvious why it works then) and this is expected platform behavior on Windows.

Seems a bad question based on your answer and the Python docs you linked, but still: Is there a solution/workaround you could think of to make that possible what I want to achieve without using shell=True? Why do I need that? I cannot only provide the full path to executable I need to run, as the executable that is run will require to find other tools that are only the updated PATH environment variable.
The constraint here is, that I cannot pollute the PATH prior to running Python. That's why it is like it is.

Thanks for the explanation on the WindowsApps directory.

[eryksun]

What I want to achieve (i.e., the subprocess knowing the updated value for PATH) is not possible without passing shell=True (obvious why it works then) and this is expected platform behavior on Windows.

If you modify the value of PATH in the env mapping, the child process will have the modified value. But FileNotFoundError is raised here because CreateProcessW(), which is called in the current process, fails with ERROR_FILE_NOT_FOUND (2). CreateProcessW() searches for the "pwsh.exe" executable (based on the command line "pwsh --version") in the current application directory (e.g. the directory that contains "python.exe"), maybe the current working directory (e.g. if the environment variable NoDefaultCurrentDirectoryInExePath is not defined), the current system directories (e.g. "%SystemRoot%\SysWOW64" if it's a 32-bit app running on 64-bit Windows), and the directories in the current value of the PATH environment variable.

The difference compared to POSIX is partially due to the fact that creating a child process is fundamentally different on a platform such as Windows that doesn't use a combination of fork() followed by post-fork operations in the child process that prepare to call exec*(). For example, POSIX execvp() and execlp() search for a simple relative filename (i.e. no slash in the name) using the current value of PATH. If the name contains a slash, it's resolved relative to the current working directory. However, given fork() is called to create a child process, this search can occur (and usually does occur) after the current working directory and value of PATH have been modified in the child.

That said, Microsoft could have implemented CreateProcessW() to search for the executable based on the provided values of lpCurrentDirectory and lpEnvironment. That would have been more consistent for the child process. For example, consider a command line that executes "bin\spam.exe". As is, the executable path is resolved relative to the current working directory of the current process. This means that the value "bin\spam.exe" in the command line will be meaningless in the child process if its initial working directory is set to something else via the lpCurrentDirectory parameter. If CreateProcessW() instead resolved "bin\spam.exe" against the value of lpCurrentDirectory, then the command line in the child process would be consistent with its initial working directory. The potential for nonsense here is Microsoft's design mistake. You can avoid the ambiguity by using a fully-qualified path in the command line, found via shutil.which().

[swaldhoer]

Ah now I think I get the full picture.

So what I basically need to do is:

import os
import shutil
import subprocess
new_env = os.environ.copy()
path_1 = r"C:\foo" # contains a "a.exe"
path_2 = r"C:\bla" # contains a "b.exe"; "a.exe" also needs to know about "b.exe" when run
new_env["PATH"] = fr"{env['PATH']}{os.pathsep}{path_1}{os.pathsep}{path_2}"
a_exe = shutil.which("a", path=new_env["PATH"])

# the subprocess can now work, as we provide an absolute path to the executable
# to run, and then the started a_exe process is able to find b.exe on its path
# provided through the 'env'.
subprocess.run([a_exe,"some", "args"], env=new_env)

Is my understanding correct?

And, I need to say it again, thanks for your answers, so detailed, so good.
Really enjoyable to learn! Thanks.

[eryksun]

Yes, search for the executable using shutil.which() with the updated value of PATH from the new environment.

Note that on Windows shutil.which() tries the extensions listed in the PATHEXT environment variable, and ".COM" is typically listed before ".EXE", so "a.com" may be found before "a.exe", if it exists. You could search for "a.exe" instead.