[doc] subprocess security considerations needs a Windows-specific exception

[zooba]

The documentation at https://docs.python.org/3/library/subprocess.html#security-considerations says that "this implementation will never implicitly call a system shell".

While this is technically true, on Windows the underlying CreateProcess API may create a system shell, which then exposes arguments to shell parsing. This happens when passed a .bat or .cmd file.

PSRT review of the issue determined that we can't safely detect and handle this situation without causing new issues and making it more complex for users to work around when they want to intentionally launch a batch file without shell processing. For the two cases of untrusted input, an untrusted application/argv[0] is already vulnerable, and an untrusted argument/argv[1:] is safe provided argv[0] is controlled. However, we do need to inform developers of the inconsistency so they can check their own use.

We'll use this issue to ensure we get good wording. First proposal in the next comment.

Thanks to RyotaK for reporting responsibly to the Python Security Response Team.

Linked PRs

• gh-114539: Clarify implicit launching of shells by subprocess #117996
• [3.12] gh-114539: Clarify implicit launching of shells by subprocess (GH-117996) #118002
• [3.11] gh-114539: Clarify implicit launching of shells by subprocess (GH-117996) #118003
• [3.10] gh-114539: Clarify implicit launching of shells by subprocess (GH-117996) #118004
• [3.9] gh-114539: Clarify implicit launching of shells by subprocess (GH-117996) #118005
• [3.8] gh-114539: Clarify implicit launching of shells by subprocess (GH-117996) #118006

[zooba]

Starting proposal:

On Windows, batch files (*.bat or *.cmd) may be launched by the operating system in a system shell. This could result in arguments being parsed according to shell rules, but without any escaping added by Python. If you are intentionally launching a batch file with arguments from untrusted sources, consider passing shell=True to allow Python to escape special characters.

[terryjreedy]

Is the situation really as vague as 'may be' (unless forced by shell=True? Is it repeatable (deterministic) for the same call?

[Ry0taK]

@zooba Thanks for opening the issue!

Regarding the wording, using shell=True doesn't escape arguments properly.

subprocess.run([".\\test.bat", "&calc.exe"], shell=True)

The above snippet will spawn the following process, which still executes calc.exe:

C:\WINDOWS\system32\cmd.exe /c ".\test.bat &calc.exe"

@terryjreedy It's always launched using cmd.exe because batch files can't be parsed without it

[zooba]

Is the situation really as vague as 'may be' (unless forced by shell=True? Is it repeatable (deterministic) for the same call?

If we assume that the file system can change at any time, then it's not deterministic, even for the same call. But assuming nobody modifies the file you are executing, it should always use the same mechanism.

I do know more details about when/why it might vary, but we don't want to document them. There's also always the potential that Microsoft could change the behaviour and/or add an option to change the behaviour, and we don't want our docs to be invalidated by that. The main aim is to encourage users to pass shell=True if they know they're launching a batch file, and to be clear that we won't escape anything if they launch a batch file with shell=False.

Regarding the wording, using shell=True doesn't escape arguments properly.

I know we've discussed this at various points over the years... it ought to be a long-standing bug if it's not working right now. But we might need to make sure we're correctly handling command characters (and compatibly across platforms).

[zooba]

I do know more details about when/why it might vary, but we don't want to document them.

I guess I can add that the main issue is that a .bat or .cmd extension doesn't guarantee that it's actually a batch file, and that factors into how it may be run. If the extension is one of those and it's actually a batch file with no deliberate tricks, it'll always be run under cmd.exe.

[gpshead]

Yep. IIRC Windows and related software has had a notorious number of security bugs related to unfortunate magic file type detection based on contents rather than what the name, extension, or mime type says... We can't prevent a fundamental platform issue on the Python side.

[zooba]

In defence of this case, by passing the path to CreateProcess you're kind of implying that you think it's an executable (akin to CPython's concrete C API). Do the same thing with other APIs and it'll use different tests (e.g. ShellExecute is more like our abstract C API).

But it's the fallback behaviour in this (and other) cases that cause problems. Which might also be why I'm instinctively against adding "abstract" behaviour to our concrete C APIs... 🤔

[gpshead]

FYI - here's what Rust did implementation wise: rust-lang/rust@f66a096

For the few languages that chose to attempt to prevent users from getting into this Windows messy area, I believe they're landing those changes in public now-ish. It'll be interesting to watch how those attempts fare in practice.

[zooba]

Thanks Greg. We should probably get our doc update in.

As a reminder, this is my proposed addition, and there have been no changes yet:

On Windows, batch files (*.bat or *.cmd) may be launched by the operating system in a system shell. This could result in arguments being parsed according to shell rules, but without any escaping added by Python. If you are intentionally launching a batch file with arguments from untrusted sources, consider passing shell=True to allow Python to escape special characters.

[gpshead]

I believe the coordinated disclosure and CVEs for various languages just landed. The original reporter has a public blog post now: https://flatt.tech/research/posts/batbadbut-you-cant-securely-execute-commands-on-windows/ Link to that from the doc note if deemed appropriate, or at least back to this GH issue so people who do want mitigation within Python have a singular place to chime in.

[gpshead]

Rust at least happily mentions that their mitigations cannot handle all cases: https://blog.rust-lang.org/2024/04/09/cve-2024-24576.html#mitigations

[zooba]

Main change is committed to active branches - security fix branches will be merged by the RMs. There's no further need to keep this open.

If issues in shell quoting are discovered (they will always exist, FWIW), please open a new issue.