shutil.which doesn't find files without PATHEXT extension on Windows

[rmccampbell7]

BPO	31405
Nosy	@pfmoore, @tjguk, @zware, @SpecLad, @serhiy-storchaka, @eryksun, @zooba, @asottile, @Zheaoli, @tirkarthi

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-09-09.18:12:09.040>
labels = ['3.8', 'type-bug', 'library', 'OS-windows']
title = "shutil.which doesn't find files without PATHEXT extension on Windows"
updated_at = <Date 2021-01-25.22:47:48.848>
user = 'https://bugs.python.org/rmccampbell7'

bugs.python.org fields:

activity = <Date 2021-01-25.22:47:48.848>
actor = 'mitchelly'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2017-09-09.18:12:09.040>
creator = 'rmccampbell7'
dependencies = []
files = []
hgrepos = []
issue_num = 31405
keywords = []
message_count = 12.0
messages = ['301785', '326497', '326498', '326500', '326501', '326502', '326506', '326514', '360439', '360482', '360485', '385676']
nosy_count = 12.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'SpecLad', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'rmccampbell7', 'Anthony Sottile', 'Manjusaka', 'xtreak', 'mitchelly']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31405'
versions = ['Python 3.8']

Linked PRs

• GH-75586: Fix case where PATHEXT isn't applied to items in PATH (Windows) #103179

[rmccampbell7]

On windows, shutil.which does not match the semantics of built-in command lookup. If you pass the name of a script like foo.py and the PATHEXT variable doesn't include .py it will search for foo.py.exe, foo.py.bat, foo.py.cmd, etc. but not foo.py, which should be the first name checked.

[tirkarthi]

I am adding windows as a component during triaging since PATHEXT seems to be windows specific. Unfortunately, I couldn't verify this since I don't have windows system to check this against master so leaving it to 3.6.

Thanks

[serhiy-storchaka]

Isn't this an expected behavior?

[pfmoore]

I don't think this is expected behaviour. It's not documented what should happen in this case but the behaviour suggested by the OP (to search for the path as given, followed by [path+e for e in os.environ['PATHEXT'].split(os.pathsep)] seems reasonable to me.

The details and corner cases need thrashing out, which I don't have time to do right now, but the principle seems sound.

[pfmoore]

On further reflection, I'm less sure that the proposed behaviour is the best option, but I do think this warrants further consideration.

[rmccampbell7]

This is how windows looks up commands, as well as the built in "where" command. (Note that windows doesn't actually distinguish between "executable" files and just plain old files, so this could be confusing for UNIX users... a text file for instance will simply open in the default text editor when "executed" by the shell.)

[zooba]

It certainly doesn't match "which" semantics, but given the F_OK and X_OK flags I can see cases where it ought not to. I'm not sure it does what it implies for those either though.

I can see uses for "find files according to 'which'" and "find executable files according to 'which'".

Either way, I don't see an easy way to change the semantics any earlier than 3.8, so changing the target.

[eryksun]

In a patch review 1 for bpo-24505, I had a discussion with Toby Tobkin about extending the behavior of shutil.which() on Windows. This was to match the behavior of the CMD shell, including securing the search path via NeedCurrentDirectoryForExePath, searching for the exact filename, and implementing a real execute-access check.

CMD won't directly execute a file that lacks execute access -- even a script or data file. We can implement this with a real implementation of os.access() that checks file security, for which there's an existing issue. (My previous suggestion to use CreateProcess and AssocQueryString to check for execute access was over the top. At the time I was thinking about building out support for a high-level shutil.execute, but it's out of scope here.)

Note that CMD will only try to execute an extensionless file if "." is in PATHEXT. (In the Windows file namespace, "name" and "name." are equivalent.) This obviously works for PE binaries, but we can also define an association for "." in the registy, which is useful for extensionless script files. Just associate "." files with a launcher that tries CreateProcess and falls back on a shebang line.

[asottile]

should I open a new issue for this, or is this an appropriate task to add to this discussion

shutil.which also doesn't apply PATHEXT when the path contains a directory separator

C:\Users\IEUser\astpretty>venv\Scripts\python --version
Python 3.7.5

C:\Users\IEUser\astpretty>venv\Scripts\python -c "import shutil; print(repr(shutil.which(r'venv\Scripts\python')))"
None

C:\Users\IEUser\astpretty>venv\Scripts\python -c "import shutil; print(repr(shutil.which(r'venv\Scripts\python.exe')))"
'venv\\Scripts\\python.exe'

[Zheaoli]

Hello Anthony

would you mind to execute this command on your PC?

python -c "import os; print(os.environ.get("PATHEXT", "").split(os.pathsep))"

and show the result about this code?

[asottile]

sure, it's the standard PATHEXT (basically a fresh vm from modern.ie) -- the only reason I noticed this is I was reading the source:

cpython/Lib/shutil.py

Lines 1367 to 1373 in 5bbac8c

	# If we're given a path with a directory part, look it up directly rather
	# than referring to PATH directories. This includes checking relative to the
	# current directory, e.g. ./script
	if os.path.dirname(cmd):
	if _access_check(cmd, mode):
	return cmd
	return None

C:\Users\IEUser>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW

[mitchelly]

I am struggling with the same issue as Anthony. To provide a more direct response to Manjusaka's query:

python -c "import os; print(os.environ.get(\"PATHEXT\", \"\").split(os.pathsep))"
['.COM', '.EXE', '.BAT', '.CMD', '.VBS', '.VBE', '.JS', '.JSE', '.WSF', '.WSH', '.MSC']

Like Anthony, wondering if this should be it's own issue. Simply supporting paths with separators shouldn't, I think, lead to any of the ambiguities of matching a file exactly, without PATHEXT treatment.