os.path.exists(os.devnull) regression on windows

[dkagedal]

BPO	1311
Nosy	@loewis, @facundobatista, @kdwyer, @zooba
PRs	[3.8] bpo-37834: Normalise handling of reparse points on Windows (GH-15231) #15370

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-02-23.21:19:36.246>
created_at = <Date 2007-10-22.12:04:18.402>
labels = ['3.8', 'library', '3.9']
title = 'os.path.exists(os.devnull) regression on windows'
updated_at = <Date 2019-08-21.22:52:45.584>
user = 'https://bugs.python.org/dkagedal'

bugs.python.org fields:

activity = <Date 2019-08-21.22:52:45.584>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2008-02-23.21:19:36.246>
closer = 'akuchling'
components = ['Library (Lib)']
creation = <Date 2007-10-22.12:04:18.402>
creator = 'd_kagedal'
dependencies = []
files = []
hgrepos = []
issue_num = 1311
keywords = []
message_count = 14.0
messages = ['56644', '56645', '56652', '56674', '56684', '56692', '56733', '56734', '56735', '56795', '56873', '350122', '350124', '350126']
nosy_count = 6.0
nosy_names = ['loewis', 'facundobatista', 'ggenellina', 'd_kagedal', 'kdwyer', 'steve.dower']
pr_nums = ['15370']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1311'
versions = ['Python 3.8', 'Python 3.9']

[dkagedal]

When moving from Python 2.4 to Python 2.5, my program stopped working on
win32 because of a change in os.path.exists. I couldn't find any
description of the change, so I can only assume it's a bug.

The os.devnul variable contains the name of the "null file", which is
"/dev/null" on posix, and "nul" on win32. As I understand it, "NUL" is a
file that exists in every directory on win32.

Opening the "nul" file with open("nul", "r") works fine, but
os.path.exists("nul") returns False. In Python 2.4, it returns True.

[dkagedal]

It's called os.devnull, and nothing else.

[facundobatista]

You migrated only of Python version, or also of windows installation?

I checked Py25 and Py23 in my Win 2k, and in both I have the same behaviour:

C:\Python23>python
Python 2.3.5 (#62, Feb  8 2005, 16:23:02) [MSC v.1200 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.stat("nul")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
OSError: [Errno 22] Invalid argument: 'nul'
>>> os.path.exists("nul")
False
>>>

I checked the os.stat() function because the documentation says that if
it gives an error, exists() will return False (and that this is how it's
implemented).

BTW, note that if you call to the GetFileAttributesEx() function of
kernel32.dll by yourself, it will give error for the "NUL" file, showing
that it actually does not exist.

Because of this, I'd say that current behaviour of os.exists() is ok,
but I want to know the answer of my question regarding your change
before closing this as not-a-bug.

Thanks!

[dkagedal]

I tried it on two different machines, and got two different answers:

conan$ /cygdrive/c/Python25/python -c 'import os.path; print
os.path.exists("nul")'
False
conan$ /cygdrive/c/Python24/python -c 'import os.path; print
os.path.exists("nul")'
False

conan$ /cygdrive/c/Python24/python -c 'import os; print os.stat("nul")'
Traceback (most recent call last):
  File "<string>", line 1, in ?
OSError: [Errno 22] Invalid argument: 'nul'
conan$ /cygdrive/c/Python25/python -c 'import os; print os.stat("nul")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
WindowsError: [Error 87] The parameter is incorrect: 'nul'

titti$ /cygdrive/c/Python24/python -c 'import os.path; print
os.path.exists("nul")'
True
titti$ /cygdrive/c/Python25/python -c 'import os.path; print
os.path.exists("nul")'
False

titti$ /cygdrive/c/Python24/python -c 'import os; print os.stat("nul")'
(33206, 0L, 3, 1, 0, 0, 0L, -1, -1, -1)
titti$ /cygdrive/c/Python25/python -c 'import os; print
os.stat("nul")'Traceback (most recent call last):
  File "<string>", line 1, in <module>
WindowsError: [Error 87] The parameter is incorrect: 'nul'

I ran it from a cygwin prompt, but the pythons are native.

So you are correct that it doesn't work as I expected in Python 2.4
either on the "conan" host.. On "titti", there is a difference in how
os.path.exist works between 2.4 and 2.5. On "conan" there is actually
also a difference, but only in the exception raised by os.stat.

I don't know what the differences between these installation are.

[kdwyer]

I tried this under Python 2.3.3, 2.5.1 (native) and 2.3.4 (cygwin). The
operating system is Windows 2000 SP4.

C:\Python23>python
Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os.path
>>> print os.path.exists('nul')
True

C:\Python25>python
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os.path
>>> print os.path.exists('nul')
False

$ python
Python 2.3.4 (#1, Jun 13 2004, 11:21:03)
[GCC 3.3.1 (cygming special)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os.path
>>> print os.path.exists('nul')
True

So there does seem to be a change in behaviour at 2.5.

[kdwyer]

Ok, it seems that Python 2.5 implements two new functions
Py_GetFileAttributesExA and Py_GetFileAttributesExW in posixmodule.c
within the #ifdef MS_WINDOWS block that may return
ERROR_INVALID_PARAMETER to os.stat, which will percolate WindowsError up
to os.exists():

In both functions we find:

static BOOL WINAPI
Py_GetFileAttributesExA(LPCSTR pszFile, 
		       GET_FILEEX_INFO_LEVELS level,
                       LPVOID pv)
{
	BOOL result;
	LPWIN32_FILE_ATTRIBUTE_DATA pfad = pv;
	/* First try to use the system's implementation, if that is
	   available and either succeeds to gives an error other than
	   that it isn't implemented. */
	check_gfax();
	if (gfaxa) {
		result = gfaxa(pszFile, level, pv);
		if (result || GetLastError() != ERROR_CALL_NOT_IMPLEMENTED)
			return result;
	}
	/* It's either not present, or not implemented.
	   Emulate using FindFirstFile. */
	if (level != GetFileExInfoStandard) {
		SetLastError(ERROR_INVALID_PARAMETER);
		return FALSE;
...


static BOOL WINAPI
Py_GetFileAttributesExW(LPCWSTR pszFile, 
		       GET_FILEEX_INFO_LEVELS level,
                       LPVOID pv)
{
	BOOL result;
	LPWIN32_FILE_ATTRIBUTE_DATA pfad = pv;
	/* First try to use the system's implementation, if that is
	   available and either succeeds to gives an error other than
	   that it isn't implemented. */
	check_gfax();
	if (gfaxa) {
		result = gfaxw(pszFile, level, pv);
		if (result || GetLastError() != ERROR_CALL_NOT_IMPLEMENTED)
			return result;
	}
	/* It's either not present, or not implemented.
	   Emulate using FindFirstFile. */
	if (level != GetFileExInfoStandard) {
		SetLastError(ERROR_INVALID_PARAMETER);
		return FALSE;
...

I'm neither a C nor a win32api programmer - can anyone explain the
purpose of this code?

[loewis]

The purpose of Py_GetFileAttributesEx* is to wrap GetFileAttributesEx,
on systems where it doesn't exist (Windows 95 in particular). If it
doesn't exist, it is emulated; if it exists, it is directly called.

[loewis]

As Facundo found out, the behavior of os.path.exists is fairly
irrelevant here, as that functions is trivial. What really matters is
whether os.stat succeeds for NUL. Can those users for whom it succeeds
please report what Windows versions they are using, and what precisely
the stat result for NUL is (and perhaps also for CON, PRN, etc)?

[loewis]

Please disregard Cygwin Python for this discussion. It (probably) uses
the stat implementation from cygwin1.dll, which may work differently
from Cygwin release to Cygwin release.

[facundobatista]

>>> import os.path
>>> os.path.exists("con")
False
>>> os.path.exists("nul")
False
>>> os.path.exists("prn")
False

This is in Windows 2000 (5.00.2195) sp4, using *both* Python 2.3.5 and
2.5.1, no cygwin.

Personally, I'm +1 with Mark Hammond about this:

I agree it is unfortunate that the behaviour has changed, 
but these special names are broken enough on Windows that 
rely on the kernel32 function and behaves like it says 
seems the sanest thing to do.

Taking in consideration that *now* it behaves equally in different
windows systems, but not before, I think this issue should be closed as
"invalid" (it's not a bug).

I'll wait some days to get more behaviour responses, though.

Regards,

[ggenellina]

All these tests on Windows XP SP4, executing os.stat("nul")

Python 2.1 thru 2.4 raises:
OSError: [Errno 22] Invalid argument: 'nul'

Python 2.5 gives a different error:
WindowsError: [Error 87] El parámetro no es correcto: 'nul'

[zooba]

New changeset df2d4a6 by Steve Dower in branch 'master':
bpo-37834: Normalise handling of reparse points on Windows (GH-15231)
df2d4a6

[zooba]

Well, I was trying not to resurrect this old issue, but looks like I did it accidentally.

Hi there, old hands! We miss you :)

We also figured out a new [l]stat() implementation on Windows that handles symlinks and junctions better, and also non-file types such as NUL. So I guess I'll mark this as resolved in Python 3.8

[zooba]

New changeset 9eb3d54 by Steve Dower in branch '3.8':
bpo-37834: Normalise handling of reparse points on Windows (GH-15370)
9eb3d54