os.listdir(...) on deep path on windows in python2.7 fails with errno 123

[asottile]

BPO	32539
Nosy	@pfmoore, @tjguk, @zware, @serhiy-storchaka, @eryksun, @zooba, @asottile
PRs	[2.7] bpo-32539: Fix OSError for os.listdir for extended paths on Windows #5169

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-01-15.21:40:26.673>
created_at = <Date 2018-01-12.16:56:45.525>
labels = ['type-bug', 'library', 'OS-windows']
title = 'os.listdir(...) on deep path on windows in python2.7 fails with errno 123'
updated_at = <Date 2018-01-15.21:40:26.672>
user = 'https://github.com/asottile'

bugs.python.org fields:

activity = <Date 2018-01-15.21:40:26.672>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-01-15.21:40:26.673>
closer = 'serhiy.storchaka'
components = ['Library (Lib)', 'Windows']
creation = <Date 2018-01-12.16:56:45.525>
creator = 'Anthony Sottile'
dependencies = []
files = []
hgrepos = []
issue_num = 32539
keywords = ['patch']
message_count = 4.0
messages = ['309862', '309866', '310017', '310018']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'Anthony Sottile']
pr_nums = ['5169']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32539'
versions = ['Python 2.7']

[asottile]

On windows, a deep path can be accessed by prefixing the path with \\?\

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx?f=255&MSPPError=-2147217396#maxpath

The call to listdir() fails because it uses a posix separator.

A quick patch to fix this is to use SEP here:

cpython/Modules/posixmodule.c

Line 2388 in ab95b30

namebuf[len++] = '/';

(I can submit a patch for this)

Here's a stacktrace

>>> os.listdir(r'\\?\C:\Temp')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: '\\\\?\\C:\\Temp/*.*'

This works fine in python3, as the code has been refactored to use SEP already.

>>> os.listdir(r'\\?\C:\Temp')
[]

[eryksun]

This should be fixed. That said, we have to use a unicode string for a long path anyway. Prior to Windows 8, the conversion from ANSI to Unicode in the system runtime library uses a static MAX_PATH buffer, so the ANSI API is inherently limited to MAX_PATH. You'll see this in the documentation of all functions that allow using \\?\ extended paths:

In the ANSI version of this function, the name is limited to 
MAX_PATH characters. To extend this limit to 32,767 wide 
characters, call the Unicode version of the function and 
prepend "\\?\" to the path.

Decoding ANSI strings uses a dynamically-sized buffer in Windows 8, but the change is undocumented and should not be relied upon.

Unfortunately, we can't reliably use a raw unicode string literal in Python 2, since \u and \U escapes are still evaluated. We can instead use forward slashes and normalize via os.path.normpath.

[serhiy-storchaka]

New changeset 27f32e9 by Serhiy Storchaka (Anthony Sottile) in branch '2.7':
bpo-32539: Fix OSError for os.listdir() for extended-length paths on Windows (bpo-5169)
27f32e9

[serhiy-storchaka]

Thank you Anthony for your patch!