resolve() on Windows makes some pathological paths unusable

[KevinNorris]

BPO	22299
Nosy	@pitrou, @tjguk, @zware, @eryksun, @zooba
Files	22299_1.patch 22299_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-08-29.13:17:08.234>
labels = ['type-bug', '3.8', '3.9', '3.10', 'library', 'OS-windows']
title = 'resolve() on Windows makes some pathological paths unusable'
updated_at = <Date 2021-02-23.09:21:35.542>
user = 'https://bugs.python.org/KevinNorris'

bugs.python.org fields:

activity = <Date 2021-02-23.09:21:35.542>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2014-08-29.13:17:08.234>
creator = 'Kevin.Norris'
dependencies = []
files = ['36549', '36550']
hgrepos = []
issue_num = 22299
keywords = ['patch']
message_count = 28.0
messages = ['226060', '226061', '226072', '226076', '226077', '226081', '226082', '226084', '226428', '226431', '226433', '226435', '226441', '226458', '226462', '226484', '226492', '226493', '226514', '226516', '226519', '226524', '226533', '226540', '226586', '226796', '226799', '387556']
nosy_count = 6.0
nosy_names = ['pitrou', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'Kevin.Norris']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22299'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[KevinNorris]

Run Python as an administrator:

    >>> import pathlib 
    >>> pth = pathlib.Path('//?/C:/foo.')
    >>> pth.mkdir()
    >>> pth.resolve().rmdir()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\Python34\lib\pathlib.py", line 1141, in rmdir
        self._accessor.rmdir(self)
      File "C:\Python34\lib\pathlib.py", line 323, in wrapped
        return strfunc(str(pathobj), *args)
    FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\\foo.'
    >>> pth.rmdir()

You do not need to be an administrator so long as you can create a directory in the requested location, but the \\?\ prefix only works with absolute paths so it's easier to demonstrate in the root of the drive.

[KevinNorris]

When the directory name is '...', the error is different:

    >>> pth = pathlib.Path('//?/C:/...')
    >>> pth.mkdir()
    >>> pth.resolve().rmdir()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\Python34\lib\pathlib.py", line 1141, in rmdir
        self._accessor.rmdir(self)
      File "C:\Python34\lib\pathlib.py", line 323, in wrapped
        return strfunc(str(pathobj), *args)
    PermissionError: [WinError 5] Access is denied: 'C:\\...'
    >>> pth.rmdir()
    >>>

[pitrou]

Why is it a "pathological path"? Can you explain?

[zooba]

Is resolve() using an *A() API rather than *W()? The \\?\ prefix does not work with *A() APIs.

Also, names that are all dots are not supported by Windows at all. I'd expect mkdir() to fail on that, but the \\?\ prefix disables some validation, so it's possible that it is getting through that way.

[pitrou]

resolve() should use the *W APIs since it is using only functions from the os module with str objects. Perhaps you want to double-check that, since I don't have a Windows VM anymore.

[eryksun]

The \\?\ extended-path prefix bypasses normal path processing. The path is passed directly to the filesystem driver. For example, to accommodate the POSIX namespace, NTFS allows any character except NUL and slash, so it happily creates a directory named "foo.". This name is invalid in the Win32 namespace.

resolve() should skip calling _ext_to_normal on the result of _getfinalpathname if the input path is extended.

http://hg.python.org/cpython/file/c0e311e010fc/Lib/pathlib.py#l178

[zooba]

Removing the _ext_to_normal() call in resolve() looks like the right fix to me.

[pitrou]

Does one of you want to provide a patch? (with tests)

[zooba]

Patch attached. (Kinda feel like this was too simple...)

[eryksun]

It should only skip _ext_to_normal for an already extended path, i.e. a path that starts with ext_namespace_prefix. Otherwise it needs to call _ext_to_normal. For example:

Strip the prefix in this case:

    >>> os.path._getfinalpathname('C:\\Windows')       
    '\\\\?\\C:\\Windows'

but not in this case:

    >>> os.path._getfinalpathname(r'\\?\GLOBALROOT\Device\HarddiskVolume1\Windows')
    '\\\\?\\C:\\Windows'

[zooba]

Ah, thought it was too simple. I didn't realise that _getfinalpathname adds the prefix.

New patch soon.

[zooba]

Strips the prefix if it wasn't in the original path - otherwise, keeps it.