normpath does not work with local literal paths

[mandel]

BPO	15286
Nosy	@pfmoore, @pitrou, @tjguk, @briancurtin, @zware, @eryksun, @zooba
Files	literal-normpath.patch: Patch that adds support for normpath on literal local paths.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-07-07.16:47:21.864>
labels = ['type-bug', '3.8', '3.9', '3.10', 'library', 'OS-windows']
title = 'normpath does not work with local literal paths'
updated_at = <Date 2021-03-21.23:12:21.028>
user = 'https://bugs.python.org/mandel'

bugs.python.org fields:

activity = <Date 2021-03-21.23:12:21.028>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2012-07-07.16:47:21.864>
creator = 'mandel'
dependencies = []
files = ['26307']
hgrepos = ['141']
issue_num = 15286
keywords = ['patch', 'needs review']
message_count = 8.0
messages = ['164909', '164921', '164925', '164985', '164990', '220031', '236992', '389263']
nosy_count = 8.0
nosy_names = ['paul.moore', 'pitrou', 'tim.golden', 'brian.curtin', 'mandel', 'zach.ware', 'eryksun', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue15286'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[mandel]

Local literal paths are those paths that do use the \\?\ that allows to have paths longer that the MAX_PATH set by Windows (http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx#short_vs.\_long_names).

While UNC (http://en.wikipedia.org/wiki/Path_%28computing%29) paths should not be normalized, local paths that do use the \\?\ prefix should so that developers that use such a trink to allow longer paths on windows do not have to wrapp the call in the following way:

LONG_PATH_PREFIX = '\\?\'
path = path.replace(LONG_PATH_PREFIX, '')
result = LONG_PATH_PREFIX + os.path.normpath(path)

The possible solution would be for the normalization to work and return the path normalized with the prefix added.

[briancurtin]

Hi mandel :)

With the exception of the "import string" inside of _get_letters (policy is to do all imports at the top), it looks ok just by reading.

Assigning to myself to complete it after I return from holiday in a few days (unless someone beats me).

[pitrou]

I think this is wrong. The MSDN doc says:

“Because it turns off automatic expansion of the path string, the "\\?\" prefix also allows the use of ".." and "." in the path names, which can be useful if you are attempting to perform operations on a file with these otherwise reserved relative path specifiers as part of the fully qualified path.”

(from http://msdn.microsoft.com/en-us/library/aa365247%28v=vs.85%29.aspx#namespaces)

So by "normalizing" the extended path, you are actually changing its meaning.

Furthermore, normpath() is generally wrong in the face of symlinks, meaning it shouldn't be used at all.

[mandel]

Antoine,

What the MSDN is stating is that the Windows functions from COM will not normalize the path if it is prefixed by \\?\. That is, if a user wanted to do:

path = r'\\?\C:\Users\mandel\..\Desktop\test'
with open(path, 'w') as fd:
    fd.write('hello!')

he will get the following:

[Errorno 22] Invalid argument. r'\\?\C:\Users\mandel\..\Desktop\test'

The same think would happen if a C function is used, that is, open is doing the right thing. On the other hand, the same code without the \\?\ works.

This makes it even more important to allow the normpath users to normalize such paths, that is, a developer knows that the path has more than 260 chars and wants to make sure that the path can be written in the system:

May I ask you why you mention the symbolic links? I know that if one of the segments of the path is a symbolic link there are problems but this is not related to \\?\ or am I confused? Just curious :)

Brian,

The ntpath module is a little mess (look at my other patch http://bugs.python.org/issue15275) and I think there are more performance problems hidden there somewhere...

I imported string within the function because the same is done in expandvars (around line 430) and wanted to follow the style that was already in use in the file. I do agree that imports at the top are the way to go :)

[pitrou]

May I ask you why you mention the symbolic links? I know that if one
of the segments of the path is a symbolic link there are problems but
this is not related to \\?\ or am I confused? Just curious :)

No, it is not related with "\\?\" but I'm pointing out that normpath()
isn't very useful because of that. And Windows has symlink support
nowadays :-)

For the record, I'm trying to build a saner path-handling library at
http://pypi.python.org/pypi/pathlib/ . I hope to propose it for
inclusion in 3.4.

[BreamoreBoy]

As Antoine's pathlib made it into 3.4 is the patch here now obsolete or what? Also note the reference to bpo-15275.

[BreamoreBoy]

As ntpath was cleaned up on bpo-15275 do we need this patch or not, especially given that pathlib made it into 3.4?

[eryksun]

This old issue still needs to be fixed. The check for special_prefixes in ntpath.normpath() must be removed in order to be consistent with WinAPI GetFullPathNameW(). In Windows, one can just call ntpath.abspath() to ensure that nt._getfullpathname() is called. But "Lib/ntpath.py" is cross-platform.

[UlrichEckhardt]

I don't think the described behaviour is a bug, exactly because "\\?\" paths don't support virtual path elements like "." and ".." and "resolving" them from such a path would change the meaning. However, I think that would deserve a warning in the documentation. The code in ntpath.py actually explains what happens pretty well:

            # in the case of paths with these prefixes:
            # \\.\ -> device names
            # \\?\ -> literal paths
            # do not do any normalization, but return the path
            # unchanged apart from the call to os.fspath()

[zooba]

I agree. Noting this in the docs should be fine - we shouldn't normalise something that's been explicitly marked as "not to be normalised"