`os.path.realpath` raises a `ValueError` when path contains a null byte

[stefanhoelzl]

Bug report

Bug description:

import os
os.path.realpath("\x00")  # raises unexpected ValueError

The ValueError is initially raised by os.lstat.

This behavior is inconsistent with other path operations catching the ValueError raised by os.lstat:
os.path.lexsists: https://github.com/python/cpython/blob/main/Lib/genericpath.py#L30
os.path.exists: https://github.com/python/cpython/blob/main/Lib/genericpath.py#L20
os.path.islink: https://github.com/python/cpython/blob/main/Lib/genericpath.py#L64
os.path.ismount: https://github.com/python/cpython/blob/main/Lib/posixpath.py#L197
os.path.ismount: https://github.com/python/cpython/blob/main/Lib/posixpath.py#L213

CPython versions tested on:

3.10, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-117596: Fix realpath ValueError on null byte #117573
• gh-117596: Add more tests for os.path with invalid paths #134189
• [3.14] gh-117596: Add more tests for os.path with invalid paths (GH-134189) #134265
• [3.13] gh-117596: Add more tests for os.path with invalid paths (GH-134189) #134266

[stefanhoelzl]

I already created a PR #117573 for this issue

[serhiy-storchaka]

I do not think there is a bug. Functions like os.path.exists() answer the question "is there a node (of the specified type) with such path?" Since the valid path cannot contain the null character, the answer is "no". You get the same answer for a path containing non-encodable characters. os.path.realpath() answers the different question -- "what is the real path of the specified node?"

os.path.realpath() ignores some OSError in non-strict mode. This allows it to process paths to non-existent nodes. These errors can be fixed by making a change to the file system. But no matter what you do with the file system, an invalid path (a path containing the null characters or non-encodable characters) will remain invalid.

[stefanhoelzl]

In case of os.path.exists I can agree.

I guess my selection of examples was not very good.
os.path.basename, os.path.dirname, os.fspath, os.path.abspath, os.path.relpath ... (list is not complete) also do not raise a ValueError when given a path with a null byte.
All the methods do not raise a ValueError when given a path with a null byte, they just accept it as part of the path.

I would except consistency across all path operations, but I found only os.path.realpath with another behaviour. So I assumed this is the one with the bug.

Python 3.13.0a5+ (heads/fix-realpath-value-error-dirty:8da4ebb996, Apr  7 2024, 17:05:12) [GCC 13.2.1 20230918 (Red Hat 13.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.fspath("\x00")
'\x00'
>>> os.path.abspath("\x00")
'/var/home/stefan/Development/repos/cpython/\x00'
>>> os.path.basename("\x00")
'\x00'
>>> os.path.dirname("\x00")
''
>>> os.path.relpath("\x00")
'\x00'
>>> os.path.normpath("\x00")
'\x00'
>>> 

[serhiy-storchaka]

os.path.dirname and os.path.relpath do not work with filesystem. They could check for the null character etc, but nobody is interesting in making them intentionally slower. In functions which interact with OS, it happens as the part of conversion to OS-specific representation.

[encukou]

Serhiy's model sounds good to me. To summarize:

• exists and is* functions should return False if the path contains NUL
• functions that deal with the OS should raise ValueError (before handing the path to an OS call)
• functions that manipulate the path should either raise ValueError (if they can detect it easily), or treat NUL as a normal character (the same way as an invalid character like | is handled in ntpath.dirname)

But, it looks like that's not what we do on Windows.

[serhiy-storchaka]

I added some tests for realpath() with invalid paths in PR #134147. Then I decided that this issue was a better fit and separated the tests into a separate PR #134189. And added more tests.

Now, whatever we decide, there are already tests that will show changes in behavior.