Optimize pathlib path construction

[barneygale]

Pathlib is slow. One of the most obvious symptoms is that pathlib.PurePath objects are slow to construct. We should be able to speed construction up without making other parts of pathlib slower.

Two possible approaches:

1. Optimize the existing machinary of path construction: __new__(), _from_parts(), _parse_parts(), _parse_args().
2. Perform less work in the constructor: defer parsing, joining and normalization until needed.

Linked PRs

• GH-101362: Optimise pathlib by deferring path normalisation #101560
• GH-101362: Check pathlib.Path flavour compatibility at import time #101664
• GH-101362: Call join() only when >1 argument supplied to pathlib.PurePath() #101665
• GH-101362: Optimise PurePath(PurePath(...)) #101667
• GH-101362: Omit path anchor from pathlib.PurePath()._parts #102476

[barneygale]

I'd like to land #101363 before I put the first PR up for this issue.

[zooba]

Does the PR cope with WindowsPurePath and PosixPurePath and converting between them?

[barneygale]

Could you clarify? The PRs maintain the behaviour that attempting to instantiate pathlib.PurePath will give you a PureWindowsPath or a PurePosixPath depending on your platform, but pathlib doesn't support converting between Windows and POSIX paths as I understand it.

[zooba]

This behaviour should be preserved:

Python 3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathlib import *
>>> p = PureWindowsPath("a/b/c")
>>> p
PureWindowsPath('a/b/c')
>>> PurePosixPath(p)
PurePosixPath('a/b/c')
>>> PurePath(p)
PureWindowsPath('a/b/c')
>>> PosixPath(p)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__3847v3x7pw1km\lib\pathlib.py", line 962, in __new__
    raise NotImplementedError("cannot instantiate %r on your system"
NotImplementedError: cannot instantiate 'PosixPath' on your system
>>> WindowsPath(p)
WindowsPath('a/b/c')
>>>

[barneygale]

Right! That will be broken by #101667 as things stand:

>>> from os import fspath
>>> from pathlib import *
>>> p = PureWindowsPath("a/b/c")
>>> p
PureWindowsPath('a/b/c')
>>> fspath(p)
'a\\b\\c'
>>> PurePosixPath(fspath(p))
PurePosixPath('a\\b\\c')
>>> PurePosixPath(p)
PurePosixPath('a\\b\\c')

It doesn't appear to be documented or tested behaviour, and it feels odd to me that PurePosixPath(p) can give a different result to PurePosixPath(fspath(p)). I'll try to find a good way forwards!

[barneygale]

This feature doesn't really work with drives or roots:

>>> PurePosixPath(PureWindowsPath('//server/share/dir'))
PurePosixPath('\\\\server\\share\\/dir')
>>> PurePosixPath(PureWindowsPath('c:/dir'))
PurePosixPath('c:\\/dir')
>>> PurePosixPath(PureWindowsPath('/dir'))
PurePosixPath('\\/dir')

As far as I can tell, no one has ever logged a bug about it.

However, using PurePosixPath(PureWindowsPath(...).as_posix()) works everywhere:

>>> PurePosixPath(PureWindowsPath('//server/share/dir').as_posix())
PurePosixPath('//server/share/dir')
>>> PurePosixPath(PureWindowsPath('c:/dir').as_posix())
PurePosixPath('c:/dir')
>>> PurePosixPath(PureWindowsPath('/dir').as_posix())
PurePosixPath('/dir')

So I'm tempted to conclude that converting with PurePosixPath(PureWindowsPath(...)) alone is not supported, and that it only happened to work in some limited circumstances. Users can add .as_posix() to signal their intent and make it work reliably. What do you reckon?

[zooba]

I think the direct conversion should either be consistent with fspath(p) or p.parts, but no strong preference which. As you say, neither makes a huge amount of sense, and the workaround (.as_posix()) has been there for a long time.

Might be worth adding a short note to the docs warning that the constructor can't reliably convert from different pathlib objects, and maybe specify whether it will fspath or .parts them.

On another perf note, is it possible that parsing the path up front isn't necessary? Obviously it'll save the most time to keep a single string literal around and parse it later, but I don't personally have a good feel for whether that's common or not. (Obviously if it's available pre-parsed then keep it.)

[barneygale]

On another perf note, is it possible that parsing the path up front isn't necessary? Obviously it'll save the most time to keep a single string literal around and parse it later, but I don't personally have a good feel for whether that's common or not. (Obviously if it's available pre-parsed then keep it.)

That's my plan! We can return the unnormalized path from PurePath.__fspath__() and only normalize/parse on demand, e.g. when users call .name or .parent. This would speed up construction, joining, and most methods defined in the Path class.

[barneygale]

For anyone following along, I think this ticket can be resolved if/when this PR lands:

• GH-101362: Omit path anchor from pathlib.PurePath()._parts #102476

[barneygale]

Resolving this issue. It's now much cheaper to construct pathlib.PurePath and Path objects. Some operations like str() and joinpath() are also cheaper.

Future work:

• Optimize pathlib.PurePath.__fspath__() #102783
• Path.rglob performance issues in deeply nested directories compared to glob.glob(recursive=True) #102613
• Add C implementation of os.path.splitroot() #102511
• Add C implementation of os.path.join()