Race condition in `pathlib` on `3.14t`

[Liam-DeVoe]

Bug report

Bug description:

(First noticed here and again here).

path.is_dir() seems to have a race condition under the free-threading build of 3.14:

E           and: '/home/runner/work/pytest-run-parallel/pytest-run-parallel/.tox/3.14t/lib/python3.14t/site-packages/hypothesis/database.py:455: in fetch'
E           and: '    if not kp.is_dir():'
E           and: '           ^^^^^^^^^^^'
E           and: '/opt/hostedtoolcache/Python/3.14.0-rc.2/x64-freethreaded/lib/python3.14t/pathlib/__init__.py:684: in is_dir'
E           and: '    return os.path.isdir(self)'
E           and: '           ^^^^^^^^^^^^^^^^^^^'
E           and: '<frozen genericpath>:51: in isdir'
E           and: '    ???'
E           and: '/opt/hostedtoolcache/Python/3.14.0-rc.2/x64-freethreaded/lib/python3.14t/pathlib/__init__.py:191: in __fspath__'
E           and: '    return str(self)'
E           and: '           ^^^^^^^^^'
E           and: '/opt/hostedtoolcache/Python/3.14.0-rc.2/x64-freethreaded/lib/python3.14t/pathlib/__init__.py:257: in __str__'
E           and: '    self._str = self._format_parsed_parts(self.drive, self.root,'
E           and: '                                          ^^^^^^^^^^'
E           and: '/opt/hostedtoolcache/Python/3.14.0-rc.2/x64-freethreaded/lib/python3.14t/pathlib/__init__.py:352: in drive'
E           and: '    self._drv, self._root, self._tail_cached = self._parse_path(self._raw_path)'
E           and: '                                                                ^^^^^^^^^^^^^^'
E           and: '_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ '
E           and: ''
E           and: "self = PosixPath('/home/runner/work/pytest-run-parallel/pytest-run-parallel/.hypothesis/examples/f4ec64430b7f30d4')"
E           and: ''
E           and: '    @property'
E           and: '    def _raw_path(self):'
E           and: '        paths = self._raw_paths'
E           and: '        if len(paths) == 1:'
E           and: '            return paths[0]'
E           and: '        elif paths:'
E           and: '            # Join path segments from the initializer.'
E           and: '>           path = self.parser.join(*paths)'
E           and: '                   ^^^^^^^^^^^^^^^^^^^^^^^^'
E           and: "E           TypeError: join() missing 1 required positional argument: 'a'"

I can only reproduce this on 3.14t, not 3.14. Full CI logs: https://github.com/Liam-DeVoe/pytest-run-parallel/actions/runs/17775999693/job/50523739550?pr=2.

I don't have an isolated reproduction, but the CI here can be run to produce this error every ~20 iterations, if that's helpful: Liam-DeVoe/pytest-run-parallel#2.

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs

• gh-139001: Fix thread-safety issue in pathlib.Path #139066
• [3.14] gh-139001: Fix thread-safety issue in pathlib.Path (gh-139066) #139926

[duaneg]

cc @barneygale

Path objects are indeed not thread-safe, but AIUI this is not a bug: they are not documented as being safe, so should be assumed unsafe. It is the user's responsibility to ensure multi-threaded access is suitably protected. This is complicated by the fact that pathlib does lazy initialisation, so performing what looks like a read-only predicate check or property access may implicitly mutate the object and cause such problems.

It looks like the Hypothesis code overlooked that possibility, and is buggy as a result. Which is understandable: what they are doing looks safe and indeed seems to be, in practice, at present, in GIL builds. It could be fixed by ensuring shared Path objects are effectively immutable (e.g. by accessing their root property before they are shared), by adding locking, or by creating separate Path objects for each thread instead of sharing them.

Standalone repro (tweak N and/or ITERS as required):

import concurrent.futures
from pathlib import Path
import threading

raw_path = "a/b/c"

def test(barrier, path):
    barrier.wait()
    return path._raw_path

N = 4
ITERS = 100
for _ in range(ITERS):
    with concurrent.futures.ThreadPoolExecutor(max_workers=N) as executor:
        path = Path(*raw_path.split("/"))
        barrier = threading.Barrier(N)
        futs = [executor.submit(test, barrier, path) for _ in range(N)]
        for raw in [f.result() for f in futs]:
            assert raw == raw_path, f"expected '{raw_path}', got '{raw}'"

[Liam-DeVoe]

Yes, we are indeed sharing Path objects across threads in Hypothesis, good call:

    def _key_path(self, key: bytes) -> Path:
        try:
            return self.keypaths[key]
        except KeyError:
            pass
        self.keypaths[key] = self.path / _hash(key)
        return self.keypaths[key]

    ...

    def fetch(self, key: bytes) -> Iterable[bytes]:
        kp = self._key_path(key)
        if not kp.is_dir():
            return

I agree that modules should be assumed thread-unsafe in the absence of any commentary to the contrary, so I am happy for this issue to be closed. I will say that my - naive - assumption was that Path objects are immutable, and safe to share across threads. Is the non-thread-safe reality worth calling out explicitly in the docs?

[duaneg]

Yeah, it does very much look safe, and it is understandable users are assuming that without digging through the implementation to confirm it. A note in the docs might help, but let's see what the maintainer thinks.

Note, that _key_path method is also racy in its own right, although I think it will work in practice with the current dictionary implementation, and AIUI that is unlikely to change in future versions of CPython.

[nickovs]

Yes, we are indeed sharing Path objects across threads in Hypothesis, good call:
...
I will say that my - naive - assumption was that Path objects are immutable, and safe to share across threads. Is the non-thread-safe reality worth calling out explicitly in the docs?

I would argue that this would be worth calling out in the docs for no other reason than people tend to treat (and likely think about) Path objects rather like strings, and str objects are immutable. Add to this the fact that accessing read-only properties can cause internal mutation and the case that their behaviour is non-intuitive is pretty compelling. If the behaviour is non-intuitive then IMHO it should be documented.

[colesbury]

This feels like something we should fix in CPython

Add to this the fact that accessing read-only properties can cause internal mutation and the case that their behaviour is non-intuitive is pretty compelling. If the behaviour is non-intuitive then IMHO it should be documented.

I agree. I'd prefer we make it thread-safe, but at a minimum we should document the behavior.

[colesbury]

This looks like a regression in 3.14. I think PurePath is immutable in 3.13 and earlier. The race condition exists in the GIL-enabled build too, even though it happens less frequently in practice.

EDIT: 3.13 -> 3.14