Skip to content

gh-154916: Fix data race in ga_iter_reduce under free-threading - #154944

Open
tekinertekin wants to merge 2 commits into
python:mainfrom
tekinertekin:fix-ga-iter-reduce-race
Open

gh-154916: Fix data race in ga_iter_reduce under free-threading#154944
tekinertekin wants to merge 2 commits into
python:mainfrom
tekinertekin:fix-ga-iter-reduce-race

Conversation

@tekinertekin

@tekinertekin tekinertekin commented Jul 30, 2026

Copy link
Copy Markdown

Issue: #154916

ga_iternext takes gi->obj out with _Py_atomic_exchange_ptr and then drops the
reference, while ga_iter_reduce reads the same field twice without synchronisation:

if (gi->obj)
    return Py_BuildValue("N(O)", iter, gi->obj);

The two reads can straddle that exchange, so the guard can observe a non-NULL
pointer that is then passed to Py_BuildValue after the owning thread has
already called Py_DECREF on it. That makes this a potential use-after-free
rather than only a torn read. ThreadSanitizer trips on the guard read first,
which is what the issue reports.

The fix takes a single strong reference instead:

  • _Py_XGetRef in the free-threaded build, plain Py_XNewRef otherwise, with
    "N(N)" so Py_BuildValue consumes the reference we now own.
  • _Py_XGetRef documents that the writer must set maybe-weakref on the stored
    object, otherwise its try-incref can never succeed from another thread and it
    spins. ga_iter stores with a plain Py_NewRef, so the writer side needs
    _PyObject_SetMaybeWeakref.

Racing with next() may legitimately observe either the object or the exhausted
iterator; both reductions are correct and that does not change here.

Verification

Built on arm64 macOS with --disable-gil --with-thread-sanitizer --with-pydebug.
Reproducer: 4 threads calling next() and 4 calling __reduce__() on one shared
iter(list[int]).

TSAN warnings result
before 2 — ga_iternext:946ga_iter_reduce:1000, both orderings SIGABRT
after 0 clean
  • test_genericalias, test_types, test_typing: 913 tests pass.
  • test_genericalias test_types -R 3:3: no leaks. The ON change moves
    ownership of the reference, so this seemed worth checking explicitly.
  • __reduce__ output and pickle round-trip are unchanged for both a live and an
    exhausted iterator.

One open question

I kept this lock-free to stay symmetric with ga_iternext as gh-154108 left it.
The alternative is the list_item_impl pattern — a critical section plus
_Py_NewRefWithLock — but that only synchronises if ga_iternext takes the lock
too, which would mean revising the design just merged in gh-154108.
_Py_XGetRef has no other call site in the tree, so if you would rather have the
critical-section version, I am happy to redo it that way.

ga_iternext takes gi->obj out with an atomic exchange and then drops the
reference, while ga_iter_reduce read the same field twice without
synchronisation.  The two reads can straddle the exchange, so the guard
can observe a non-NULL pointer that is then passed to Py_BuildValue after
the owning thread has already released it.

Take a single strong reference instead, and mark the stored object as
maybe-weakref in ga_iter, which _Py_XGetRef requires of the writer.
@python-cla-bot

python-cla-bot Bot commented Jul 30, 2026

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant