-
-
Notifications
You must be signed in to change notification settings - Fork 32.3k
gh-123471: Make itertools.product and itertools.combinations thread-safe #132814
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
kumaraditya303
merged 7 commits into
python:main
from
eendebakpt:itertools_combinations_ft
Jun 30, 2025
Merged
gh-123471: Make itertools.product and itertools.combinations thread-safe #132814
kumaraditya303
merged 7 commits into
python:main
from
eendebakpt:itertools_combinations_ft
Jun 30, 2025
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…thon into itertools_combinations_ft
kumaraditya303
approved these changes
Jun 25, 2025
kumaraditya303
approved these changes
Jun 30, 2025
AndPuQing
pushed a commit
to AndPuQing/cpython
that referenced
this pull request
Jul 11, 2025
…read-safe (python#132814) Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Pranjal095
pushed a commit
to Pranjal095/cpython
that referenced
this pull request
Jul 12, 2025
…read-safe (python#132814) Co-authored-by: Kumar Aditya <kumaraditya@python.org>
picnixz
pushed a commit
to picnixz/cpython
that referenced
this pull request
Jul 13, 2025
…read-safe (python#132814) Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We make concurrent iteration over
itertools.combinations
anditertools.product
thread safe in the free-threading build.The original
combinations_next
is renamed tocombinations_next_with_lock_held
andcombinations_next
is now callingcombinations_next_with_lock_held
with a lock (similar foritertools.product
)We use a lock because it is easy to implement and avoids quite a bit of complexity (we have two pieces of mutable state to deal with:
op->indices
andop->result
).Issues that can occur without the locks:
op->result
can be overwritten, resulting in memory leakscpython/Modules/itertoolsmodule.c
Line 2342 in a4ea80d
The increment of
op->indices[i]
at https://github.com/python/cpython/blob/main/Modules/itertoolsmodule.c#L2379 is not safe. It can go out-of-bounds since the check is done earlier. This can lead to a segfaultThe refcount check for re-use of the result tuple https://github.com/python/cpython/blob/main/Modules/itertoolsmodule.c#L2346 is not valid in the FT build. Not sure whether this leads to crashes, but it will result in memory leaks.
Updating the indices is not atomic itertoolsmodule.c#L2380-L2381. Non-atomic updates can lead to out-of-bounds issues.
The tests in this PR trigger some of these issues, although some are not visible (e.g. the memory leak), and it typically requires more iterations to result in a segfault. On my system > 2000 iterations gives a very high probability of triggering a segfault. The number of iterations is set much lower to keep the duration of the test < 0.1 second.
Performance with the locks is about 5% less for a single-thread (see the corresponding issue).
I refactored the tests to avoid duplicated code. Currently
combinations
andproduct
are in the test, butcwr
andpermutations
have the same style and could be added as well (in a followup PR).Could we do this without a full lock? It depends a bit on the iterator. For
product
we could make the rollover check safe by changingindices[i] == PyTuple_GET_SIZE(pool)
intoindices[i] >= PyTuple_GET_SIZE(pool)
and use atomic operations in all operations dealing withop->indices
orop->result
. That would still leave memory leaks, but these are not crashes. And determining whether this actually safe (not crashing) requires some very careful reviews.