gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it #107275
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Most changes to Python require a NEWS entry. Please add it using the blurb_it web app or the blurb command-line tool. |
pitrou
reviewed
Aug 16, 2023
Lib/test/_test_multiprocessing.py
Outdated
| @@ -6155,3 +6155,9 @@ class SemLock(_multiprocessing.SemLock): | |||
| name = f'test_semlock_subclass-{os.getpid()}' | |||
| s = SemLock(1, 0, 10, name, False) | |||
| _multiprocessing.sem_unlink(name) | |||
|
|
|||
| def test_semlock_mixed_context(self): | |||
There was a problem hiding this comment.
Does this test actually need to be Linux-only? (see SemLockTests definition above)
There was a problem hiding this comment.
This is indeed also relevant on macos. Moved the test to be able to test it there.
On windows, there is only spawn, so there is no way to get "mixed" context.
Lib/test/_test_multiprocessing.py
Outdated
|
|
||
| def test_semlock_mixed_context(self): | ||
| queue = multiprocessing.get_context("fork").Queue() | ||
| p = multiprocessing.get_context("spawn").Process(target=close_queue, args=(queue,)) |
There was a problem hiding this comment.
Do we also want to test with "forkserver" method?
There was a problem hiding this comment.
Good catch! Added it.
It now ensure that all cases work. Let me know if that properly covers all the cases you had in mind.
albanD
force-pushed
the
fix-issue-77377
branch
from
b6b25ff
to
152800c
Compare
pitrou
reviewed
Aug 23, 2023
Lib/test/_test_multiprocessing.py
Outdated
| @@ -5421,6 +5421,24 @@ def test_preload_resources(self): | |||
| print(err) | |||
| self.fail("failed spawning forkserver or grandchild") | |||
|
|
|||
| @unittest.skipIf(sys.platform == "win32", "Only Spawn on windows so no risk of mixing") | |||
There was a problem hiding this comment.
Style nit, but can you try to wrap the lines below around ~80 characters max?
There was a problem hiding this comment.
Updated!
Thanks for the update on the blurb, I wasn't sure what to put there
pitrou
added
needs backport to 3.11
|
GH-108377 is a backport of this pull request to the 3.12 branch. |
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Aug 23, 2023
…ed Process before serializing it (pythonGH-107275) Ensure multiprocessing SemLock is valid for spawn Process before serializing it. Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early. --------- (cherry picked from commit 1700d34) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
|
GH-108378 is a backport of this pull request to the 3.11 branch. |
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Aug 23, 2023
…ed Process before serializing it (pythonGH-107275) Ensure multiprocessing SemLock is valid for spawn Process before serializing it. Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early. --------- (cherry picked from commit 1700d34) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
pitrou
added a commit
that referenced
this pull request
Aug 23, 2023
…sed Process before serializing it (GH-107275) (#108378) gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it (GH-107275) Ensure multiprocessing SemLock is valid for spawn Process before serializing it. Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early. --------- (cherry picked from commit 1700d34) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Yhg1s
pushed a commit
that referenced
this pull request
Aug 23, 2023
…sed Process before serializing it (GH-107275) (#108377) gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it (GH-107275) Ensure multiprocessing SemLock is valid for spawn Process before serializing it. Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early. --------- (cherry picked from commit 1700d34) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
pitrou
added a commit
that referenced
this pull request
Aug 30, 2023
…108568) gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Aug 30, 2023
… case (pythonGH-108568) pythongh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- (cherry picked from commit add8d45) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
pitrou
added a commit
that referenced
this pull request
Aug 30, 2023
…e case (GH-108568) (#108692) gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- (cherry picked from commit add8d45) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Yhg1s
pushed a commit
that referenced
this pull request
Aug 30, 2023
…e case (GH-108568) (#108691) gh-108520: Fix bad fork detection in nested multiprocessing use case (GH-108568) gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- (cherry picked from commit add8d45) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes both the example in the initial report #77377 and other reports on the PyTorch repo.
Note that as far as I can tell, all objects that lead to segfault with mixed context are all caused by SemLock created with fork sent over a Spawn process. That's why it is the only object updated here and the only pair of ctx covered. Let me know if I'm mistaken and I can find a way to update other objects as well.
Also windows behavior is not changed here (and thus not tested).
Please let me know if the error message and/or the test are not appropriate and I can work on improving them.