gh-98552: Fix preloading '__main__' with forkserver being broken for a #99515
+118
−30
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n for a long time (9 years). We do this by using spawn.get_preparation_data() and then using the values in that dictionary appropriately. This lets us also fix the setting of sys.path (which never worked) and other minor properties of the forkserver process. While updating the test to verify these fixes, we also discovered that forkserver was not flushing stdout/stderr before it forked to create a new process. This caused the updated test to fail because unflushed output in the forkserver would then be printed by the forked process as well. This is now fixed too.
aggieNick02
requested review from
1st1,
brettcannon,
encukou,
ericsnowcurrently,
ncoghlan and
warsaw
as code owners
Author
|
One minor thing I noticed incase @pitrou remembers from several years ago. In |
Member
|
@aggieNick02 I suppose it's a simple check that |
Author
Ah ok, thanks! |
Contributor
Member
|
this one is obsolete due to the preloading of main being fixed. but the question of the buffering issue remains, lets take that up on the issue; there may be a new, now smaller, PR worth doing. |
gpshead
pushed a commit
to gpshead/cpython
that referenced
this pull request
Nov 22, 2025
This change adds a keyword-only `raise_exceptions` parameter to `multiprocessing.set_forkserver_preload()` that defaults to False for backward compatibility. When set to True, ImportError exceptions during module preloading in the forkserver process will be raised instead of being silently ignored. This is similar in spirit to the approach attempted in pythonGH-99515, providing developers with the ability to catch import errors during forkserver module preloading for better debugging and error handling. Changes: - Add _raise_exceptions attribute to ForkServer class - Update set_forkserver_preload() to accept raise_exceptions parameter - Pass raise_exceptions flag through to forkserver main() function - Update main() to conditionally raise ImportError based on flag - Add comprehensive test coverage in test_multiprocessing_forkserver - Update documentation in Doc/library/multiprocessing.rst - Add NEWS entry for Python 3.14
gpshead
pushed a commit
to gpshead/cpython
that referenced
this pull request
Nov 22, 2025
This change adds a keyword-only `raise_exceptions` parameter to `multiprocessing.set_forkserver_preload()` that defaults to False for backward compatibility. When set to True, ImportError exceptions during module preloading in the forkserver process will be raised instead of being silently ignored. This is similar in spirit to the approach attempted in pythonGH-99515, providing developers with the ability to catch import errors during forkserver module preloading for better debugging and error handling. Changes: - Add _raise_exceptions attribute to ForkServer class - Update set_forkserver_preload() to accept raise_exceptions parameter - Pass raise_exceptions flag through to forkserver main() function - Update main() to conditionally raise ImportError based on flag - Add comprehensive test coverage in test_multiprocessing_forkserver - Update documentation in Doc/library/multiprocessing.rst - Add NEWS entry for Python 3.14
gpshead
pushed a commit
to gpshead/cpython
that referenced
this pull request
Nov 22, 2025
This change adds a keyword-only `raise_exceptions` parameter to `multiprocessing.set_forkserver_preload()` that defaults to False for backward compatibility. When set to True, ImportError exceptions during module preloading in the forkserver process will be raised instead of being silently ignored. This is similar in spirit to the approach attempted in pythonGH-99515, providing developers with the ability to catch import errors during forkserver module preloading for better debugging and error handling. Changes: - Add _raise_exceptions attribute to ForkServer class - Update set_forkserver_preload() to accept raise_exceptions parameter - Pass raise_exceptions flag through to forkserver main() function - Update main() to conditionally raise ImportError based on flag - Add comprehensive test coverage in test_multiprocessing_forkserver - Update documentation in Doc/library/multiprocessing.rst - Add NEWS entry for Python 3.14
gpshead
pushed a commit
to gpshead/cpython
that referenced
this pull request
Nov 22, 2025
This change adds a keyword-only `raise_exceptions` parameter to `multiprocessing.set_forkserver_preload()` that defaults to False for backward compatibility. When set to True, ImportError exceptions during module preloading in the forkserver process will be raised instead of being silently ignored. This is similar in spirit to the approach attempted in pythonGH-99515, providing developers with the ability to catch import errors during forkserver module preloading for better debugging and error handling. Changes: - Add _raise_exceptions attribute to ForkServer class - Update set_forkserver_preload() to accept raise_exceptions parameter - Pass raise_exceptions flag through to forkserver main() function - Update main() to conditionally raise ImportError based on flag - Add comprehensive test coverage in test_multiprocessing_forkserver - Update documentation in Doc/library/multiprocessing.rst - Add NEWS entry for Python 3.14
gpshead
pushed a commit
to gpshead/cpython
that referenced
this pull request
Nov 22, 2025
This change adds a keyword-only `raise_exceptions` parameter to `multiprocessing.set_forkserver_preload()` that defaults to False for backward compatibility. When set to True, ImportError exceptions during module preloading in the forkserver process will be raised instead of being silently ignored. This is similar in spirit to the approach attempted in pythonGH-99515, providing developers with the ability to catch import errors during forkserver module preloading for better debugging and error handling. Changes: - Add _raise_exceptions attribute to ForkServer class - Update set_forkserver_preload() to accept raise_exceptions parameter - Pass raise_exceptions flag through to forkserver main() function - Update main() to conditionally raise ImportError based on flag - Add comprehensive test coverage in test_multiprocessing_forkserver - Update documentation in Doc/library/multiprocessing.rst - Add NEWS entry for Python 3.14
gpshead
pushed a commit
to gpshead/cpython
that referenced
this pull request
Nov 22, 2025
This change adds a keyword-only `raise_exceptions` parameter to `multiprocessing.set_forkserver_preload()` that defaults to False for backward compatibility. When set to True, ImportError exceptions during module preloading in the forkserver process will be raised instead of being silently ignored. This is similar in spirit to the approach attempted in pythonGH-99515, providing developers with the ability to catch import errors during forkserver module preloading for better debugging and error handling. Changes: - Add _raise_exceptions attribute to ForkServer class - Update set_forkserver_preload() to accept raise_exceptions parameter - Pass raise_exceptions flag through to forkserver main() function - Update main() to conditionally raise ImportError based on flag - Add comprehensive test coverage in test_multiprocessing_forkserver - Update documentation in Doc/library/multiprocessing.rst - Add NEWS entry for Python 3.14
gpshead
pushed a commit
to gpshead/cpython
that referenced
this pull request
Nov 22, 2025
This change adds a keyword-only `raise_exceptions` parameter to `multiprocessing.set_forkserver_preload()` that defaults to False for backward compatibility. When set to True, ImportError exceptions during module preloading in the forkserver process will be raised instead of being silently ignored, causing the forkserver to exit and breaking all use of the forkserver multiprocessing context. This is similar in spirit to the approach attempted in pythonGH-99515, providing developers with the ability to catch import errors during forkserver module preloading for better debugging and error handling. Implementation: - Add _raise_exceptions attribute to ForkServer class - Update set_forkserver_preload() in both forkserver.py and context.py to accept raise_exceptions keyword-only parameter - Pass raise_exceptions flag through to forkserver main() function - Update main() to conditionally raise ImportError based on flag Tests: - Create new test_multiprocessing_forkserver/test_preload.py with dedicated tests for raise_exceptions functionality - Remove old test from _test_multiprocessing.py - Add tests for both True/False cases and valid/invalid modules - Ensure proper resource cleanup using try/finally Documentation: - Update Doc/library/multiprocessing.rst with detailed description of raise_exceptions behavior and consequences - Add NEWS entry with contributor attribution - Add Nick Neumann to Misc/ACKS Contributed by Nick Neumann.
gpshead
pushed a commit
to gpshead/cpython
that referenced
this pull request
Nov 22, 2025
Add a keyword-only `raise_exceptions` parameter (default False) to `multiprocessing.set_forkserver_preload()`. When True, ImportError exceptions during module preloading cause the forkserver to exit, breaking all use of the forkserver multiprocessing context. This allows developers to catch import errors during development rather than having them silently ignored. Implementation adds the parameter to both the ForkServer class method and the BaseContext wrapper, passing it through to the forkserver main() function which conditionally raises ImportError instead of ignoring it. Tests are in new test_multiprocessing_forkserver/test_preload.py with proper resource cleanup using try/finally. Documentation describes the behavior, consequences (forkserver exit, EOFError/ConnectionError on subsequent use), and recommends use during development. Based on original work by Nick Neumann in pythonGH-99515. Contributed by Nick Neumann.
gpshead
added a commit
to gpshead/cpython
that referenced
this pull request
Nov 22, 2025
Add a keyword-only `raise_exceptions` parameter (default False) to `multiprocessing.set_forkserver_preload()`. When True, ImportError exceptions during module preloading cause the forkserver to exit, breaking all use of the forkserver multiprocessing context. This allows developers to catch import errors during development rather than having them silently ignored. Implementation adds the parameter to both the ForkServer class method and the BaseContext wrapper, passing it through to the forkserver main() function which conditionally raises ImportError instead of ignoring it. Tests are in new test_multiprocessing_forkserver/test_preload.py with proper resource cleanup using try/finally. Documentation describes the behavior, consequences (forkserver exit, EOFError/ConnectionError on subsequent use), and recommends use during development. Based on original work by Nick Neumann in pythonGH-99515. Contributed by Nick Neumann. Co-authored-by: Gregory P. Smith <greg@krypto.org>
gpshead
added a commit
to gpshead/cpython
that referenced
this pull request
Nov 22, 2025
Add a keyword-only `raise_exceptions` parameter (default False) to `multiprocessing.set_forkserver_preload()`. When True, ImportError exceptions during module preloading cause the forkserver to exit, breaking all use of the forkserver multiprocessing context. This allows developers to catch import errors during development rather than having them silently ignored. Implementation adds the parameter to both the ForkServer class method and the BaseContext wrapper, passing it through to the forkserver main() function which conditionally raises ImportError instead of ignoring it. Tests are in new test_multiprocessing_forkserver/test_preload.py with proper resource cleanup using try/finally. Documentation describes the behavior, consequences (forkserver exit, EOFError/ConnectionError on subsequent use), and recommends use during development. Based on original work by Nick Neumann in pythonGH-99515. Contributed by Nick Neumann. Co-authored-by: Gregory P. Smith <greg@krypto.org>
gpshead
added a commit
to gpshead/cpython
that referenced
this pull request
Nov 23, 2025
Add a keyword-only `raise_exceptions` parameter (default False) to `multiprocessing.set_forkserver_preload()`. When True, ImportError exceptions during module preloading cause the forkserver to exit, breaking all use of the forkserver multiprocessing context. This allows developers to catch import errors during development rather than having them silently ignored. Implementation adds the parameter to both the ForkServer class method and the BaseContext wrapper, passing it through to the forkserver main() function which conditionally raises ImportError instead of ignoring it. Tests are in new test_multiprocessing_forkserver/test_preload.py with proper resource cleanup using try/finally. Documentation describes the behavior, consequences (forkserver exit, EOFError/ConnectionError on subsequent use), and recommends use during development. Based on original work by Nick Neumann in pythonGH-99515. Contributed by Nick Neumann. Co-authored-by: Gregory P. Smith <greg@krypto.org>
gpshead
added a commit
to gpshead/cpython
that referenced
this pull request
Nov 23, 2025
Add a keyword-only `raise_exceptions` parameter (default False) to `multiprocessing.set_forkserver_preload()`. When True, ImportError exceptions during module preloading cause the forkserver to exit, breaking all use of the forkserver multiprocessing context. This allows developers to catch import errors during development rather than having them silently ignored. Implementation adds the parameter to both the ForkServer class method and the BaseContext wrapper, passing it through to the forkserver main() function which conditionally raises ImportError instead of ignoring it. Tests are in new test_multiprocessing_forkserver/test_preload.py with proper resource cleanup using try/finally. Documentation describes the behavior, consequences (forkserver exit, EOFError/ConnectionError on subsequent use), and recommends use during development. Based on original work by Nick Neumann in pythonGH-99515. Contributed by Nick Neumann. Co-authored-by: aggieNick02 <nick@pcpartpicker.com> Co-authored-by: Claude (Sonnet 4.5) <claude@anthropic.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
gpshead
added a commit
to gpshead/cpython
that referenced
this pull request
Nov 23, 2025
Add a keyword-only `raise_exceptions` parameter (default False) to `multiprocessing.set_forkserver_preload()`. When True, ImportError exceptions during module preloading cause the forkserver to exit, breaking all use of the forkserver multiprocessing context. This allows developers to catch import errors during development rather than having them silently ignored. Implementation adds the parameter to both the ForkServer class method and the BaseContext wrapper, passing it through to the forkserver main() function which conditionally raises ImportError instead of ignoring it. Tests are in new test_multiprocessing_forkserver/test_preload.py with proper resource cleanup using try/finally. Documentation describes the behavior, consequences (forkserver exit, EOFError/ConnectionError on subsequent use), and recommends use during development. Based on original work by Nick Neumann in pythonGH-99515. Contributed by Nick Neumann. Co-authored-by: aggieNick02 <nick@pcpartpicker.com> Co-authored-by: Claude (Sonnet 4.5) <noreply@anthropic.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
long time (9 years).
We do this by using spawn.get_preparation_data() and then using the values in that dictionary appropriately. This lets us also fix the setting of sys.path (which never worked) and other minor properties of the forkserver process.
While updating the test to verify these fixes, we also discovered that forkserver was not flushing stdout/stderr before it forked to create a new process. This caused the updated test to fail because unflushed output in the forkserver would then be printed by the forked process as well. This is now fixed too.