From 306a21ff31c55821d669a50e97470be4df227158 Mon Sep 17 00:00:00 2001 From: Valentin Lorentz Date: Sun, 20 Oct 2024 08:20:46 +0200 Subject: [PATCH 1/4] Doc: Explain that multiprocessing only supports unpicklable functions through the fork method In multiprocessing and concurrent.futures's documentations, as well as the release notes for Python 3.14. This moves the paragraph in 3.14's release notes from the deprecations section to 'Porting to Python 3.14' because it is a breaking change, which follows a deprecation in Python 3.12. This explicitly mentions `PicklingError` in this section, so it is easy to find for somehow who gets the error and searches its name through the page. --- Doc/library/concurrent.futures.rst | 3 ++- Doc/library/multiprocessing.rst | 7 +++++++ Doc/whatsnew/3.14.rst | 21 +++++++++++++-------- 3 files changed, 22 insertions(+), 9 deletions(-) diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index 45a73705f10e92..9a69c2546e6e67 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -412,7 +412,8 @@ to a :class:`ProcessPoolExecutor` will result in deadlock. .. versionchanged:: 3.14 The default process start method (see :ref:`multiprocessing-start-methods`) changed away from *fork*. If you - require the *fork* start method for :class:`ProcessPoolExecutor` you must + require the *fork* start method for :class:`ProcessPoolExecutor` or need to run + :ref:`unpicklable functions `, you must explicitly pass ``mp_context=multiprocessing.get_context("fork")``. .. _processpoolexecutor-example: diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 036b8f44b9ff3b..24267c6b16cc75 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -115,6 +115,8 @@ to start a process. These *start methods* are will not be inherited. Starting a process using this method is rather slow compared to using *fork* or *forkserver*. + Functions run with this method must be :ref:`picklable `. + Available on POSIX and Windows platforms. The default on Windows and macOS. *fork* @@ -124,6 +126,9 @@ to start a process. These *start methods* are inherited by the child process. Note that safely forking a multithreaded process is problematic. + Functions run with this method do *not* need to be + :ref:`picklable `. + Available on POSIX systems. .. versionchanged:: 3.14 @@ -146,6 +151,8 @@ to start a process. These *start methods* are side-effect so it is generally safe for it to use :func:`os.fork`. No unnecessary resources are inherited. + Functions run with this method must be :ref:`picklable `. + Available on POSIX platforms which support passing file descriptors over Unix pipes such as Linux. The default on those. diff --git a/Doc/whatsnew/3.14.rst b/Doc/whatsnew/3.14.rst index ad841538ccc547..68e7a71b8278bd 100644 --- a/Doc/whatsnew/3.14.rst +++ b/Doc/whatsnew/3.14.rst @@ -451,14 +451,6 @@ Deprecated as a single positional argument. (Contributed by Serhiy Storchaka in :gh:`109218`.) -* :mod:`multiprocessing` and :mod:`concurrent.futures`: - The default start method (see :ref:`multiprocessing-start-methods`) changed - away from *fork* to *forkserver* on platforms where it was not already - *spawn* (Windows & macOS). If you require the threading incompatible *fork* - start method you must explicitly specify it when using :mod:`multiprocessing` - or :mod:`concurrent.futures` APIs. - (Contributed by Gregory P. Smith in :gh:`84559`.) - * :mod:`os`: :term:`Soft deprecate ` :func:`os.popen` and :func:`os.spawn* ` functions. They should no longer be used to @@ -625,6 +617,8 @@ Others :meth:`~object.__index__`. (Contributed by Mark Dickinson in :gh:`119743`.) +.. _whatsnew314-porting-to-python314: + Porting to Python 3.14 ====================== @@ -643,6 +637,17 @@ Changes in the Python API This temporary change affects other threads. (Contributed by Serhiy Storchaka in :gh:`69998`.) +* :mod:`multiprocessing` and :mod:`concurrent.futures`: + The default start method (see :ref:`multiprocessing-start-methods`) changed + away from *fork* to *forkserver* on platforms where it was not already + *spawn* (Windows & macOS). If you require the threading incompatible *fork*, + which can run :ref:`unpicklable functions `, + start method you must explicitly specify it when using :mod:`multiprocessing` + or :mod:`concurrent.futures` APIs. + Code running pickable functions on these platforms without explicitly opting + into the *fork* method will now raise :exc:`pickle.PicklingError`. + (Contributed by Gregory P. Smith in :gh:`84559`.) + Build changes ============= From afeb1aad1fdf730287dd20457dbdce53e30e3f82 Mon Sep 17 00:00:00 2001 From: Valentin Lorentz Date: Sun, 20 Oct 2024 19:10:37 +0200 Subject: [PATCH 2/4] Explicit which platforms switch from fork to forkserver --- Doc/whatsnew/3.14.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.14.rst b/Doc/whatsnew/3.14.rst index 68e7a71b8278bd..3926f93e096d86 100644 --- a/Doc/whatsnew/3.14.rst +++ b/Doc/whatsnew/3.14.rst @@ -639,8 +639,10 @@ Changes in the Python API * :mod:`multiprocessing` and :mod:`concurrent.futures`: The default start method (see :ref:`multiprocessing-start-methods`) changed - away from *fork* to *forkserver* on platforms where it was not already - *spawn* (Windows & macOS). If you require the threading incompatible *fork*, + away from *fork* to *forkserver* on POSIX platforms which support passing file + descriptors over Unix pipes such as Linux. Windows and macOS are keep *spawn* + as their default method. + If you require the threading incompatible *fork*, which can run :ref:`unpicklable functions `, start method you must explicitly specify it when using :mod:`multiprocessing` or :mod:`concurrent.futures` APIs. From f3b37e402ec6977ca9a71040d918d54824e35bf1 Mon Sep 17 00:00:00 2001 From: Valentin Lorentz Date: Sun, 20 Oct 2024 19:10:41 +0200 Subject: [PATCH 3/4] Fix grammar --- Doc/whatsnew/3.14.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.14.rst b/Doc/whatsnew/3.14.rst index 3926f93e096d86..2eda5922fc9858 100644 --- a/Doc/whatsnew/3.14.rst +++ b/Doc/whatsnew/3.14.rst @@ -642,9 +642,9 @@ Changes in the Python API away from *fork* to *forkserver* on POSIX platforms which support passing file descriptors over Unix pipes such as Linux. Windows and macOS are keep *spawn* as their default method. - If you require the threading incompatible *fork*, + If you require the threading incompatible *fork* start method, which can run :ref:`unpicklable functions `, - start method you must explicitly specify it when using :mod:`multiprocessing` + you must explicitly specify it when using :mod:`multiprocessing` or :mod:`concurrent.futures` APIs. Code running pickable functions on these platforms without explicitly opting into the *fork* method will now raise :exc:`pickle.PicklingError`. From d43520730a18922ef2436d836582a7658b4720d0 Mon Sep 17 00:00:00 2001 From: Val Lorentz Date: Mon, 21 Oct 2024 07:40:29 +0200 Subject: [PATCH 4/4] Fix grammar Co-authored-by: Jelle Zijlstra --- Doc/whatsnew/3.14.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/whatsnew/3.14.rst b/Doc/whatsnew/3.14.rst index 2eda5922fc9858..5cbb050f564b85 100644 --- a/Doc/whatsnew/3.14.rst +++ b/Doc/whatsnew/3.14.rst @@ -640,13 +640,13 @@ Changes in the Python API * :mod:`multiprocessing` and :mod:`concurrent.futures`: The default start method (see :ref:`multiprocessing-start-methods`) changed away from *fork* to *forkserver* on POSIX platforms which support passing file - descriptors over Unix pipes such as Linux. Windows and macOS are keep *spawn* + descriptors over Unix pipes, such as Linux. Windows and macOS keep *spawn* as their default method. - If you require the threading incompatible *fork* start method, + If you require the threading-incompatible *fork* start method, which can run :ref:`unpicklable functions `, you must explicitly specify it when using :mod:`multiprocessing` or :mod:`concurrent.futures` APIs. - Code running pickable functions on these platforms without explicitly opting + Code running picklable functions on these platforms without explicitly opting into the *fork* method will now raise :exc:`pickle.PicklingError`. (Contributed by Gregory P. Smith in :gh:`84559`.)