gh-85299: Add note warning about entry point guard for asyncio example #93457

slateny · 2022-06-03T09:33:42Z

https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor

kumaraditya303 · 2022-10-13T13:43:20Z

Doc/library/asyncio-eventloop.rst

@@ -1208,6 +1208,12 @@ Executing code in thread or process pools

      asyncio.run(main())

+   Note that on macOS, an entry point guard may be needed as the example would


It would be better to just add __name__ == '__main__'. This happens on Windows too not just macOS.

That's a good point, Windows also defaults to spawn so looks like it runs into the same issue as mac. Should this note still remain as an fyi, or just change the example to fix the problem and leave out the note? It's customary to have the guard in general but knowing why it fails without might still be helpful, but I don't feel too strongly either way.

gvanrossum · 2022-10-13T17:59:50Z

Doc/library/asyncio-eventloop.rst

@@ -1208,6 +1208,12 @@ Executing code in thread or process pools

      asyncio.run(main())

+   Note that on macOS, an entry point guard may be needed as the example would


bedevere-bot · 2022-10-13T17:59:59Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

gvanrossum · 2022-10-14T16:14:52Z

Okay, I am beginning to understand the issue. It's due to the process pool example. If I comment out this block of code from the example:

    # 3. Run in a custom process pool:
    with concurrent.futures.ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, cpu_bound)
        print('custom process pool', result)

everything is fine. If I leave that in the run (on macOS) fails spectacularly with a RuntimeError that has many lines of traceback, then a multi-line error message, and then more traceback lines. Adding the __main__ guard makes this go away.

I presume that's one of the many reasons not to use ProcessPoolExecutor. :-(

I propose to rephrase the note to mention the platform and just link to https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods, not to the change note.

gvanrossum · 2022-10-15T23:17:16Z

Doc/library/asyncio-eventloop.rst

+   Note that on Windows and macOS, the entry point guard may be necessary as the
+   example would otherwise cause a :exc:`RuntimeError`. For more details, see the
+   `Safe importing of main module
+   <https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods>`_.


I propose to simplify the note to the following:

Note that the entry point guard (if __name__ == '__main__') is required for option 3, due to the peculiarities of multiprocessing, which is used by ProcessPoolExecutor. See (link).

There should be three cross-references:

multiprocessing (the module)

ProcessPoolExecutor (the class)

Safe importing of main module

For the latter I recommend to add a label to the target section in multiprocessing.rst rather than linking to the website. (@CAM-Gerlach can provide details on how to do this.)

My understanding though is that option 3 fails when spawn/forkserver is the process start method, and only Windows + mac default to spawn, while unix defaults to fork, so saying 'is required' may be a bit too strong. If 'may be required' instead, then the reader might wonder why 'may', in which case I think a link to https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods somewhere would help explain things.

That's too subtle for the readers. We're not stating the law, we're just explaining the example.

See my comment for a suggestion implementing the above and the details Guido mentioned.

Doc/library/asyncio-eventloop.rst

Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>

gvanrossum

Thanks!

miss-islington · 2022-10-16T17:06:16Z

Thanks @slateny for the PR, and @gvanrossum for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

miss-islington · 2022-10-16T17:06:16Z

Thanks @slateny for the PR, and @gvanrossum for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

…example (pythonGH-93457) (cherry picked from commit 79fd6cc) Co-authored-by: Stanley <46876382+slateny@users.noreply.github.com>

bedevere-bot · 2022-10-16T17:06:26Z

GH-98334 is a backport of this pull request to the 3.11 branch.

gvanrossum · 2022-10-16T17:06:31Z

Let’s see if we can backport this.

bedevere-bot · 2022-10-16T17:06:31Z

GH-98335 is a backport of this pull request to the 3.10 branch.

…example (pythonGH-93457) (cherry picked from commit 79fd6cc) Co-authored-by: Stanley <46876382+slateny@users.noreply.github.com>

GH-93457) (cherry picked from commit 79fd6cc) Co-authored-by: Stanley <46876382+slateny@users.noreply.github.com>

* main: (31 commits) pythongh-95913: Move subinterpreter exper removal to 3.11 WhatsNew (pythonGH-98345) pythongh-95914: Add What's New item describing PEP 670 changes (python#98315) Remove unused arrange_output_buffer function from zlibmodule.c. (pythonGH-98358) pythongh-98174: Handle EPROTOTYPE under macOS in test_sendfile_fallback_close_peer_in_the_middle_of_receiving (python#98316) pythonGH-98327: Reduce scope of catch_warnings() in _make_subprocess_transport (python#98333) pythongh-93691: Compiler's code-gen passes location around instead of holding it on the global compiler state (pythonGH-98001) pythongh-97669: Create Tools/build/ directory (python#97963) pythongh-95534: Improve gzip reading speed by 10% (python#97664) pythongh-95913: Forward-port int/str security change to 3.11 What's New in main (python#98344) pythonGH-91415: Mention alphabetical sort ordering in the Sorting HOWTO (pythonGH-98336) pythongh-97930: Merge with importlib_resources 5.9 (pythonGH-97929) pythongh-85525: Remove extra row in doc (python#98337) pythongh-85299: Add note warning about entry point guard for asyncio example (python#93457) pythongh-97527: IDLE - fix buggy macosx patch (python#98313) pythongh-98307: Add docstring and documentation for SysLogHandler.createSocket (pythonGH-98319) pythongh-94808: Cover `PyFunction_GetCode`, `PyFunction_GetGlobals`, `PyFunction_GetModule` (python#98158) pythonGH-94597: Deprecate child watcher getters and setters (python#98215) pythongh-98254: Include stdlib module names in error messages for NameErrors (python#98255) Improve speed. Reduce auxiliary memory to 16.6% of the main array. (pythonGH-98294) [doc] Update logging cookbook with an example of custom handling of levels. (pythonGH-98290) ...

GH-93457) (cherry picked from commit 79fd6cc) Co-authored-by: Stanley <46876382+slateny@users.noreply.github.com>

Add note warning about entry point guard for asyncio example

97d022e

slateny requested review from 1st1 and asvetlov as code owners June 3, 2022 09:33

bedevere-bot added docs Documentation in the Doc dir awaiting review labels Jun 3, 2022

slateny added the skip news label Jun 3, 2022

kumaraditya303 reviewed Oct 13, 2022

View reviewed changes

kumaraditya303 requested review from kumaraditya303 and gvanrossum October 13, 2022 13:43

gvanrossum requested changes Oct 13, 2022

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels Oct 13, 2022

Add entry guard, clarify note on platform

16cb255

gvanrossum reviewed Oct 15, 2022

View reviewed changes

CAM-Gerlach reviewed Oct 16, 2022

View reviewed changes

Doc/library/asyncio-eventloop.rst Outdated Show resolved Hide resolved

slateny and others added 3 commits October 16, 2022 06:06

Add ref target in multiprocessing

0129bdf

Use rewording for multiprocessing

fd68a5e

Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>

Add missing grave sign

8ba4b0c

gvanrossum approved these changes Oct 16, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting changes labels Oct 16, 2022

gvanrossum merged commit 79fd6cc into python:main Oct 16, 2022

bedevere-bot removed the awaiting merge label Oct 16, 2022

gvanrossum added needs backport to 3.10 only security fixes needs backport to 3.11 only security fixes labels Oct 16, 2022

bedevere-bot removed the needs backport to 3.11 only security fixes label Oct 16, 2022

bedevere-bot removed the needs backport to 3.10 only security fixes label Oct 16, 2022

miss-islington added a commit that referenced this pull request Oct 16, 2022

gh-85299: Add note warning about entry point guard for asyncio example (

2630738

GH-93457) (cherry picked from commit 79fd6cc) Co-authored-by: Stanley <46876382+slateny@users.noreply.github.com>

miss-islington added a commit that referenced this pull request Oct 16, 2022

gh-85299: Add note warning about entry point guard for asyncio example (

e26c526

GH-93457) (cherry picked from commit 79fd6cc) Co-authored-by: Stanley <46876382+slateny@users.noreply.github.com>

slateny deleted the s/85299 branch October 16, 2022 20:40

pablogsal pushed a commit that referenced this pull request Oct 22, 2022

gh-85299: Add note warning about entry point guard for asyncio example (

a2820a0

GH-93457) (cherry picked from commit 79fd6cc) Co-authored-by: Stanley <46876382+slateny@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-85299: Add note warning about entry point guard for asyncio example #93457

gh-85299: Add note warning about entry point guard for asyncio example #93457

slateny commented Jun 3, 2022

kumaraditya303 Oct 13, 2022

gvanrossum Oct 13, 2022

slateny Oct 14, 2022

gvanrossum Oct 13, 2022

bedevere-bot commented Oct 13, 2022

gvanrossum commented Oct 14, 2022

gvanrossum Oct 15, 2022 •

edited

slateny Oct 16, 2022

gvanrossum Oct 16, 2022

CAM-Gerlach Oct 16, 2022

gvanrossum left a comment

miss-islington commented Oct 16, 2022

miss-islington commented Oct 16, 2022

bedevere-bot commented Oct 16, 2022

gvanrossum commented Oct 16, 2022

bedevere-bot commented Oct 16, 2022

		@@ -1208,6 +1208,12 @@ Executing code in thread or process pools

		asyncio.run(main())

		Note that on macOS, an entry point guard may be needed as the example would

gh-85299: Add note warning about entry point guard for asyncio example #93457

gh-85299: Add note warning about entry point guard for asyncio example #93457

Conversation

slateny commented Jun 3, 2022

kumaraditya303 Oct 13, 2022

Choose a reason for hiding this comment

gvanrossum Oct 13, 2022

Choose a reason for hiding this comment

slateny Oct 14, 2022

Choose a reason for hiding this comment

gvanrossum Oct 13, 2022

Choose a reason for hiding this comment

bedevere-bot commented Oct 13, 2022

gvanrossum commented Oct 14, 2022

gvanrossum Oct 15, 2022 • edited

Choose a reason for hiding this comment

slateny Oct 16, 2022

Choose a reason for hiding this comment

gvanrossum Oct 16, 2022

Choose a reason for hiding this comment

CAM-Gerlach Oct 16, 2022

Choose a reason for hiding this comment

gvanrossum left a comment

Choose a reason for hiding this comment

miss-islington commented Oct 16, 2022

miss-islington commented Oct 16, 2022

bedevere-bot commented Oct 16, 2022

gvanrossum commented Oct 16, 2022

bedevere-bot commented Oct 16, 2022

gvanrossum Oct 15, 2022 •

edited