multiprocessing docs bury very important 'callback=' args

[chadmiller-amzn]

BPO	33082
Nosy	@pitrou, @ned-deily, @applio, @JulienPalard, @Mariatta, @chadmiller-amzn
PRs	bpo-33082: Document callback= in 'multiprocessing' functions #6122 bpo-33082: Include multiprocessing Pool callback in example & docstrings #6455

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-09-13.17:16:38.320>
created_at = <Date 2018-03-15.18:50:50.020>
labels = ['3.8', 'type-feature', '3.7', 'docs']
title = "multiprocessing docs bury very important 'callback=' args"
updated_at = <Date 2019-09-13.17:16:38.319>
user = 'https://github.com/chadmiller-amzn'

bugs.python.org fields:

activity = <Date 2019-09-13.17:16:38.319>
actor = 'davin'
assignee = 'docs@python'
closed = True
closed_date = <Date 2019-09-13.17:16:38.320>
closer = 'davin'
components = ['Documentation']
creation = <Date 2018-03-15.18:50:50.020>
creator = 'chadmiller-amzn'
dependencies = []
files = []
hgrepos = []
issue_num = 33082
keywords = ['patch']
message_count = 9.0
messages = ['313905', '313906', '313915', '314162', '314267', '319100', '352366', '352367', '352375']
nosy_count = 7.0
nosy_names = ['pitrou', 'ned.deily', 'docs@python', 'davin', 'mdk', 'Mariatta', 'chadmiller-amzn']
pr_nums = ['6122', '6455']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue33082'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[chadmiller-amzn]

Callbacks are really important in multiprocessing. Doc writer almost ignores them.

[chadmiller-amzn]

chadmiller-amzn#1

(Putting that in "GitHub PR" field says "Edit Error: Unknown PR format, acceptable formats are: full github URL, #pr_number, pr_number")

[Mariatta]

Can you make your PR in python/cpython GitHub repo, instead of your own fork?

[pitrou]

I don't really agree that "callbacks are really important in multiprocessing" (I think I've hardly ever used them). I agree with the other doc changes in the PR.

[chadmiller-amzn]

On topic: My CLA is signed as of Monday, 19 March. My status here is not updated yet.

pitrou, off-topic: Without callbacks, users who want multiprocessing functions to return something, not just mutate state somewhere else, must gather jobs in a list and continually iterate through them polling to see if each has finished yet and conditionally popping it from the list. It's expensive and ugly and error-prone. Callbacks are really great, you should try them again.

So much better:

pool.apply_async(func, args, callback=when_finished_call_with_result)

[ned-deily]

These doc changes have been languishing. We have a difference of opinion about adding a callback example. I don't have a strong feeling one way or the other. Davin or anyone else, what say you?

[JulienPalard]

I agree Antoine on this one, if one want the result, It'll get it from the returned value (.get method in the example), or simply by using the not-async versions and directly get the results as a return value of the call.

Also the given example in the documentation won't work as the result is never waited for, the pool gets destroyed before the sleep have the time to sleep.

[JulienPalard]

I'd wait for Davin's review, but I'd keep the documentation as they are.

[applio]

I appreciate the functionality offered by the callbacks and have found good uses for them, as Chad clearly does/has. That said, the thought of expanding the documentation on the callbacks had not come up for me.

Reading through the proposed changes to the prose explanations, the choice of words has changed but not significantly and virtually no new concepts are being explained.

I agree with Julien that the docs should stay as they are.

Chad: Thank you for advocating for things you think more people need to know about even if we do not update the docs this time.