Fix multiprocessing timeout

[zanieb]

Summary

The value retrieval in the multiprocessing_timeout was using a locally defined helper function
which could not be serialized by the native Python pickler resulting in an error running tasks with
a timeout while using a process based scheduler.

Example

from prefect import task, Flow
from prefect.engine.executors import LocalDaskExecutor
from random import randint

@task
def foo(x):
    return [randint(0, i) for i in range(x)]

@task(timeout=10)
def bar(y):
    return y * 2

with Flow("foobar") as flow:
    y = foo(10)
    z = bar.map(y) 

if __name__ == "__main__":
    state = flow.run(executor=LocalDaskExecutor())

Results in the error AttributeError: Can't pickle local object 'multiprocessing_timeout.<locals>.retrieve_value'

Also fixes a timeout issue with function pickling similar to #2634

Changes

• Moves the multiprocessing_timeout.retrieve_value function to a top-level function
• Renames retrieve_value to multiprocessing_safe_retrieve_value
• multiprocessing_safe_retrieve_value takes a cloudpickle serialized callable so tasks defined in scripts can be passed
• Adds the mproc and mproc_local executors to the flow timeout test
• Minor documentation improvements in the multiprocessing_timeout function
• Adds a __processes attribute to distributed test executors
• Adds a local multiprocess test executor
• Fixes the synchronous test executor to use the correct scheduler
• Adds test coverage for daemon process / soft limit timeout enforcement

Importance

• Fixes a bug in multiprocessing timeouts
• Improves user experience when defining timeout tasks in scripts

Checklist

This PR:

• adds new tests (if appropriate)
• adds a change file in the changes/ directory (if appropriate)
• updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)

[zanieb]

The referenced pickling error does not appear to occur on Linux just OSX 🤷

[jcrist]

Nice find, I'll give this a longer review after lunch.

The referenced pickling error does not appear to occur on Linux just OSX 🤷

In python 3.8 osx switched to using spawn by default, while linux still uses fork (where pickling won't be needed).

[codecov]

Codecov Report

Merging #3511 into master will decrease coverage by 0.07%.
The diff coverage is 45.83%.

[jcrist]

One final tweak and this looks good to me.