TST: Use safer context for ProcessPoolExecutor #8715
Conversation
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 29 files ±0 29 suites ±0 11h 27m 43s ⏱️ - 7m 33s For more details on these failures, see this check. Results for commit efb95e4. ± Comparison against base commit 5b2e2b4. ♻️ This comment has been updated with latest results. |
distributed/tests/test_worker.py
Outdated
| @@ -2199,7 +2200,7 @@ async def test_bad_executor_annotation(c, s, a, b): | |||
|
|
|||
| @gen_cluster(client=True) | |||
| async def test_process_executor(c, s, a, b): | |||
| with ProcessPoolExecutor() as e: | |||
| with ProcessPoolExecutor(mp_context=multiprocessing.get_context("spawn")) as e: | |||
There was a problem hiding this comment.
Might be best to use distributed.utils.get_mp_context which picks the context that is being configured by default. It also sets some preload things for a forkserver which isn't truly relevant. However, this would make the tests sensitive to what we're setting as defaults which sounds like a good thing to have
When running `test_process_executor_kills_process` just by itself, I get a warning that threads have been used and that the `fork` multiprocessing context is unsafe. If I run the full test suite, then the same test hangs. I don't know where the threads are coming from (as I didn't use xdist), but explicitly use distributed's default context (which is the safer `spawn` multiprocess context) to avoid these issues.
When running
test_process_executor_kills_processjust by itself, I get a warning that threads have been used and that theforkmultiprocessing context is unsafe. If I run the full test suite, then the same test hangs.I don't know where the threads are coming from (as I didn't use xdist), but explicitly use the safer
spawnmultiprocess context to avoid these issues.pre-commit run --all-files