[manifest] Explicitly terminate the multiprocessing.Pool #27175
Merged
Conversation
b80fbbf
to
9df5a58
Python requires that a Pool object be terminated before it is GC'd (https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool): ``` Warning: multiprocessing.pool objects have internal resources that need to be properly managed (like any other resource) by using the pool as a context manager or by calling close() and terminate() manually. Failure to do this can lead to the process hanging on finalization. Note that is not correct to rely on the garbage colletor to destroy the pool as CPython does not assure that the finalizer of the pool will be called (see object.__del__() for more information). ``` Chromium has been seeing hangs in 'wpt manifest' invocations since moving to Python 3 for WPT, which sounds a lot like 'Failure to do this can lead to the process hanging on finalization.' So let's terminate our Pool properly :)
9df5a58
to
eae6a51
eb6aa97
into
master
48 checks passed
48 checks passed
Azure Pipelines (affected tests without changes: Safari Technology Preview)
affected tests without changes: Safari Technology Preview succeeded
Details
Azure Pipelines (affected tests: Safari Technology Preview)
affected tests: Safari Technology Preview succeeded
Details
Azure Pipelines (tools/ unittests: Windows + Python 3.6)
tools/ unittests: Windows + Python 3.6 succeeded
Details
Azure Pipelines (tools/ unittests: Windows + Python 3.8)
tools/ unittests: Windows + Python 3.8 succeeded
Details
Azure Pipelines (tools/ unittests: macOS + Python 3.6)
tools/ unittests: macOS + Python 3.6 succeeded
Details
Azure Pipelines (tools/ unittests: macOS + Python 3.8)
tools/ unittests: macOS + Python 3.8 succeeded
Details
Azure Pipelines (tools/wpt/ tests: Windows + Python 3.6)
tools/wpt/ tests: Windows + Python 3.6 succeeded
Details
Azure Pipelines (tools/wpt/ tests: Windows + Python 3.8)
tools/wpt/ tests: Windows + Python 3.8 succeeded
Details
Azure Pipelines (tools/wpt/ tests: macOS + Python 3.6)
tools/wpt/ tests: macOS + Python 3.6 succeeded
Details
Azure Pipelines (tools/wpt/ tests: macOS + Python 3.8)
tools/wpt/ tests: macOS + Python 3.8 succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: Windows + Python 3.6)
tools/wptrunner/ unittests: Windows + Python 3.6 succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: Windows + Python 3.8)
tools/wptrunner/ unittests: Windows + Python 3.8 succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: Windows)
tools/wptrunner/ unittests: Windows succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: macOS + Python 3.6)
tools/wptrunner/ unittests: macOS + Python 3.6 succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: macOS + Python 3.8)
tools/wptrunner/ unittests: macOS + Python 3.8 succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: macOS)
tools/wptrunner/ unittests: macOS succeeded
Details
Azure Pipelines (wpt.fyi hook: safari-preview-affected-tests)
wpt.fyi hook: safari-preview-affected-tests succeeded
Details
Azure Pipelines (wpt.fyi hook: safari-preview-affected-tests-without-changes)
wpt.fyi hook: safari-preview-affected-tests-without-changes succeeded
Details
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Python requires that a Pool object be terminated before it is GC'd
(https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool):
Chromium has been seeing hangs in 'wpt manifest' invocations since
moving to Python 3 for WPT, which sounds a lot like 'Failure to do this
can lead to the process hanging on finalization.' So let's
terminate our Pool properly :)