[FIX] LegacyMultiProc hangs up indefinitely

[oesteban]

This PR relates to #2700, and should fix the problem
underlying #2548.

I first considered adding a control thread that monitors
the Pool of workers, but that would require a large overhead
keeping track of PIDs and polling very often.

Just adding the core file of bpo-22393
should fix #2548

Summary

Fixes # .

List of changes proposed in this PR (pull-request)

Acknowledgment

• (Mandatory) I acknowledge that this contribution will be available under the Apache 2 license.

[oesteban]

(I haven't tested this yet, though)

[effigies]

As the Python 3.8 version of this, I'm not sure how well this will play with 2.7 and 3.4, at the least.

[oesteban]

Yep, unfortunately, the fix is python +3.7. I'd say we start recommending switching to python 3.7 if the workflow is memory-hungry.

[effigies]

Given the recent numpy bug, I'm hesitant to suggest moving scientific applications to 3.7 until upstream projects have had a bit more time to find and fix critical bugs.

How hard would this be to port to 3.6? I suspect if we get it working there, 3.5 and 3.4 will fall into place quickly.

I'm okay with abandoning 2.7, as long as we don't break current usage further.

[oesteban]

Some thoughts:

• As per your link, anaconda fixed the bug in numpy 1.15.2, and it is fixed with 1.15.3 with Pypi. I guess we should update the pinned versions if python == 3.7.
• I'm not proposing to discontinue python <3.7, I would just make this fix available for python +3.7.
• If nothing of the above works, then I'd say it should not be too hard to backport to 3.6 or even 3.5 (prior to 3.5 it would be harder). Although I'm not sure it is worthwhile.

[codecov-io]

Codecov Report

Merging #2773 into master will decrease coverage by 3.4%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master   #2773      +/-   ##
=========================================
- Coverage    67.5%   64.1%   -3.41%     
=========================================
  Files         340     338       -2     
  Lines       43271   43223      -48     
  Branches     5364    5362       -2     
=========================================
- Hits        29210   27707    -1503     
- Misses      13360   14446    +1086     
- Partials      701    1070     +369

Flag	Coverage Δ
#smoketests	?
#unittests	64.1% <100%> (-0.82%)	⬇️

Impacted Files	Coverage Δ
nipype/pipeline/plugins/legacymultiproc.py	65.51% <100%> (+0.51%)	⬆️
nipype/interfaces/nilearn.py	40% <0%> (-56.67%)	⬇️
nipype/utils/spm_docs.py	25.92% <0%> (-44.45%)	⬇️
nipype/interfaces/freesurfer/base.py	49.59% <0%> (-30.9%)	⬇️
nipype/utils/logger.py	59.7% <0%> (-29.86%)	⬇️
nipype/algorithms/rapidart.py	35.39% <0%> (-29.21%)	⬇️
nipype/interfaces/spm/base.py	58.41% <0%> (-29.05%)	⬇️
nipype/utils/provenance.py	55.73% <0%> (-28.99%)	⬇️
nipype/interfaces/fsl/model.py	55.26% <0%> (-25.35%)	⬇️
nipype/testing/fixtures.py	77.33% <0%> (-21.34%)	⬇️
... and 42 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 199d9a6...b220c86. Read the comment docs.