Use spawn() for multiprocessing instead of fork() (#1565)

* Use spawn() for starting new processes instead of fork()

* multiprocess: Explicitly use spawn on linux, use the default on mac

* Add changelog note for #1565

AUTHORS

@@ -27,3 +27,4 @@ Brooke Husic
 Tim Hempel
 Sander Roet
 Sebastian Falkner
+Finn Krein

doc/source/CHANGELOG.rst

@@ -19,7 +19,8 @@ Changelog
 - Use :code:`int` instead of :code:`numpy.int` to solve :code:`numpy` 1.20
   DeprecationWarning :pr:`1504`
 - Umbrella sampling: fix periodic width initialization :pr:`1522`
-- Solve 2 code:`SyntaxWarnings: "is" with a literal.` in utils/reader_utils.py :pr:`1530` 
+- Solve 2 code:`SyntaxWarnings: "is" with a literal.` in utils/reader_utils.py :pr:`1530`
+- Use spawn() for multiprocessing on linux instead of fork(), fixes worker processes sometimes going into idle for n_jobs>1 :pr:`1565`
 
 2.5.7 (9-24-2019)
 -----------------

pyemma/_base/estimator.py

@@ -22,6 +22,9 @@
 import sys
 import os
 
+from platform import system
+from multiprocess import get_context
+
 from threadpoolctl import threadpool_limits
 
 from pyemma._ext.sklearn.base import BaseEstimator as _BaseEstimator
@@ -339,8 +342,10 @@ def estimate_param_scan(estimator, X, param_sets, evaluate=None, evaluate_args=N
                       limit_threads)
                      for estimator, param_set in zip(estimators, param_sets))
 
-        from pathos.multiprocessing import Pool
-        pool = Pool(processes=n_jobs)
+        if system() == "Linux":
+            pool = get_context("spawn").Pool(processes=n_jobs)
+        else:
+            pool = get_context().Pool(processes=n_jobs)
         args = list(task_iter)
 
         from contextlib import closing