ERROR: sklearn.tests.test_pipeline.test_feature_union_parallel

[chenyuqing]

When I test the scikit-learn installation , it occurs the error message like that:

======================================================================
ERROR: sklearn.externals.joblib.test.test_parallel.test_pre_dispatch_race_condit
ion
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\Python27\lib\site-packages\nose-1.3.4-py2.7.egg\nose\case.py", line 1
97, in runTest
    self.test(*self.arg)
  File "E:\Python27\lib\site-packages\sklearn\externals\joblib\test\test_paralle
l.py", line 335, in test_pre_dispatch_race_condition
    delayed(square)(i) for i in range(n_tasks))
  File "E:\Python27\lib\site-packages\sklearn\externals\joblib\parallel.py", lin
e 672, in __call__
    self._pool.terminate()  # terminate does a join()
  File "E:\Python27\lib\site-packages\sklearn\externals\joblib\pool.py", line 56
6, in terminate
    super(MemmapingPool, self).terminate()
  File "E:\Python27\lib\multiprocessing\pool.py", line 456, in terminate
    self._terminate()
  File "E:\Python27\lib\multiprocessing\util.py", line 207, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "E:\Python27\lib\multiprocessing\pool.py", line 504, in _terminate_pool
    p.terminate()
  File "E:\Python27\lib\multiprocessing\process.py", line 137, in terminate
    self._popen.terminate()
  File "E:\Python27\lib\multiprocessing\forking.py", line 312, in terminate
    _subprocess.TerminateProcess(int(self._handle), TERMINATE)
WindowsError: [Error 5] Access is denied

======================================================================
ERROR: Test that linear regression also works with sparse data
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\Python27\lib\site-packages\nose-1.3.4-py2.7.egg\nose\case.py", line 1
97, in runTest
    self.test(*self.arg)
  File "E:\Python27\lib\site-packages\sklearn\linear_model\tests\test_base.py",
line 78, in test_linear_regression_sparse
    ols.fit(X, y.ravel())
  File "E:\Python27\lib\site-packages\sklearn\linear_model\base.py", line 359, i
n fit
    out = lsqr(X, y)
  File "E:\Python27\lib\site-packages\scipy\sparse\linalg\isolve\lsqr.py", line
436, in lsqr
    test3 = 1 / acond
ZeroDivisionError: float division by zero

======================================================================
ERROR: sklearn.metrics.tests.test_pairwise.test_pairwise_parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\Python27\lib\site-packages\nose-1.3.4-py2.7.egg\nose\case.py", line 1
97, in runTest
    self.test(*self.arg)
  File "E:\Python27\lib\site-packages\sklearn\metrics\tests\test_pairwise.py", l
ine 128, in test_pairwise_parallel
    S2 = _parallel_pairwise(X, None, euclidean_distances, n_jobs=3)
  File "E:\Python27\lib\site-packages\sklearn\metrics\pairwise.py", line 972, in
 _parallel_pairwise
    for s in gen_even_slices(Y.shape[0], n_jobs))
  File "E:\Python27\lib\site-packages\sklearn\externals\joblib\parallel.py", lin
e 589, in __call__
    raise ImportError('[joblib] Attempting to do parallel computing '
ImportError: [joblib] Attempting to do parallel computing without protecting you
r import on a system that does not support forking. To use parallel-computing in
 a script, you must protect you main loop using "if __name__ == '__main__'". Ple
ase see the joblib documentation on Parallel for more information

======================================================================
ERROR: sklearn.tests.test_pipeline.test_feature_union_parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\Python27\lib\site-packages\nose-1.3.4-py2.7.egg\nose\case.py", line 1
97, in runTest
    self.test(*self.arg)
  File "E:\Python27\lib\site-packages\sklearn\tests\test_pipeline.py", line 349,
 in test_feature_union_parallel
    fs_parallel.fit(X)
  File "E:\Python27\lib\site-packages\sklearn\pipeline.py", line 334, in fit
    for name, trans in self.transformer_list)
  File "E:\Python27\lib\site-packages\sklearn\externals\joblib\parallel.py", lin
e 589, in __call__
    raise ImportError('[joblib] Attempting to do parallel computing '
ImportError: [joblib] Attempting to do parallel computing without protecting you
r import on a system that does not support forking. To use parallel-computing in
 a script, you must protect you main loop using "if __name__ == '__main__'". Ple
ase see the joblib documentation on Parallel for more information

----------------------------------------------------------------------
Ran 3342 tests in 173.391s

FAILED (SKIP=25, errors=4)

[larsmans]

More details needed. What platform? Which version? How did you install it?

[amueller]

Windows, which seems to be the issue.... @ogrisel we do run continuous integration on appveyor, right? Btw, where is the output of that?

[ogrisel]

AppVeyor tests pass:

https://ci.appveyor.com/project/sklearn-ci/scikit-learn/history

This seems to be an issue with multiprocessing (at least in Python 2.7):

http://stackoverflow.com/questions/17076679/windowserror-access-is-denied-on-calling-process-terminate

I never observed it myself when debugging stuff under windows.

[ogrisel]

In Python 3.5, the problem seems to be less likely has the timeout has been set to 1.0 second, here is the end of the multiprocessing/popen_spawn_win32.py file:

    def wait(self, timeout=None):
        if self.returncode is None:
            if timeout is None:
                msecs = _winapi.INFINITE
            else:
                msecs = max(0, int(timeout * 1000 + 0.5))

            res = _winapi.WaitForSingleObject(int(self._handle), msecs)
            if res == _winapi.WAIT_OBJECT_0:
                code = _winapi.GetExitCodeProcess(self._handle)
                if code == TERMINATE:
                    code = -signal.SIGTERM
                self.returncode = code

        return self.returncode

    def poll(self):
        return self.wait(timeout=0)

    def terminate(self):
        if self.returncode is None:
            try:
                _winapi.TerminateProcess(int(self._handle), TERMINATE)
            except OSError:
                if self.wait(timeout=1.0) is None:
                    raise

Under Python 2.7, the corresponding code (multiprocessing/forking.py at line 312) is not as robust as the time is 0.1s:

        def wait(self, timeout=None):
            if self.returncode is None:
                if timeout is None:
                    msecs = _subprocess.INFINITE
                else:
                    msecs = max(0, int(timeout * 1000 + 0.5))

                res = _subprocess.WaitForSingleObject(int(self._handle), msecs)
                if res == _subprocess.WAIT_OBJECT_0:
                    code = _subprocess.GetExitCodeProcess(self._handle)
                    if code == TERMINATE:
                        code = -signal.SIGTERM
                    self.returncode = code

            return self.returncode

        def poll(self):
            return self.wait(timeout=0)

        def terminate(self):
            if self.returncode is None:
                try:
                    _subprocess.TerminateProcess(int(self._handle), TERMINATE)
                except WindowsError:
                    if self.wait(timeout=0.1) is None:
                        raise

I will try to see if I can put a workaround in joblib.

[lesteve]

Closing. This was fixed in joblib/joblib#355 which is in joblib 0.10.0 and joblib 0.10.2 is bundled in the upcoming 0.18 release.