Importing sklearn messes with CPU affinity, causing n_jobs=-1 to become useless

[dniku]

Description

Please see a PoC notebook.

tl:dr: import sklearn causes CPU affinity to be set to {0}. Resetting it causes RandomizedSearchCV to freeze.

Expected Results

n_jobs=-1 uses all cores and does not freeze.

Actual Results

Only one core is loaded at 100% and nothing happens for 30+ minutes. Not resetting CPU affinity or not specifying n_jobs=-1 causes the freeze to go away.

Versions

Linux-4.14.36-1-MANJARO-x86_64-with-arch-Manjaro-Linux
Python 3.6.5 (default, Apr 12 2018, 22:45:43) 
[GCC 7.3.1 20180312]
NumPy 1.14.2
SciPy 1.0.1
Scikit-Learn 0.19.1

[rth]

I can't reproduce the affinity results with a somewhat similar configuration,

Linux-4.11.6-gentoo-x86_64-Intel-R-_Core-TM-_i5-6200U_CPU_@_2.30GHz-with-gentoo-2.4.1
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56) 
[GCC 7.2.0]
NumPy 1.14.2
SciPy 1.0.1
Scikit-Learn 0.19.1

(for me the affinity remains unchanged). If you re-start the interpreter and run just the scikit-learn related imports together with fix_affinity(), can you reproduce it?

As to the freezing with multiprocessing, it's a known issue see http://scikit-learn.org/stable/faq.html#why-do-i-sometime-get-a-crash-freeze-with-n-jobs-1-under-osx-or-linux and #5115 (comment)

[jnothman]

Maybe it's worth creating a PR which adds a test assertion to see if any of the ci environments can replicate. Does this happen when importing sklearn? Or particular submodules? Also worth checking if importing joblib alone does this, in which case it should be raised there.

[rth]

Wrote a unit test for this in master...rth:cpu-affinity : it passes on both Appveyor and Travis CI (cf build logs of the last commit). It's a bit brittle (requires Python 3 and might fail /be skiped on some architectures), so I'm not sure I should make a PR.

Generally, after a quick search, I couldn't find anything that suggests that scikit-learn has the capacity to affect the CPU affinity (unless I missed something). joblib also doesn't mention it, but I wonder about side effects of multiprocessing.. @lesteve @aabadie

OpenBlas is another suspect..

[dniku]

I've experimented a bit more with this.

I'm running Manjaro (a derivative of Arch) and initially I submitted this report having python-numpy-openblas installed. Then I did the following.

• Replaced the OpenBLAS version with the vanilla Numpy from the repository (python-numpy) and couldn't reproduce the issue.
• Replaced the vanilla Numpy with the OpenBLAS version again and got strange results: the first time I ran the notebook after reinstallation I did not observe affinity being changed, but the freeze occurred regardless. On subsequent runs, affinity was changed.
• Replaced the OpenBLAS version with python-numpy-mkl and observed both affinity being changed and the freeze on the first run.

Also, I'm 99% sure that affinity is changed after import sklearn, not after importing specific modules. I've just tested it with python-numpy-mkl and I'm almost confident I tested it with python-numpy-openblas too.

It seems surprising though that importing Numpy does not affect affinity, while importing sklearn does.

[dniku]

There is clearly some undefined behavior going on here. With python-numpy-mkl, I just got the following log in terminal ipython:

$ ipython
Python 3.6.5 (default, Apr 12 2018, 22:45:43) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.3.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import os

In [2]: os.sched_getaffinity(0)
Out[2]: {0, 1, 2, 3, 4, 5, 6, 7}

In [3]: import sklearn

In [4]: os.sched_getaffinity(0)
Out[4]: {0}

In [5]:Do you really want to exit ([y]/n)? 
$ ipython
Python 3.6.5 (default, Apr 12 2018, 22:45:43) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.3.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import os

In [2]: os.sched_getaffinity(0)
Out[2]: {0, 1, 2, 3, 4, 5, 6, 7}

In [3]: import sklearn.externals.joblib

In [4]: os.sched_getaffinity(0)
Out[4]: {0, 1, 2, 3, 4, 5, 6, 7}

Then I re-ran the Jupyter notebook multiple times ("restart kernel and run all cells") and one time the affinity wasn't changed, while all the other times it was changed at the sklearn.externals.joblib import. The freeze occurred every time regardless.

[lesteve]

Just putting a snippet for easier reproducibility:

import os
import multiprocessing
import sys

def fix_affinity():
    # 0 means current process
    affinity = os.sched_getaffinity(0)
    if len(affinity) != multiprocessing.cpu_count():
        print("Something has messed with CPU affinity. Current affinity is {}. Fixing".format(affinity),
              file=sys.stderr)
        os.sched_setaffinity(0, set(range(multiprocessing.cpu_count())))

        assert len(os.sched_getaffinity(0)) == multiprocessing.cpu_count(), os.sched_getaffinity(0)
    else:
        print("Affinity is OK: {}".format(affinity))

fix_affinity()

import sklearn

fix_affinity()

As @rth I can not reproduce the problem on my machine. The output I get is:

Affinity is OK: {0, 1, 2, 3}
Affinity is OK: {0, 1, 2, 3}

[rth]

Also note that len(affinity) != multiprocessing.cpu_count() does not necessarily mean that something is wrong. e.g. on Travis CI that runs in Docker containers, the CPU count is 32, but we can only use 2 of those i.e. len(affinity) = 2 see https://github.com/tomMoral/loky/issues/111#issuecomment-372975178 for more details.

@Pastafarianist you are not using any container or VM technologies for your tests, right?

[dniku]

@lesteve you forgot import sys (and I did too).
@rth I'm not using any containers. My installation is a fresh Manjaro + python-numpy-mkl from AUR + python-scikit-learn from Arch community repository on a laptop. I'm running jupyter notebook directly from the fish shell, without Docker or virtualenv or anything else.

I've just taken the script by @lesteve, added import sys, and run it twice (doing nothing in between). I got the following output:

$ python affinity.py 
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}
$ python affinity.py 
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}
Something has messed with CPU affinity. Current affinity is {0}. Fixing

I'd like to stress that I cannot reproduce this with vanilla python-numpy from the Arch repository, only with python-numpy-openblas and python-numpy-mkl from AUR, and, furthermore, import numpy does not break affinity, but import sklearn does.

[lesteve]

@lesteve you forgot import sys (and I did too).

Good point fixed! Just a tiny piece of advice for the future: the easier you make your problem to reproduce, the more likely you are to get good feed-back.

Do you have the same problem when you install numpy and scikit-learn with pip, rather than with your package manager?

[dniku]

@lesteve surprisingly, yes, the problem manifests with numpy and scikit-learn from PyPI. I've installed both via pip (pip install --user numpy scikit-learn --force) and modified your version of the script to print the paths to the imported modules.

import os
import sys
import multiprocessing

def fix_affinity():
    # 0 means current process
    affinity = os.sched_getaffinity(0)
    if len(affinity) != multiprocessing.cpu_count():
        print("Something has messed with CPU affinity. Current affinity is {}. Fixing".format(affinity),
              file=sys.stderr)
        os.sched_setaffinity(0, set(range(multiprocessing.cpu_count())))

        assert len(os.sched_getaffinity(0)) == multiprocessing.cpu_count(), os.sched_getaffinity(0)
    else:
        print("Affinity is OK: {}".format(affinity))

fix_affinity()

import numpy as np
print(np.__file__)
fix_affinity()

import sklearn
print(sklearn.__file__)
fix_affinity()

The output I'm getting is

$ python affinity.py 
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}
/home/ser/.local/lib/python3.6/site-packages/numpy/__init__.py
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}
/home/ser/.local/lib/python3.6/site-packages/sklearn/__init__.py
Something has messed with CPU affinity. Current affinity is {0}. Fixing

This was tested when I still had python-numpy-mkl installed system-wide. I replaced it with vanilla python-numpy and the problem did not go away: the output was exactly the same.

As expected, after uninstalling numpy and scikit-learn from ~/.local (pip uninstall numpy scikit-learn) and using vanilla python-numpy the problem did not manifest:

$ python affinity.py 
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}
/usr/lib/python3.6/site-packages/numpy/__init__.py
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}
/usr/lib/python3.6/site-packages/sklearn/__init__.py
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}

[jnothman]

let's try localise the problem: does it occur when importing numpy alone? when importing joblib (i.e. not from sklearn.externals)?

[dniku]

@jnothman

1. I tried importing only numpy and exiting right after the affinity check. Affinity is not changed in this case.

$ python affinity.py 
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}
/home/ser/.local/lib/python3.6/site-packages/numpy/__init__.py
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}

2. Importing joblib does not cause the problem. I added

import joblib
print(joblib.__file__)
fix_affinity()

between importing numpy and scikit-learn and I'm getting

$ python affinity.py 
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}
/home/ser/.local/lib/python3.6/site-packages/numpy/__init__.py
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}
/home/ser/.local/lib/python3.6/site-packages/joblib/__init__.py
Affinity is OK: {0, 1, 2, 3, 4, 5, 6, 7}
/home/ser/.local/lib/python3.6/site-packages/sklearn/__init__.py
Something has messed with CPU affinity. Current affinity is {0}. Fixing

Note that numpy, joblib, and scikit-learn are installed by pip in all of these tests.

[beimingliu]

@Pastafarianist any fixes?

[dniku]

@beimingliu nope. I have not been investigating the issue since the last time.

[CentralLT]

@dniku is this still an issue or can u use - the multicore option now?