Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Joblib's BLAS disabling mechanism to prevent CPU over-subscription does not work #834
python version: 3.6.5
According to Section 2.4.7 "Avoiding over-subscription of CPU ressources" joblib turns off BLAS automatically. But it does not work for me. In the following code,
#import os #os.environ['OPENBLAS_NUM_THREADS'] = '1' #os.environ['MKL_NUM_THREADS'] = '1' #os.environ['NUMEXPR_NUM_THREADS'] = '1' import numpy as np import joblib X = np.random.rand(100,60000) # 60: no threads, 60000: lot's of threads ;) def func(): for i in range(10000): X.dot(np.transpose(X)) joblib.Parallel(n_jobs=2, verbose=10)(joblib.delayed(func)() for i in range(10000))
I can replicate the issue on 2/3 machines. On one machine it happened yesterday, today it does not. A little mysterious ;)
NAME="Ubuntu" VERSION="18.04.1 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.1 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic
The over-subscription protection of loky is only for the children worker processes (that are created when n_jobs >= 2).
As I mentioned. Even if I set
Interestingly enough though, I also tried again on all of my three machines. And again, this happened only on two out of three machines.
Let me know if you need any specific info or statistics.