Restrict use of multiple threads for parallel processing

[andportnoy]

The following works if we are the first to import NumPy:

import os
os.environ["MKL_NUM_THREADS"] = "1" 
os.environ["OMP_NUM_THREADS"] = "1" 
import numpy as np

If NumPy had already been imported, multiple threads are still being used.
More details in numpy/numpy#11826

[andportnoy]

Pandas discussion: pandas-dev/pandas#23139

[andportnoy]

Solution:

1. Check platform.
2. If Linux, run ldd on multiarray.so. If Mac, run otool -L on multiarray.so. Windows not supported.
3. Try matching OpenBLAS and MKL in the output. If either is found, use ctypes inside a context manager to set the number of threads.
4. Run computation.

References:

• Checking platform
• Parsing ldd output
• Checking linking
• Setting number of threads in MKL
• Setting number of threads in OpenBLAS

[andportnoy]

My current understanding is that on Macs pip installing NumPy results in OpenBLAS being installed in the .dylibs subdirectory of the NumPy directory in site-packages. So for Macs the strategy is to load the library directly (using the path provided by otool). Maybe that should be the approach in general (as opposed to using automatic resolution with ctypes.CDLL).

For now, we focus on Linux.

[andportnoy]

Interesting: on a machine with 48 cores MKL defaults to 24 threads, OpenBLAS to all 48.

In addition, MKL sets a hard max at 24 threads (half available?).

[andportnoy]

closed as of 6b10830