-
Notifications
You must be signed in to change notification settings - Fork 22
Description
The Data class takes all the cores detected by "multiprocessing.cpu_count()" as a default number of jobs. When this number is larger than 128, the Class crushes due to an OPENBLAS memory issue:
OpenBLAS : Program is Terminated. Because you tried to allocate too many memory regions.
This library was built to support a maximum of 128 threads - either rebuild OpenBLAS
with a larger NUM_THREADS value or set the environment variable OPENBLAS_NUM_THREADS to
a sufficiently small number. This error typically occurs when the software that relies on
OpenBLAS calls BLAS functions from many threads in parallel or when your computer has more
cpu cores than what OpenBLAS was configured to handle.
I propose setting the default value to 1 or None, or the value set in the underlying libraries we use.