Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide an option to disable multithreading while multiprocessing #2519

Open
Garyfallidis opened this issue Feb 17, 2022 · 0 comments · May be fixed by #2547
Open

Provide an option to disable multithreading while multiprocessing #2519

Garyfallidis opened this issue Feb 17, 2022 · 0 comments · May be fixed by #2547
Assignees

Comments

@Garyfallidis
Copy link
Contributor

When we are using multiprocessing for example in peaks_from_model we can setup the number of processes (for example at 4) but because some of the code will use multi-threading through numpy (Openblas, MKL) we end up using more cores than necessary (for example 12 instead of 4). This can end up delaying execution or just use too many resources freezing the computer.

A solution for this is to disable multithreading when multiprocessing. This can be done using the following snippet:

import os
os.environ['OPENBLAS_NUM_THREADS'] = '1'
os.environ['MKL_NUM_THREADS'] = '1'

I suggest that we set the values above before multiprocessing starts and set them back to default after the work is done. In that way we will not affect execution to functions that do use multithreading.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants