Decouple number of runs from active processes

[m-rph]

Hi Simon. This is what I had in mind, basically it allows joblib or multiprocessing to use to use n_jobs processes to execute n_run "jobs". Note that it is a breaking change.

Solves:

Using n_jobs to get a large number of runs results in spawning too many processes which end up filling the memory and cause thrashing. The patch limits the number of active processes to n_jobs while still allowing hyperactive to execute n_runs times the optimization process.

[SimonBlanke]

Hello @partiallytyped,

now i understand what you wanted to do. This looks very good and i think it would make sense to include this into Hyperactive in the future. I have some questions/things to add:

• This should be commited to the dev branch for now
• It is an API change. A very small one but still a change, so we have to release this in v4.0.0
• What do you mean by "it is a breaking change"?
• The default n_jobs should be "auto" and internally set to the total number of runs. If you pass an int to n_jobs it should work like in your commit. This way it works better "out of the box".

[m-rph]

What do you mean by "it is a breaking change"?
I meant that it is an API change.
It is an API change. A very small one but still a change, so we have to release this in v4.0.0

Your suggestions sound good, I will add the changes and push it.

[SimonBlanke]

I forgot to leave a formal review. The important points are:

• Commit to the dev branch
• The default n_jobs should be "auto" and internally set to the total number of runs. If you pass an int to n_jobs it should work like in your commit. This way it works better "out of the box".

Thanks again for your work!

[SimonBlanke]

Okay, very nice!

I had an idea how to make this compatible with the current API. I would like your opinion.

We could leave the n_jobs/n_runs (old/new) parameter in .add_search(...) and rename the new parameter in Hyperactive(...) to something else. Maybe one of those:

• n_processes
• n_parallel
• n_threads

I guess you have more experience with multiprocessing. Which would you choose?
After that I will resolve the conflicts and merge the pull request.

[m-rph]

I'd go with n_processes since joblib and multiprocess use processes instead of threads.

Sorry for taking too long to respond, exams..

[SimonBlanke]

@partiallytyped,
No problem it wasn't that urgent. Thank you very much for your contribution!