Change multiprocessing methodology from Process to Pool.starmap.

[Iain-S]

Hello

The existing code for handling processes in:

• mantis_ml/modules/supervised_learn/model_selection/benchmark_all_classifiers.py
• mantis_ml/modules/supervised_learn/feature_selection/run_boruta.py
• mantis_ml/modules/supervised_learn/model_selection/dnn_grid_search_cv.py
• mantis_ml/modules/supervised_learn/pu_learn/pu_learning.py

Looks something like:

procs = []
for i in range(len(train_data)):
    p = Process(train_data[i], test_data[i], results_list)
    procs.append(p)

    if len(procs) > max_procs:
        for k in procs:
            k.join()
        procs = []

for k in procs:
    k.join()

This means that any time len(procs) > max_procs, you wait on all currently running processes to finish. This is inefficient because you will always be waiting on the slowest process to finish before starting any others.

Instead, you can use Pool.starmap from multiprocessing.

It works like this:

with Pool(max_procs) as pool:
    pool.starmap(zip(train_data, test_data, repeat(results_list)))

It means that as soon as one process finishes, another will start. In my tests, the time for the first 50 training sets to run went from 60 seconds to 12 seconds.

I have proposed changes to pu_learning.py because that was the only module I knew how to test. The same logic could easily be applied to the other three.

[Iain-S]

Hi @dvitsios

Just wanted to draw your attention to this PR, in case you hadn't seen it or wanted to propose any changes.