run code on multiple GPUs

[shu-hai]

Hi, Julian,
I just start to run your step3_predict_nodules.py using your trained model.
I found it only ran on 1 GPU even I assigned 2 GPUs to it by
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"
I also muted config.gpu_options.per_process_gpu_memory_fraction = 0.5
because I am allowed to use the 2 GPUs totally, but the speed was still slow.

Could you let me know how to run the code on multiple GPUs? Thanks.

[juliandewit]

Hello,
I think tensorflow sees no way to distribute the network over multiple GPU's.
Although in theory it should be smart enough to split the batch in 2 parts en run eacht part on a separate GPU.

You could do this manually however.
I cannot type it out for you but every patient needs roughly 30x30x30 (~900) predictions .
If you predict half of them over GPU1 with a network and the other half over GPU2 with another instance of the network you will achieve 2x speedup.

[shu-hai]

Hi, Julian,
On lines 348-349 of step4_train_submissions.py, it is the following:

  if level == 1:
        dst_dir += "level2/"

Why not level1?

[juliandewit]

Indeed I also had to look twice after this time.

The level 1 models are combined into level2 folder.
The models in level2 are combined into the submission folder.

[shu-hai]

Also it gives an error on line 23 of step4_train_submissions.py:
mass_df = pandas.read_csv(settings.BASE_DIR + "masses_predictions.csv").
It cannot find the masses_predictions.csv file.
I searched this file name in the codes of first three steps，but cannot find it.
Where do you generate this file？

[juliandewit]

step2_train_mass_segmenter.py
Also has a predict phase.
This one will generate this file.

You can also leave it out. It will not change the score very much.