Does it unstable to use Multiple GPU per MPI Process?

[johnkim126]

Hello, Great Team.

I knew that Horovod can execute tensorflow code with Multiple GPU.

In the usage, It looks like horovod recommended using GPU per process.
--> Pin a server GPU to be used by this process using config.gpu_options.visible_device_list. With the typical setup of one GPU per process, this can be set to local rank

So, Does it unstable to use Multiple GPU per MPI Process?

[alsrgv]

@johnkim126, you can use multiple GPU per process - just make sure to change config.gpu_options.visible_device_list appropriately and manage data exchange between GPUs. E.g. see discussion / examples here - #96.

[johnkim126]

@alsrgv Thanks.
I have another question.
Well, Keras code with multiple GPU code can use in Horovod?
-> using from keras.utils.training_utils import multi_gpu_model in Keras code
-> and remove config.gpu_options.visible_device_list and manage data exchange between GPUs.

[ppwwyyxx]

from keras.utils.training_utils import multi_gpu_model does not scale, and that's one of the reason horovod is useful.

[johnkim126]

@ppwwyyxx Okay, Thanks :)