User ability to decide GPU device number

[siddanib]

Integration of FTorch with a distributed CPU based solver can lead to a scenario where there are N (--ntasks-per-node) MPI and M (torch::cuda::device_count()) GPUs per node (**M** <= **N**). The current implementation of FTorch appears to leverage only GPU:0 for all N MPI ranks. Providing user ability to decide which GPU to leverage can ensure that all available GPUs are used.

An initial discussion regarding this potential feature: #84.

Furthermore, there might still be multiple MPI ranks per GPU even after uniformly distributing the MPI ranks among available GPUs. The GPU probably calls these ML model copies serially. CUDA MPS could be utilized to concurrently run the ML model copies. An alternative might be to perform (gather to a single task, deploy the ML model from that task, and finally scatter to respective tasks) inside the fortran code.

[jatkinson1000]

@siddanib PR #96 has been opened to address this which builds on your suggestions (thank you!) but in a way consistent with our overall approach.

Please do feel free to take a look and see what you think.

[jatkinson1000]

@siddanib this is now implemented in the main code.
You can see information here and a worked example here.

Thank you for raising this as an improvement to the code, and all the help you provided pointing us to the relevant information.

[siddanib]

Thank you very much for providing this capability, @jatkinson1000 & @jwallwork23! I will look into this new feature and get back to you.