Call hipInit before MPI_Init.

[joaander]

Description

Call hipInit(0) before MPI_Init.

Motivation and context

OLCF suggests this as a workaround for segmentation faults in MPI_Init: https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#olcfdev-1655-occasional-seg-fault-during-mpi-init.

How has this been tested?

I attempted to reproduce the failure. Prior to this change, i did observe:

Fri Sep 22 11:49:53 2023: [PE_388]:_pmi_mmap_tmp: Warning bootstrap barrier failed: num_syncd=3, pes_this_node=8, timeout=180 secs
Fri Sep 22 11:49:53 2023: [PE_385]:_pmi_mmap_tmp: Warning bootstrap barrier failed: num_syncd=3, pes_this_node=8, timeout=180 secs
Fri Sep 22 11:49:53 2023: [PE_385]:_pmi_mmap_init:Failed to setup PMI mmap.Fri Sep 22 11:49:53 2023: [PE_385]:globals_init:_pmi_mmap_init returned -1
MPICH ERROR [Rank 0] [job id unknown] [Fri Sep 22 11:49:53 2023] [frontier04084] - Abort(1091855) (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(170): 
MPID_Init(441).......: 
MPIR_pmi_init(110)...: PMI_Init returned 1

very occasionally - approximately 3 times out of hundreds of launches ranging from 10 nodes to 500 nodes. This error occurred during both CPU-only and GPU jobs.

With ~20 submissions after this change, I did not observe the error. With such a low rate of occurrence, we will only be certain of the fix after long term testing with production runs. There is no harm in initializing the hip runtime early: https://rocm.docs.amd.com/projects/HIP/en/latest/.doxygen/docBin/html/group___driver.html#ga01baa652dda5815c594d047060496caa - it is normally initialized automatically on the first HIP call.

Change log

Fixed:

* Attempt to workaround `PMI_Init returned 1` error on OLCF Frontier.

Checklist:

• I have reviewed the Contributor Guidelines.
• I agree with the terms of the HOOMD-blue Contributor Agreement.
• My name is on the list of contributors (sphinx-doc/credits.rst) in the pull request source branch.

[tommy-waltmann]

The docs mention having to compile the code differently once this statement is added. Since you've already tested this change on frontier, I'm assuming that CMake automatically accounted for that. Is that right?

[joaander]

Those instructions link to docs that explain how to compile with HIP. We already compile with HIP in HOOMD. I think that the difference implied is for CPU only codes that are not already building with HIP, or possibly codes that use OpenMP or other libraries for interfacing with the GPU.

It is true that an ENABLE_GPU=off build of HOOMD on Frontier would potentially still have this issue. However, there should be very few use-cases to do that and I do not wish to rewrite major portions of HOOMD's cmake to allow linking to HIP with ENABLE_GPU=off.

[tommy-waltmann]

Ok, looks good then.