I/O performance hints for Summit (ECP-WarpX#2495)

* Fix conflict with upstream

* Apply suggestions from code review

* Remove space in the end of lines

* Include suggestions from PR review

* Generalize ROMIO Hints in Batch Scripts

* Fix Comment

* Fix Comment

* Remove duplication

* Formatting

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

Docs/source/install/hpc/summit.rst

@@ -234,6 +234,52 @@ parameters provided good performance:
 * **Sixteen `64x64x64` grids per MPI rank** (with default tiling in WarpX, this
   results in ~49 tiles per OpenMP thread)
 
+.. _building-summit-io-performance:
+
+I/O Performance Tuning
+----------------------
+
+.. _building-summit-large-blocks:
+
+GPFS Large Block I/O
+^^^^^^^^^^^^^^^^^^^^
+
+Setting ``IBM_largeblock_io`` to ``true`` disables data shipping, saving overhead when writing/reading large contiguous I/O chunks.
+
+.. code-block:: bash
+
+   export IBM_largeblock_io=true
+
+.. _building-summit-romio-hints:
+
+ROMIO MPI-IO Hints
+^^^^^^^^^^^^^^^^^^
+
+You might notice some parallel HDF5 performance improvements on Summit by setting the appropriate ROMIO hints for MPI-IO operations.
+
+.. code-block:: bash
+
+   export OMPI_MCA_io=romio321
+   export ROMIO_HINTS=./romio-hints
+
+You can generate the ``romio-hints`` by issuing the following command. Remember to change the number of ``cb_nodes`` to match the number of compute nodes you are using (example here: ``64``).
+
+.. code-block:: bash
+
+   cat > romio-hints << EOL
+   romio_cb_write enable
+   romio_ds_write enable
+   cb_buffer_size 16777216
+   cb_nodes 64
+   EOL
+
+The ``romio-hints`` file contains pairs of key-value hints to enable and tune collective
+buffering of MPI-IO operations. As Summit's Alpine file system uses a 16MB block size,
+you should set the collective buffer size to 16GB and tune the number of aggregators
+(``cb_nodes``) to the number of compute nodes you are using, i.e., one aggregator per node.
+
+Further details are available at `Summit's documentation page <https://docs.olcf.ornl.gov/systems/summit_user_guide.html#slow-performance-using-parallel-hdf5-resolved-march-12-2019>`__.
+
 .. _building-summit-issues:
 
 Known System Issues

Tools/BatchScripts/batch_summit.sh

@@ -31,7 +31,22 @@ export OMPI_MCA_coll_ibm_skip_barrier=true
 #export FABRIC_IFACE=mlx5_0   # ADIOS SST: select interface (1 NIC on Summit)
 #export FI_OFI_RXM_USE_SRX=1  # libfabric: use shared receive context from MSG provider
 
-# OpenMP: one thread per MPI rank and GPU
+# ROMIO has a hint for GPFS named IBM_largeblock_io which optimizes I/O with operations on large blocks
+export IBM_largeblock_io=true
+
+# MPI-I/O: ROMIO hints for parallel HDF5 performance
+export OMPI_MCA_io=romio321
+export ROMIO_HINTS=./romio-hints
+#   number of hosts: unique node names minus batch node
+NUM_HOSTS=$(( $(echo $LSB_HOSTS | tr ' ' '\n' | uniq | wc -l) - 1 ))
+cat > romio-hints << EOL
+   romio_cb_write enable
+   romio_ds_write enable
+   cb_buffer_size 16777216
+   cb_nodes ${NUM_HOSTS}
+   EOL
+
+# OpenMP: 1 thread per MPI rank
 export OMP_NUM_THREADS=1
 
 # run WarpX

Tools/BatchScripts/batch_summit_power9.sh

@@ -30,6 +30,21 @@ export OMPI_MCA_coll_ibm_skip_barrier=true
 #export FABRIC_IFACE=mlx5_0   # ADIOS SST: select interface (1 NIC on Summit)
 #export FI_OFI_RXM_USE_SRX=1  # libfabric: use shared receive context from MSG provider
 
+# ROMIO has a hint for GPFS named IBM_largeblock_io which optimizes I/O with operations on large blocks
+export IBM_largeblock_io=true
+
+# MPI-I/O: ROMIO hints for parallel HDF5 performance
+export OMPI_MCA_io=romio321
+export ROMIO_HINTS=./romio-hints
+#   number of hosts: unique node names minus batch node
+NUM_HOSTS=$(( $(echo $LSB_HOSTS | tr ' ' '\n' | uniq | wc -l) - 1 ))
+cat > romio-hints << EOL
+   romio_cb_write enable
+   romio_ds_write enable
+   cb_buffer_size 16777216
+   cb_nodes ${NUM_HOSTS}
+   EOL
+
 # OpenMP: 21 threads per MPI rank
 export OMP_NUM_THREADS=21