From bf05f03e09780cde3144ccb1589150bd76188237 Mon Sep 17 00:00:00 2001 From: Jiakun Yan Date: Sun, 2 Jul 2023 15:21:26 -0500 Subject: [PATCH] Update the documentation for the LCI parcelport. --- .../manual/using_the_lci_parcelport.rst | 26 +++++++++++++++---- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/docs/sphinx/manual/using_the_lci_parcelport.rst b/docs/sphinx/manual/using_the_lci_parcelport.rst index 986d24f945db..95e8193282cd 100644 --- a/docs/sphinx/manual/using_the_lci_parcelport.rst +++ b/docs/sphinx/manual/using_the_lci_parcelport.rst @@ -97,13 +97,29 @@ If you are using ``mpirun`` or ``srun``, you can just pass ``--hpx:ini=hpx.parcel.lci.priority=1000``, ``--hpx:ini=hpx.parcel.lci.enable=1``, and ``--hpx:ini=hpx.parcel.bootstrap=lci`` to the |hpx| applications. +If you are running on a Cray machine, you need to pass `--mpi=pmix` or `--mpi=pmi2` to srun +to enable the PMIx or PMI2 support of SLURM since LCI does not support the default Cray PMI. +For example, + +.. code-block:: shell-session + + $ srun --mpi=pmix [hpx application] + .. _tune_lci_pp: Performance tuning of the LCI parcelport ======================================== -We encourage users to increase the zero-copy serialization threshold of LCI to ``8192`` (bytes) by -either the cmake variable ``-DHPX_WITH_ZERO_COPY_SERIALIZATION_THRESHOLD=8192`` -(this will affect all the parcelports) or the command line option -``--hpx:ini=hpx.parcel.lci.zero_copy_serialization_threshold=8192`` -(this will only affect the LCI parcelport). +We encourage users to set the following environmental variables +when using the LCI parcelport to get better performance. + +.. code-block:: shell-session + + $ export LCI_SERVER_MAX_SENDS=1024 + $ export LCI_SERVER_MAX_RECVS=4096 + $ export LCI_SERVER_NUM_PKTS=65536 + $ export LCI_SERVER_MAX_CQES=65536 + $ export LCI_PACKET_SIZE=12288 + +This setting needs roughly 800MB memory per process. The memory consumption mainly +comes from the packets, which can be calculated using `LCI_SERVER_NUM_PKTS x LCI_PACKET_SIZE`. \ No newline at end of file