From 2f133c8c81c869e5323a6117637c8958d999d4d9 Mon Sep 17 00:00:00 2001 From: shudson Date: Tue, 6 Dec 2022 21:45:32 -0600 Subject: [PATCH 1/9] Add Polaris guide --- docs/platforms/platforms_index.rst | 1 + docs/platforms/polaris.rst | 103 +++++++++++++++++++++++++++++ 2 files changed, 104 insertions(+) create mode 100644 docs/platforms/polaris.rst diff --git a/docs/platforms/platforms_index.rst b/docs/platforms/platforms_index.rst index 8fbc4bcaa..ab7a21db5 100644 --- a/docs/platforms/platforms_index.rst +++ b/docs/platforms/platforms_index.rst @@ -210,6 +210,7 @@ libEnsemble on specific HPC systems. bebop cori perlmutter + polaris spock/crusher summit theta diff --git a/docs/platforms/polaris.rst b/docs/platforms/polaris.rst new file mode 100644 index 000000000..65247e8d2 --- /dev/null +++ b/docs/platforms/polaris.rst @@ -0,0 +1,103 @@ +======= +Polaris +======= + +.. To be updated if we get a libEnsemble module + +Polaris_ is a 560 node HPE system located in the ALCF_ at Argonne +National Laboratory. The compute nodes are equipped with a single AMD EPYC Milan +processor and four A100 NVIDIA GPUs. It uses the PBS scheduler to submit +jobs from login nodes to run on the compute nodes. + + +Configuring Python and Installation +----------------------------------- + +Python can be used on Polaris with the `conda` module. To install further packages you may +either create a virtual environment on top of this (if just using pip install), +or clone the base environment (if you need conda install). More details at `Python for Polaris`_. + +.. container:: toggle + + .. container:: header + + Example of Conda + virtual environment + + E.g.,~ to use conda_ with a virtual environment, load the ``conda`` module and activate + the base environment:: + + module load conda + conda activate base + + and then create a virtual environment that allows installation of further packages:: + + python -m venv /path/to-venv --system-site-packages + . /path/to-venv/bin/activate + + Where ``/path/to-venv`` can be anywhere you have write access. You now have access to machine + optimized packages such as mpi4py_. For future uses just load the conda module and run the + activate line. + +You can now pip install libEnsemble:: + + pip install libensemble + +See :doc:`here<../advanced_installation>` for more information on advanced options +for installing libEnsemble, including using Spack. + + +Ensuring use of mpiexec +----------------------- + +If using the :doc:`MPIExecutor<../executor/mpi_executor>` it is recommended to +ensure you are using ``mpiexec`` instead of ``aprun``. When setting up the executor use:: + + from libensemble.executors.mpi_executor import MPIExecutor + exctr = MPIExecutor(custom_info={'mpi_runner':'mpich', 'runner_name':'mpiexec'}) + +.. This is where platform/system files would be useful... +.. And in script could just use e.g. libE_specs['platform'] == "polaris" + +Job Submission +-------------- + +Polaris uses the PBS scheduler to submit jobs from login nodes to run on +the compute nodes. libEnsemble runs on the compute nodes using either +``multi-processing`` or ``mpi4py`` + +A simple example batch script for a libEnsemble use case that runs 5 workers (one +generator and four simulators) on one node: + +.. code-block:: bash + :linenos: + + #!/bin/bash + #PBS -A + #PBS -lwalltime=00:15:00 + #PBS -lselect=1 + #PBS -q debug + #PBS -lsystem=polaris + #PBS -lfilesystems=home:grand + + export MPICH_GPU_SUPPORT_ENABLED=1 + + cd $PBS_O_WORKDIR + + python run_libe_forces.py --comms local --nworkers 5 + +The script can be run with:: + + qsub submit_libe.sh + +Or you can run an interactive session with:: + + qsub -A -l select=1 -l walltime=15:00 -lfilesystems=home:grand -qdebug -I + +Note that you may need to reload your ``conda`` module and reactivate ``venv`` environment +again after starting the interactive session. + +.. _Polaris: https://www.alcf.anl.gov/polaris +.. _ALCF: https://www.alcf.anl.gov/ +.. _Python for Polaris: https://www.alcf.anl.gov/support/user-guides/polaris/data-science-workflows/python/index.html +.. _conda: https://conda.io/en/latest/ +.. _mpi4py: https://mpi4py.readthedocs.io/en/stable/ From 7dea5713bfd063a25ba18a0cf20906e0937e518e Mon Sep 17 00:00:00 2001 From: shudson Date: Wed, 8 Feb 2023 15:52:30 -0600 Subject: [PATCH 2/9] Add forces_gpu Polaris submission script --- .../forces/submission_scripts/submit_pbs_simple.sh | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 libensemble/tests/scaling_tests/forces/submission_scripts/submit_pbs_simple.sh diff --git a/libensemble/tests/scaling_tests/forces/submission_scripts/submit_pbs_simple.sh b/libensemble/tests/scaling_tests/forces/submission_scripts/submit_pbs_simple.sh new file mode 100644 index 000000000..c94c8aa12 --- /dev/null +++ b/libensemble/tests/scaling_tests/forces/submission_scripts/submit_pbs_simple.sh @@ -0,0 +1,10 @@ +#!/bin/bash -l +#PBS -l select=1:system=polaris +#PBS -l walltime=00:15:00 +#PBS -l filesystems=home:grand +#PBS -q debug +#PBS -A + +export MPICH_GPU_SUPPORT_ENABLED=1 +cd $PBS_O_WORKDIR +python run_libe_forces.py --comms local --nworkers 4 From 87b5796fdf443b08b676ccb15c88ef85c5bd720f Mon Sep 17 00:00:00 2001 From: shudson Date: Wed, 8 Feb 2023 15:56:42 -0600 Subject: [PATCH 3/9] Add simple pbs submit script to examples --- examples/libE_submission_scripts/submit_pbs_simple.sh | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 examples/libE_submission_scripts/submit_pbs_simple.sh diff --git a/examples/libE_submission_scripts/submit_pbs_simple.sh b/examples/libE_submission_scripts/submit_pbs_simple.sh new file mode 100644 index 000000000..161d38d40 --- /dev/null +++ b/examples/libE_submission_scripts/submit_pbs_simple.sh @@ -0,0 +1,10 @@ +#!/bin/bash -l +#PBS -l select=2 +#PBS -l walltime=00:15:00 +#PBS -q +#PBS -A + +# We selected 2 nodes - now running with 8 workers. +export MPICH_GPU_SUPPORT_ENABLED=1 +cd $PBS_O_WORKDIR +python run_libe_forces.py --comms local --nworkers 8 From 097765d41f0c14f5489b1b9ca2b6a31d5a867e7a Mon Sep 17 00:00:00 2001 From: shudson Date: Wed, 8 Feb 2023 16:24:17 -0600 Subject: [PATCH 4/9] Update Polaris guide --- docs/platforms/polaris.rst | 50 +++++++++++++++++++++----------------- 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/docs/platforms/polaris.rst b/docs/platforms/polaris.rst index 65247e8d2..434b6e0fd 100644 --- a/docs/platforms/polaris.rst +++ b/docs/platforms/polaris.rst @@ -2,8 +2,6 @@ Polaris ======= -.. To be updated if we get a libEnsemble module - Polaris_ is a 560 node HPE system located in the ALCF_ at Argonne National Laboratory. The compute nodes are equipped with a single AMD EPYC Milan processor and four A100 NVIDIA GPUs. It uses the PBS scheduler to submit @@ -13,9 +11,17 @@ jobs from login nodes to run on the compute nodes. Configuring Python and Installation ----------------------------------- -Python can be used on Polaris with the `conda` module. To install further packages you may -either create a virtual environment on top of this (if just using pip install), -or clone the base environment (if you need conda install). More details at `Python for Polaris`_. +Python and libEnsemble are available on Polaris with the `conda` module. Load the +``conda`` module and activate the base environment:: + + module load conda + conda activate base + +This also gives you access to machine optimized packages such as mpi4py_. + +To install further packages, including updating libEnsemble, you may either create +a virtual environment on top of this (if just using pip install), or clone the base +environment (if you need conda install). More details at `Python for Polaris`_. .. container:: toggle @@ -23,24 +29,17 @@ or clone the base environment (if you need conda install). More details at `Pyth Example of Conda + virtual environment - E.g.,~ to use conda_ with a virtual environment, load the ``conda`` module and activate - the base environment:: - - module load conda - conda activate base - - and then create a virtual environment that allows installation of further packages:: + E.g.,~ to create a virtual environment that allows installation of further packages:: python -m venv /path/to-venv --system-site-packages . /path/to-venv/bin/activate - Where ``/path/to-venv`` can be anywhere you have write access. You now have access to machine - optimized packages such as mpi4py_. For future uses just load the conda module and run the - activate line. + Where ``/path/to-venv`` can be anywhere you have write access. You now have access to + For future uses just load the conda module and run the activate line. -You can now pip install libEnsemble:: + You can now pip install libEnsemble:: - pip install libensemble + pip install libensemble See :doc:`here<../advanced_installation>` for more information on advanced options for installing libEnsemble, including using Spack. @@ -55,8 +54,6 @@ ensure you are using ``mpiexec`` instead of ``aprun``. When setting up the execu from libensemble.executors.mpi_executor import MPIExecutor exctr = MPIExecutor(custom_info={'mpi_runner':'mpich', 'runner_name':'mpiexec'}) -.. This is where platform/system files would be useful... -.. And in script could just use e.g. libE_specs['platform'] == "polaris" Job Submission -------------- @@ -65,8 +62,8 @@ Polaris uses the PBS scheduler to submit jobs from login nodes to run on the compute nodes. libEnsemble runs on the compute nodes using either ``multi-processing`` or ``mpi4py`` -A simple example batch script for a libEnsemble use case that runs 5 workers (one -generator and four simulators) on one node: +A simple example batch script for a libEnsemble use case that runs 5 workers +(e.g.,~ one persistent generator and four for simulations) on one node: .. code-block:: bash :linenos: @@ -93,11 +90,20 @@ Or you can run an interactive session with:: qsub -A -l select=1 -l walltime=15:00 -lfilesystems=home:grand -qdebug -I -Note that you may need to reload your ``conda`` module and reactivate ``venv`` environment +You may need to reload your ``conda`` module and reactivate ``venv`` environment again after starting the interactive session. +Demonstration +------------- + +For an example that runs a small ensemble using a C application (offloading work to the +GPU), see :doc:`forces_gpu<../tutorials/forces_gpu_tutorial>` tutorial. A video demonstration_ +of this example is available. + + .. _Polaris: https://www.alcf.anl.gov/polaris .. _ALCF: https://www.alcf.anl.gov/ .. _Python for Polaris: https://www.alcf.anl.gov/support/user-guides/polaris/data-science-workflows/python/index.html .. _conda: https://conda.io/en/latest/ .. _mpi4py: https://mpi4py.readthedocs.io/en/stable/ +.. _demonstration: https://youtu.be/Ff0dYYLQzoU From be8cf1d5699080a53918de0e9831e95adc9f39e0 Mon Sep 17 00:00:00 2001 From: shudson Date: Thu, 9 Feb 2023 09:58:39 -0600 Subject: [PATCH 5/9] Update forces_gpu tutorial --- docs/tutorials/forces_gpu_tutorial.rst | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/docs/tutorials/forces_gpu_tutorial.rst b/docs/tutorials/forces_gpu_tutorial.rst index 574ade39d..209e2742e 100644 --- a/docs/tutorials/forces_gpu_tutorial.rst +++ b/docs/tutorials/forces_gpu_tutorial.rst @@ -12,27 +12,24 @@ devices. This example is based on the :doc:`simple forces tutorial <../tutorials/executor_forces_tutorial>` with -a slightly modified simulation function. +a slightly modified simulation function (to assign GPUs) and an increased number +of particles (allows live GPU usage to be viewed). -To compile the forces application to use the GPU, ensure forces.c_ has the -``#pragma omp target`` line uncommented and comment out the equivalent -``#pragma omp parallel`` line. Then compile **forces.x** using one of the -GPU build lines in build_forces.sh_ or similar for your platform. +Compile **forces.x** using one of the GPU build lines in build_forces.sh_ +or similar for your platform. The libEnsemble scripts in this example are available under forces_gpu_ in the libEnsemble repository. -Note that at the time of writing, the calling script **run_libe_forces.py** is functionally -the same as that in *forces_simple*, but contains some commented out lines that can -be used for a variable resources example. The *forces_simf.py* file has slight modifications -to assign GPUs. - -Videos demonstrate running this example on Perlmutter_ and Spock_. +Videos demonstrate running this example on Perlmutter_, Spock_, and Polaris_. Note, in +the first two videos are from an earlier release - you no longer need to change +particle count or modify the `forces.c` file). Simulation function ------------------- -The ``sim_f`` (``forces_simf.py``) becomes as follows. The new lines are highlighted: +The ``sim_f`` (``forces_simf.py``) is as follows. The lines that are different +to the forces simple example are highlighted: .. code-block:: python :linenos: @@ -242,4 +239,5 @@ resource conflicts on each node. .. _build_forces.sh: https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/scaling_tests/forces/forces_app/build_forces.sh .. _Perlmutter: https://www.youtube.com/watch?v=Av8ctYph7-Y .. _Spock: https://www.youtube.com/watch?v=XHXcslDORjU +.. _Polaris: https://youtu.be/Ff0dYYLQzoU .. _run_libe_forces.py: https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/scaling_tests/forces/forces_gpu/run_libe_forces.py From c1630067fce1f5e2501a4d5a3bc082e99bd6476a Mon Sep 17 00:00:00 2001 From: shudson Date: Thu, 9 Feb 2023 10:16:00 -0600 Subject: [PATCH 6/9] More forces gpu tutorial updates --- docs/tutorials/forces_gpu_tutorial.rst | 54 ++++++++++++++------------ 1 file changed, 29 insertions(+), 25 deletions(-) diff --git a/docs/tutorials/forces_gpu_tutorial.rst b/docs/tutorials/forces_gpu_tutorial.rst index 209e2742e..df8606ca6 100644 --- a/docs/tutorials/forces_gpu_tutorial.rst +++ b/docs/tutorials/forces_gpu_tutorial.rst @@ -3,27 +3,22 @@ Executor - Assign GPUs ====================== This tutorial shows the most portable way to assign tasks (user applications) -to the GPU. - -In the first example, each worker will be using one GPU. We assume the workers are on a -cluster with CUDA-capable GPUs. We will assign GPUs by setting the environment -variable ``CUDA_VISIBLE_DEVICES``. An equivalent approach can be used with other -devices. +to the GPU. The libEnsemble scripts in this example are available under +forces_gpu_ in the libEnsemble repository. This example is based on the :doc:`simple forces tutorial <../tutorials/executor_forces_tutorial>` with a slightly modified simulation function (to assign GPUs) and an increased number of particles (allows live GPU usage to be viewed). -Compile **forces.x** using one of the GPU build lines in build_forces.sh_ -or similar for your platform. - -The libEnsemble scripts in this example are available under forces_gpu_ in -the libEnsemble repository. +In the first example, each worker will be using one GPU. We assume the workers are on a +cluster with CUDA-capable GPUs. We will assign GPUs by setting the environment +variable ``CUDA_VISIBLE_DEVICES``. An equivalent approach can be used with other +devices. -Videos demonstrate running this example on Perlmutter_, Spock_, and Polaris_. Note, in -the first two videos are from an earlier release - you no longer need to change -particle count or modify the `forces.c` file). +Videos demonstrate running this example on Perlmutter_, Spock_, and Polaris_. +*The first two videos are from an earlier release - you no longer need to change +particle count or modify the `forces.c` file).* Simulation function ------------------- @@ -129,6 +124,16 @@ Alternative environment variables can be simply substituted in ``set_env_to_slot On some systems ``CUDA_VISIBLE_DEVICES`` may be overridden by other assignments such as ``--gpus-per-task=1`` + +Compiling the Forces application +-------------------------------- + +First compile the forces application under the ``forces_app`` directory. + +Compile **forces.x** using one of the GPU build lines in build_forces.sh_ +or similar for your platform. + + Running the example ------------------- @@ -180,9 +185,13 @@ Further guidance on varying resource to workers can be found under the Checking GPU usage ------------------ -You can check you are running forces on the GPUs as expected by using profiling tools and/or by using -a monitoring utility. For NVIDIA GPUs, for example, the **Nsight** profiler is generally available -and can be run from the command line. To simply run `forces.x` stand-alone you could run:: +The output of `forces.x` will say if it has run on the host or device. When running +libEnsemble, this can be found under the ``ensemble`` direcotry. + +You can check you are running forces on the GPUs as expected by using profiling tools and/or +by using a monitoring utility. For NVIDIA GPUs, for example, the **Nsight** profiler is +generally available and can be run from the command line. To simply run `forces.x` stand-alone +you could run:: nsys profile --stats=true mpirun -n 2 ./forces.x @@ -199,15 +208,10 @@ runs for long enough to register on the monitor, so lets try 100,000 particles:: It is also recommended that you run without the profiler when using the `nvidia-smi` utility. This can also be used when running via libEnsemble, so long as you are on the node where the -forces applications are being run. As the default particles in the forces example is 1000, you -will need to to increase particles to see clear GPU usage in the live monitor. E.g.,~ in line 14 -to multiply the particles by 10:: - - # Parse out num particles, from generator function - particles = str(int(H["x"][0][0]) * 10) +forces applications are being run. -Alternative monitoring devices include ``rocm-smi`` (AMD) and ``intel_gpu_top`` (Intel). The latter -does not need the *watch* command. +Alternative monitoring devices include ``rocm-smi`` (AMD) and ``intel_gpu_top`` (Intel). +The latter does not need the *watch* command. Example submission script ------------------------- From cdc22c623057c82d28fb065c036d0ea235fbbd85 Mon Sep 17 00:00:00 2001 From: shudson Date: Thu, 9 Feb 2023 14:22:46 -0600 Subject: [PATCH 7/9] Fix typos and grammer --- docs/tutorials/forces_gpu_tutorial.rst | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/tutorials/forces_gpu_tutorial.rst b/docs/tutorials/forces_gpu_tutorial.rst index df8606ca6..9f13a0bda 100644 --- a/docs/tutorials/forces_gpu_tutorial.rst +++ b/docs/tutorials/forces_gpu_tutorial.rst @@ -128,7 +128,7 @@ Alternative environment variables can be simply substituted in ``set_env_to_slot Compiling the Forces application -------------------------------- -First compile the forces application under the ``forces_app`` directory. +First, compile the forces application under the ``forces_app`` directory. Compile **forces.x** using one of the GPU build lines in build_forces.sh_ or similar for your platform. @@ -143,22 +143,22 @@ eight workers. For example:: python run_libe_forces.py --comms local --nworkers 8 Note that if you are running one persistent generator that does not require -resources, then assign nine workers, and fix the number of *resource_sets* in -you calling script:: +resources, then assign nine workers and fix the number of *resource_sets* in +your calling script:: libE_specs["num_resource_sets"] = 8 See :ref:`zero resource workers` for more ways to express this. -Changing number of GPUs per worker ----------------------------------- +Changing the number of GPUs per worker +-------------------------------------- If you want to have two GPUs per worker on the same system (four GPUs per node), you could assign only four workers, and change line 24 to:: resources.set_env_to_slots("CUDA_VISIBLE_DEVICES", multiplier=2) -In this case there are two GPUs per worker (and per slot). +In this case, there are two GPUs per worker (and per slot). Varying resources ----------------- @@ -172,21 +172,21 @@ In the generator function, assign the ``resource_sets`` field of if a larger simulation requires two MPI tasks (and two GPUs), set ``resource_sets`` field to *2* for that sim_id in the generator function. -The calling script run_libe_forces.py_ contains alternative commented out lines for +The calling script run_libe_forces.py_ contains alternative commented-out lines for a variable resource example. Search for "Uncomment for var resources" In this case, the simulator function will still work, assigning one CPU processor and one GPU to each MPI rank. If you want to have one rank with multiple GPUs, then change source lines 29/30 accordingly. -Further guidance on varying resource to workers can be found under the -:doc:`resource manager<../resource_manager/resources_index>`. +Further guidance on varying the resources assigned to workers can be found under the +:doc:`resource manager<../resource_manager/resources_index>` section. Checking GPU usage ------------------ The output of `forces.x` will say if it has run on the host or device. When running -libEnsemble, this can be found under the ``ensemble`` direcotry. +libEnsemble, this can be found under the ``ensemble`` directory. You can check you are running forces on the GPUs as expected by using profiling tools and/or by using a monitoring utility. For NVIDIA GPUs, for example, the **Nsight** profiler is @@ -201,7 +201,7 @@ running (this may entail using *ssh* to get on to the node), and run:: watch -n 0.1 nvidia-smi This will update GPU usage information every 0.1 seconds. You would need to ensure the code -runs for long enough to register on the monitor, so lets try 100,000 particles:: +runs for long enough to register on the monitor, so let's try 100,000 particles:: mpirun -n 2 ./forces.x 100000 From 2c95ba59058dda3f43892583e2d23ad4b243cfa8 Mon Sep 17 00:00:00 2001 From: shudson Date: Thu, 9 Feb 2023 15:17:24 -0600 Subject: [PATCH 8/9] Remove old text --- docs/platforms/polaris.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/platforms/polaris.rst b/docs/platforms/polaris.rst index 434b6e0fd..94e269f94 100644 --- a/docs/platforms/polaris.rst +++ b/docs/platforms/polaris.rst @@ -34,8 +34,8 @@ environment (if you need conda install). More details at `Python for Polaris`_. python -m venv /path/to-venv --system-site-packages . /path/to-venv/bin/activate - Where ``/path/to-venv`` can be anywhere you have write access. You now have access to - For future uses just load the conda module and run the activate line. + Where ``/path/to-venv`` can be anywhere you have write access. For future sessions, + just load the conda module and run the activate line. You can now pip install libEnsemble:: From 19b2a79764f9882b17f2c6902a2f557d9b4048a4 Mon Sep 17 00:00:00 2001 From: Jeffrey Larson Date: Tue, 14 Feb 2023 16:47:02 -0600 Subject: [PATCH 9/9] Small edits --- docs/platforms/polaris.rst | 14 +++++++------- docs/tutorials/forces_gpu_tutorial.rst | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/platforms/polaris.rst b/docs/platforms/polaris.rst index 94e269f94..29255cc91 100644 --- a/docs/platforms/polaris.rst +++ b/docs/platforms/polaris.rst @@ -2,7 +2,7 @@ Polaris ======= -Polaris_ is a 560 node HPE system located in the ALCF_ at Argonne +Polaris_ is a 560-node HPE system located in the ALCF_ at Argonne National Laboratory. The compute nodes are equipped with a single AMD EPYC Milan processor and four A100 NVIDIA GPUs. It uses the PBS scheduler to submit jobs from login nodes to run on the compute nodes. @@ -17,11 +17,11 @@ Python and libEnsemble are available on Polaris with the `conda` module. Load th module load conda conda activate base -This also gives you access to machine optimized packages such as mpi4py_. +This also gives you access to machine-optimized packages such as mpi4py_. To install further packages, including updating libEnsemble, you may either create -a virtual environment on top of this (if just using pip install), or clone the base -environment (if you need conda install). More details at `Python for Polaris`_. +a virtual environment on top of this (if just using ``pip install``) or clone the base +environment (if you need ``conda install``). More details at `Python for Polaris`_. .. container:: toggle @@ -35,7 +35,7 @@ environment (if you need conda install). More details at `Python for Polaris`_. . /path/to-venv/bin/activate Where ``/path/to-venv`` can be anywhere you have write access. For future sessions, - just load the conda module and run the activate line. + just load the ``conda`` module and run the activate line. You can now pip install libEnsemble:: @@ -97,8 +97,8 @@ Demonstration ------------- For an example that runs a small ensemble using a C application (offloading work to the -GPU), see :doc:`forces_gpu<../tutorials/forces_gpu_tutorial>` tutorial. A video demonstration_ -of this example is available. +GPU), see the :doc:`forces_gpu<../tutorials/forces_gpu_tutorial>` tutorial. A video demonstration_ +of this example is also available. .. _Polaris: https://www.alcf.anl.gov/polaris diff --git a/docs/tutorials/forces_gpu_tutorial.rst b/docs/tutorials/forces_gpu_tutorial.rst index 9f13a0bda..dd84904d1 100644 --- a/docs/tutorials/forces_gpu_tutorial.rst +++ b/docs/tutorials/forces_gpu_tutorial.rst @@ -169,7 +169,7 @@ calling script. In the generator function, assign the ``resource_sets`` field of :ref:`H` for each point generated. For example -if a larger simulation requires two MPI tasks (and two GPUs), set ``resource_sets`` +if a larger simulation requires two MPI tasks (and two GPUs), set the ``resource_sets`` field to *2* for that sim_id in the generator function. The calling script run_libe_forces.py_ contains alternative commented-out lines for