Fix srun invocation on Great Lakes.

[joaander]

Description

Switch back to mpirun on Great Lakes.

Motivation and Context

This change was made in #722. My workflows (hoomd-validation) work correctly with the most current release of flow, but not the main branch. When flow forks to srun, somehow my environment variables are all lost. For example, one of errors I got testing this was singularity not found even though I checked in the shell and singularity was in the PATH and the filesystem where singularity was stored was accessible.

Strangely, everything works if I directly called the srun command from the shell:

srun -n 4  /home/joaander/miniforge3/bin/python /gpfs/accounts/sglotzer_root/sglotzer0/joaander/hoomd-validation/hoomd_validation/project.py exec hard_disk_nvt_gpu agg-88ab2b6305a8b249743764fc16e5c22e

I don't know what causes this behavior, but the expedient solution is to switch back to mpirun which we have used for a long time on Great Lakes with fewer issues.

Checklist:

• I am familiar with the Contributing Guidelines.
• I agree with the terms of the Contributor Agreement.
• My name is on the list of contributors.
• The changes introduced by this pull request are covered by existing or newly introduced tests.
• The package documentation and framework documentation in signac-docs are up to date with these changes.
• I have updated the changelog and added any related issue and pull request numbers for future reference.

[codecov]

Codecov Report

Merging #779 (b65d02a) into main (9ba71de) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main     #779   +/-   ##
=======================================
  Coverage   69.11%   69.11%           
=======================================
  Files          45       45           
  Lines        4348     4348           
  Branches     1055      979   -76     
=======================================
  Hits         3005     3005           
  Misses       1137     1137           
  Partials      206      206           

Files	Coverage Δ
flow/environments/umich.py	85.71% <100.00%> (ø)

[joaander]

Wait, this needs more testing on GPU nodes where srun configures the appropriate cgroups and mpirun does not. I may need to find a fix for the environment variable issue.

Edit: Never mind, I was misinterpreting the output. srun and mpirun on great lakes both allocate all assigned GPUs to all ranks on a given node. cgroups limit users to the assigned nodes, but do not assign individual gpus to ranks.

[joaander]

Upon further testing with a 4 V100 GPU job (which slurm assigned 3 GPUs to 1 node and 1 GPU to another node), mpirun assigns multiple tasks to the same GPU while srun correctly distributes the tasks. Note: we still need --gpus-per-task which I will add separately when working on #777.

I found that srun --export=ALL solved the environment variable forwarding issue. I also found that srun buffered stdout/stderr until the job completed, so I also added -u to disable buffering.

srun -u --export=ALL correctly runs hoomd-validation jobs both when hoomd is compiled from source and from the glotzerlab-software container. I am waiting for one more test on the 4 GPU A100 node.

[joaander]

This works on both V100 and A100 nodes on Great Lakes.

[joaander]

@b-butler This PR now includes both fixes to the Great Lakes environment necessary to make the template functional for the next release.

• Use cpus-per-task to ensure all processes requested via np are placed on one node.
• Use gpus-per-task to allow multi-GPU jobs to correctly distribute to the assigned GPUs.

This formulation works only with homogenous jobs. How do I call homogeneous_openmp_mpi_config to ensure this?

I did not fix --cpus-per-task for all environments, as we are planning a more general fix in #777.

[b-butler]

I made some changes to the implementation (moving to the template) since we are refactoring and nranks will be removed. The code is equivalent.