Skip to content

Commit

Permalink
Allow to specify MPI launcher and its params at test time
Browse files Browse the repository at this point in the history
This commit introduces the following environment variables that affect
`make test` (or `ctest`) behavior of ALPSCore:

| Variable                      | Default                  | Usual value    | Meaning                                                           |
| `ALPS_TEST_MPIEXEC`           | `${MPIEXEC}`             | `mpiexec`      | MPI launcher                                                      |
| `ALPS_TEST_MPI_NROC_FLAG`     | `${MPIEXEC_NUMPROC_FLAG}`| `-n`           | flag to specify the number of MPI processes                       |
| `ALPS_TEST_MPI_NPROC`         | 1                        | 1              | How many MPI processes to launch in MPI-enabled tests             |
| `ALPS_TEST_MPIEXEC_PREFLAGS`  | `${MPIEXEC_PREFLAGS}`    | (empty string) | MPI launcher arguments preceding the executable name              |
| `ALPS_TEST_MPIEXEC_POSTFLAGS` | `${MPIEXEC_POSTFLAGS}`   | (empty string) | MPI launcher arguments preceding the arguments for the executable |

The `${...}` above are CMake variables, normally set by `find_MPI` module.

Related: issue #211.

This should close #296.

Intended use:

** Case 1: Vanilla MPI-enabled environment. **

The command to run an MPI program using 2 processes: `mpiexec -n 2 some_test`

Setting the variables to run each MPI-enabled tests on 2 processes: `ALPS_TEST_MPI_NPROC=2 make test`

** Case 2: NERSC Cori **

(Disclaimer: not tested with an actual Cori run)

Users are not supposed to run `mpiexec`. One has to allocate interactive nodes first.

Allocating 2 Haswell nodes for 30 minutes: `salloc -N 2 -C haswell -q interactive -t 0:30:00`

Command to run on the allocated nodes: `srun some_test`

Setting the variables to run each MPI-enabled tests on the allocated nodes:
`ALPS_TEST_MPIEXEC=srun ALPS_TEST_MPI_NPROC=' ' ALPS_TEST_MPI_NPROC_FLAG=' ' make test`

(note the variables are assigned spaces, not empty strings!)

** Case 3: Blue Waters **

(Disclaimer: not tested on actual Blue Waters machine)

The `aprun` command is supposed to be used to launch parallel processes from an interactive node
(see https://bluewaters.ncsa.illinois.edu/using-aprun ).

Command to run on 16 cores, using 8 cores per node (that is, 2 nodes), placing the processes on adjacent cores:
`aprun -N 8 -d 1 -n 16 some_test`

Setting the variables to run each MPI-enabled tests with this configuration:
`ALPS_TEST_MPIEXEC=aprun ALPS_TEST_MPI_NPROC=16 ALPS_TEST_MPI_NPROC_FLAG='-N 8 -d 1 -n' make test`
  • Loading branch information
galexv committed Mar 29, 2019
1 parent c1be569 commit 83febef
Showing 1 changed file with 13 additions and 4 deletions.
17 changes: 13 additions & 4 deletions common/cmake/ALPSEnableTests.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -120,10 +120,19 @@ function(alps_add_gtest test)
set(run_ok_cmd_ "$<TARGET_FILE:${test}> ${test_xml_output_}")
set(shell_cmd_ "if ${build_cmd_}\; then ${run_fail_cmd_}\; false\; else ${run_ok_cmd_}\; fi")
set(cmd_ "/bin/sh" "-c" "${shell_cmd_}")
elseif (arg_PARTEST AND MPIEXEC)
# FIXME: if compiler supports MPI directly, the MPIEXEC program is not deduced!
# FIXME: in the MPI test command, POSIX shell is assumed
set(cmd_ "/bin/sh" "-c" "${MPIEXEC} ${MPIEXEC_NUMPROC_FLAG} \${ALPS_TEST_MPI_NPROC:-1} ${MPIEXEC_PREFLAGS} $<TARGET_FILE:${test}> ${MPIEXEC_POSTFLAGS} ${test_xml_output_}")
elseif (arg_PARTEST AND ALPS_HAVE_MPI)
# if unspecified, we assume the standard mpi launcher, according to MPI-3.1 specs, Chapter 8.8
# (see https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report/node228.htm#Node228)
if (NOT MPIEXEC)
set(MPIEXEC "mpiexec")
endif()
if (NOT MPIEXEC_NUMPROC_FLAG)
set(MPIEXEC_NUMPROC_FLAG "-n")
endif()

# we also allow test-time override of the MPI launcher and its arguments
# (NOTE: POSIX shell is assumed)
set(cmd_ "/bin/sh" "-c" "\${ALPS_TEST_MPIEXEC:-${MPIEXEC}} \${ALPS_TEST_MPI_NPROC_FLAG:-${MPIEXEC_NUMPROC_FLAG}} \${ALPS_TEST_MPI_NPROC:-1} \${ALPS_TEST_MPIEXEC_PREFLAGS:-${MPIEXEC_PREFLAGS}} $<TARGET_FILE:${test}> \${ALPS_TEST_MPIEXEC_POSTFLAGS:-${MPIEXEC_POSTFLAGS}} ${test_xml_output_}")
else()
set(cmd_ $<TARGET_FILE:${test}> ${test_xml_output_})
endif()
Expand Down

0 comments on commit 83febef

Please sign in to comment.