Building DGSWEM HPX on Hermione

Getting an account on Hermione:

Contact Zach Byerly (zbyerly@cct.lsu.edu)

Interactive slurm session

Hermione uses SLURM To start an interactive session on one of the Marvin nodes, run: srun -t 3:00:00 --partition=marvin --pty bash -i

Using pre-built libraries:

Pre-built HPX and LGD libraries are located in: /home/zbyerly/local_install_debug and /home/zbyerly/local_install

Clone the dgswem-hpx branch: git clone -b dgswem-hpx git@github.com:foci/dgswem.git dgswem-hpx

Create build directory: mkdir dgswem-hpx/build cd dgswem-hpx/build

CMake command (debug): cmake -DCMAKE_CXX_COMPILER=/opt/gcc/5.2/bin/g++ -DCMAKE_C_COMPILER=/opt/gcc/5.2/bin/gcc -DCMAKE_PREFIX_PATH=/home/zbyerly/local_install_debug/ -DCMAKE_BUILD_TYPE=Debug ..

CMake command (release): cmake -DCMAKE_CXX_COMPILER=/opt/gcc/5.2/bin/g++ -DCMAKE_C_COMPILER=/opt/gcc/5.2/bin/gcc -DCMAKE_PREFIX_PATH=/home/zbyerly/local_install/ -DCMAKE_BUILD_TYPE=Release ..

CMake command (no LGD): cmake -DNO_LGD=true ..

CMake command (no LGD, fortran-only serial): cmake -DNO_LGD=true -DFORTRAN_SERIAL=true ..

Building Executables

After this is done, several executables can be built:

dgswem_hpx - HPX version of DGSWEM. Runs in parallel, using multiple domains.

dgswem_cpp_serial - Serial cpp version of the code. doesn't use LGD or HPX, but operates on multiple domains.

dgswem_fortran_serial - currently not working. intended to run on a non-decomposed mesh. (Must be built with CMake flag -DFORTRAN_SERIAL=true)

adcpost - combines output files into single file.

error - currently may not be working. Compares output from two different runs, must combine output files using adcpost first.

Running dgswem_hpx

First, add the gcc 5.2 library path:

export LD_LIBRARY_PATH=/opt/gcc/5.2/lib64:$LD_LIBRARY_PATH

Run from inside a directory with a decomposed grid. Currently, the total number of timesteps is hard-coded into the dgswem_hpx.cpp source code. /PATH/TO/dgswem-hpx/build/src/dgswem_hpx --hpx:print-counter=/threads{locality#0}/count/cumulative --hpx:ini=hpx.stacks.small_size=0x20000 --n_timesteps 1000 --hpx:threads 4

The number of threads is determined by --hpx:threads <number of threads>.

The number of timesteps is determined by --n_timesteps <number of timesteps>.

There are a variety of different HPX performance counters that can be used: HPX Performance Counters.

Running on multiple localities:

srun -t 9:00:00 -N 2 --partition=marv_noht ~/dgswem-hpx-dev/build_release/src/dgswem_hpx --hpx:print-counter=/threads{locality#*}/count/cumulative --hpx:ini=hpx.stacks.small_size=0x20000 --n_timesteps 1000 --hpx:localities 2