From 7c6a990776f86c0b944bb33f91d9855d42384349 Mon Sep 17 00:00:00 2001 From: Jered Dominguez-Trujillo Date: Thu, 29 Jun 2023 14:42:36 -0600 Subject: [PATCH 1/3] USE_CALIPER flag for Ume and Neel as co-author --- README.md | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 54 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 75e04f7..a53d967 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ Client/Controller design to service sparse memory requests (0, 1, and 2 levels o - OpenMP (for calibration test) - Python3 (for scripts) - MPI (for use with Ume client) +- Caliper (for profiling Ume client) ### Submodules - [Ume](https://github.com/lanl/UME) @@ -59,8 +60,7 @@ make | Option | Description | Default | Status | Compile Definitions (Pre-Processor) | | -------------------- | ----------------------------------------- | ------- | ------------ | ----------------------------------- | | USE\_MPI | Build with MPI | OFF | Complete | | -| UME\_PROFILING | Build with Profiling Enabled | OFF | Complete | UME\_PROFILING | - +| USE\_CALIPER | Build with Caliper | OFF | Complete | | #### Intrinsics @@ -122,6 +122,36 @@ cmake -DScoria_REQUIRE_CALIBRATION_TESTS=ON -DScoria_REQUIRE_AVX=ON -DScoria_REQ make ``` +Build UME with AVX Extension and Scoria +``` +cmake -DUSE_MPI=ON -DUSE_CALIPER -DREQUIRE_AVX +make +``` + +The `ume_mpi` and `ume_serial` executables should be in the clients/UME/src directory + +Build UME + Scoria with Caliper Profiling and AVX +``` +mkdir caliper_build +cd caliper_build +cmake -DUSE_CALIPER=ON -DUSE_MPI=ON -DScoria_REQUIRE_AVX=ON .. +make +``` + +The `ume_mpi` and `ume_serial` executables should be in the clients/UME/src directory + +Build baseline (non-scoria) UME for Caliper profiling without AVX +``` +cd clients/UME +mkdir caliper_build +cd caliper_build +cmake -DUSE_CALIPER=ON -DUSE_MPI=ON ../ +make +``` + +The `ume_mpi` and `ume_serial` executables should be in the src directory + + The `test`, `test_clients`, `test_calibration`, and `test_calibration_client` executables should be in the `tests` directory, the `simple_client` and `spatter` executables should be in the `clients` directory, along with the `scoria` executable in the base build directory. The `test_clients` and `test_calibration_client` executables, when ran with the `scoria` controller, should now output both internal and external bandwidth measurements and timings. @@ -209,14 +239,14 @@ hwloc-bind node:0 ./scoria Terminal window 2 ``` -hwloc-bind node:0 ./tests/test_client 1048576 +hwloc-bind node:0 ./tests/test_client ``` ### Scripts **Note: To use the scripts, Scoria must have been built without internal timing, i.e. `-DScoria_REQUIRE_TIMING=OFF`** -The `scripts/simple_test_bw.py` contains a script to launch both Scoria and the test client. It is configurable with the following options: +`scripts/simple_test_bw.py` contains a script to launch both Scoria and the test client. It is configurable with the following options: | Short Option | Long Option | Description | Default | | ------------ | ----------- | ------------------------------------------ | ---------- | @@ -228,6 +258,16 @@ The `scripts/simple_test_bw.py` contains a script to launch both Scoria and the The output will be a log file with the bandwidth data and bar charts of the bandwidth for each test at each thread count. If AVX or SVE is enabled, those results will be saved to an individual figure with the appropriate name. +`scripts/scoria-vs-ume.sh` contains a script to build Ume + Scoria with AVX and Caliper enabled, and to build a standalone Ume executable with Caliper. It then runs both with an Ume input file of your choosing and with the specified number of ranks, and outputs profiling data in the form of a text file or a JSON file that can be read by Hatchet. It is configurable with the following options: + +| Short Option | Description | Default | +| ------------ | --------------------------------------------------- | --------------------------------- | +| -c | (Optional) CALI\_CONFIG setting | runtime-report(output=report.log) | +| -f | Absolute path to Input Deck for Ume | None | +| -n | (Optional) Number of ranks to use to launch MPI run | 1 | +| -s | (Optional) Scoria root directory | pwd | +| -p | (Optional) List of PAPI Counters to collect | None | + #### Example ``` @@ -235,6 +275,15 @@ cd scoria python3 scripts/simple_test_bw.py -l output.log -p scoria.png -n 8388608 -s node:0 -b node:0 ``` +``` +bash scripts/scoria-vs-ume.sh -c "hatchet-region-profile" -n -f +``` + +``` +bash scripts/scoria-vs-ume.sh -n -f -p "PAPI_DP_OPS,PAPI_TOT_CYC,PAPI_TOT_INS,PAPI_LD_INS,PAPI_SR_INS,PAPI_BR_INS,PAPI_LST_INS" +``` + + ## License [![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause) @@ -244,3 +293,4 @@ Triad National Security, LLC (Triad) owns the copyright to Scoria. The license i ## Authors and acknowledgment - Jered Dominguez-Trujillo, [_jereddt@lanl.gov_](mailto:jereddt@lanl.gov) - Jonas Lippuner, [_jonas@l2quant.com_](mailto:jonas@l2quant.com) [_jlippuner@lanl.gov_](mailto:jlippuner@lanl.gov) +- Neel Patel, [_nmpatel@lanl.gov_](mailto:nmpatel@lanl.gov) [_nmpatel@kansas.edu_](mailto:nmpatel@kansas.edu) From 90e5cbe04d297a0aeaebb8f6170f7b30762b76f9 Mon Sep 17 00:00:00 2001 From: Jered Dominguez-Trujillo Date: Thu, 29 Jun 2023 14:42:55 -0600 Subject: [PATCH 2/3] Scoria + Ume vs Ume Caliper script --- scripts/scoria-vs-ume.sh | 75 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100755 scripts/scoria-vs-ume.sh diff --git a/scripts/scoria-vs-ume.sh b/scripts/scoria-vs-ume.sh new file mode 100755 index 0000000..82ed153 --- /dev/null +++ b/scripts/scoria-vs-ume.sh @@ -0,0 +1,75 @@ +#!/bin/bash +usage() { + echo "Usage ./scoria-vs-ume.sh -f [-c 'caliconfig'] [-n 'ranks'] [-s 'scoria-root'] + -f : Specify the absolute path to an Ume input file + -c : Optional, CALI_CONFIG environment variable (Default: 'runtime-report(output=report.log)') + -n : Optional, Specify the number of MPI ranks to run Ume with (Default: 1) + -s : Optional, Root directory of the Scoria project (Default: pwd) + -p : Optional, List of PAPI hardware counters to collect" +} + +CALI_CONFIG="runtime-report(output=report.log)" +RANKS=1 +SCORIA_OPUS=`pwd` + +while getopts "f:c:n:s:p:" opt; do + case $opt in + c) CALI_CONFIG=$OPTARG + ;; + f) INPUT=$OPTARG + ;; + n) RANKS=$OPTARG + ;; + s) SCORIA_OPUS=$OPTARG + ;; + p) + export CALI_SERVICES_ENABLE=event:papi:trace:recorder + export CALI_PAPI_COUNTERS="${OPTARG}" + export CALI_RECORDER_FILENAME=papi-counters-${CALI_PAPI_COUNTERS}.cali + export CALI_PAPI_ENABLE_MULTIPLEXING="true" + ;; + h) usage; exit 1 + ;; + esac +done + +if [ -z "${INPUT}" ]; then + usage + exit 1 +fi + +echo "Running scoria-vs-ume.sh" +echo "CALI_CONFIG: ${CALI_CONFIG}" +echo "INPUT: ${INPUT}" +echo "RANKS: ${RANKS}" +echo "SCORIA_ROOT: ${SCORIA_OPUS}" + +cd $SCORIA_OPUS + +SCORIA_BUILD=$SCORIA_OPUS/caliper_build +mkdir -p $SCORIA_BUILD + +UME_OPUS=$SCORIA_OPUS/clients/UME +UME_BUILD=$UME_OPUS/caliper_build +mkdir -p $UME_BUILD + +export CALI_CONFIG=${CALI_CONFIG} + +scoria_ume_mpi=$SCORIA_BUILD/clients/UME/src/ume_mpi +ume_mpi=$UME_BUILD/src/ume_mpi + +# Ume + Scoria +cd $SCORIA_BUILD +cmake -DUSE_CALIPER=ON -DScoria_REQUIRE_AVX=ON -DUSE_MPI=ON ../ +make -j + +RELINPUT=`realpath --relative-to=${PWD} $INPUT` +mpirun -n $RANKS $scoria_ume_mpi $RELINPUT + +# Ume +cd $UME_BUILD +cmake -DUSE_CALIPER=ON -DUSE_MPI=ON ../ +make -j + +RELINPUT=`realpath --relative-to=${PWD} $INPUT` +mpirun -n $RANKS $ume_mpi $RELINPUT From 300bf9551cedcc4a0fd108ffadb9c99c94784c84 Mon Sep 17 00:00:00 2001 From: Jered Dominguez-Trujillo Date: Thu, 29 Jun 2023 17:06:01 -0600 Subject: [PATCH 3/3] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a53d967..428fcf7 100644 --- a/README.md +++ b/README.md @@ -239,7 +239,7 @@ hwloc-bind node:0 ./scoria Terminal window 2 ``` -hwloc-bind node:0 ./tests/test_client +hwloc-bind node:0 ./tests/test_client 1048576 ``` ### Scripts