From 82831b670a509945c84de49228207d88ca849321 Mon Sep 17 00:00:00 2001
From: boeschf <48126478+boeschf@users.noreply.github.com>
Date: Mon, 19 May 2025 18:01:36 +0200
Subject: [PATCH 1/9] port from confluence

---
 docs/running/jobreport.md | 272 ++++++++++++++++++++++++++++++++++++++
 mkdocs.yml                |   1 +
 2 files changed, 273 insertions(+)
 create mode 100644 docs/running/jobreport.md

diff --git a/docs/running/jobreport.md b/docs/running/jobreport.md
new file mode 100644
index 00000000..7874d424
--- /dev/null
+++ b/docs/running/jobreport.md
@@ -0,0 +1,272 @@
+[](){#ref-jobreport}
+# Job report
+
+A batch job summary report is often requested in project proposals at CSCS to demonstrate the effective use of GPUs.
+[jobreport](https://github.com/eth-cscs/alps-jobreport/releases) is used in two stages.
+The first stage monitors an application and records the GPU usage statistics.
+The monitoring stage must be executed within a `slurm` environment.
+The information is recorded as `.csv` data within a directory  `jobreport_${SLURM_JOB_ID}` or a directory supplied on the command line.
+The second stage prints this information in a tabular form that can be inserted into a project proposal.
+
+## Downloading the job summary report
+
+Precompiled binary for the `jobreport` utility can be obtained directly from the [repository](https://github.com/eth-cscs/alps-jobreport/releases) or via the command line:
+
+```console
+$ wget https://github.com/eth-cscs/alps-jobreport/releases/download/v0.1/jobreport
+$ chmod +x ./jobreport
+```
+## Command line options
+
+A full list of command line options with explanations can be obtained by running the command with the `--help` option:
+
+```console
+$ ./jobreport --help
+Usage: jobreport [-v -h] [subcommand] -- COMMAND
+
+Options:
+  -h, --help                        Show this help message
+  -v, --version                     Show version information
+
+Subcommands:
+  monitor                           Monitor the performance metrics for a job. (Default)
+    -h, --help                      Shows help message
+    -o, --output <path>             Specify output directory (default: ./jobreport_<SLURM_JOB_ID>)
+    -u, --sampling_time <seconds>   Set the time between samples (default: automatically determined)
+    -t, --max_time <time>           Set the maximum monitoring time (format: DD-HH:MM:SS, default: 24:00:00)
+  print                             Print a job report
+    -h, --help                      Shows help message
+    -o, --output <path>             Output path for the report file
+  container-hook                    Write enroot hook for jobreport
+    -h, --help                      Shows help message
+    -o, --output <path>             Output path for the enroot hook file
+                                    (default: $HOME/.config/enroot/hooks.d/cscs_jobreport_dcgm_hook.sh)
+
+Arguments:
+  COMMAND                           The command to run as the workload
+```
+
+## Reported Information
+
+The final output from `jobreport` is a table summarizing the most important details of how your application used the compute resources during its execution.
+The report is divided into two parts: a general summary and GPU specific values.
+
+### Job statistics
+
+| Field | Description |
+| ----- | ----------- |
+| Job Id | The Slurm job id |
+| Step Id | The slurm step id. A job step in SLURM is a subdivision of a job started with srun |
+| User | The user account that submitted the job |
+| SLURM Account | The project account that will be billed |
+| Start Time, End Time, Elapsed Time | The time the job started and ended, and how long it ran |
+| Number of Nodes | The number of nodes allocated to the job |
+| Number of GPUs | The number of GPUs allocated to the job |
+| Total Energy Consumed | The total energy consumed based on the average power usage (below) over the elapsed time |
+| Average Power Usage | The average power draw over the elapsed time in Watts (W), summed over all GPUs |
+| Average SM Utilization | The percentage of the process's lifetime during which Streaming Multiprocessors (SM) were executing a kernel, averaged over all GPUs |
+| Average Memory Utilization | The percentage of a process's lifetime during which global (device) memory was being read or written, averaged over all GPUs |
+
+### GPU specific values
+
+| Field | Description |
+| ----- | ----------- |
+| Host | The compute node executing a job step |
+| GPU | The GPU id on a node |
+| Elapsed | The elapsed time |
+| SM Utilization % | The percentage of the process's lifetime during which Streaming Multiprocessors (SM) were executing a kernel |
+| Memory Utilization % | The percentage of process's lifetime during which global (device) memory was being read or written |
+
+## Example with slurm: srun
+
+The simplest example to test `jobreport` is to run it with the sleep command.
+It is important to separate `jobreport` (and its options) and your command  with `--`.
+
+```console
+$ srun -A my_account -t 5:00 --nodes=1 ./jobreport -- sleep 5
+$ ls
+jobreport_16133
+$ ./jobreport print jobreport_16133
+Summary of Job Statistics
++-----------------------------------------+-----------------------------------------+
+| Job Id                                  | 16133                                   |
++-----------------------------------------+-----------------------------------------+
+| Step Id                                 | 0                                       |
++-----------------------------------------+-----------------------------------------+
+| User                                    | jpcoles                                 |
++-----------------------------------------+-----------------------------------------+
+| SLURM Account                           | unknown_account                         |
++-----------------------------------------+-----------------------------------------+
+| Start Time                              | 03-07-2024 15:32:24                     |
++-----------------------------------------+-----------------------------------------+
+| End Time                                | 03-07-2024 15:32:29                     |
++-----------------------------------------+-----------------------------------------+
+| Elapsed Time                            | 5s                                      |
++-----------------------------------------+-----------------------------------------+
+| Number of Nodes                         | 1                                       |
++-----------------------------------------+-----------------------------------------+
+| Number of GPUs                          | 4                                       |
++-----------------------------------------+-----------------------------------------+
+| Total Energy Consumed                   | 0.5 Wh                                  |
++-----------------------------------------+-----------------------------------------+
+| Average Power Usage                     | 348.8 W                                 |
++-----------------------------------------+-----------------------------------------+
+| Average SM Utilization                  | 0%                                      |
++-----------------------------------------+-----------------------------------------+
+| Average Memory Utilization              | 0%                                      |
++-----------------------------------------+-----------------------------------------+
+
+GPU Specific Values
++---------------+------+------------------+------------------+----------------------+
+| Host          | GPU  | Elapsed          | SM Utilization % | Memory Utilization % |
+|               |      |                  | (avg/min/max)    | (avg/min/max)        |
++---------------+------+------------------+------------------+----------------------+
+| nid006212     | 0    | 5s               |   0 /   0 /   0  |   0 /   0 /   0      |
+| nid006212     | 1    | 5s               |   0 /   0 /   0  |   0 /   0 /   0      |
+| nid006212     | 2    | 5s               |   0 /   0 /   0  |   0 /   0 /   0      |
+| nid006212     | 3    | 5s               |   0 /   0 /   0  |   0 /   0 /   0      |
++---------------+------+------------------+------------------+----------------------+
+```
+
+!!! warning "`jobreport` requires successful completion of the application"
+
+    The `jobreport` tool requires the application to complete successfully.
+    If the application crashes or the job is killed by `slurm` prematurely, `jobreport` will not be able to write any output.
+
+!!! info "workaround known issue on macOS"
+    Currently, there is an issue when generating the report file via jobreport print on the macOS terminal:
+    
+    ```console
+    what(): locale::facet::_S_create_c_locale name not valid
+    /var/spool/slurmd/job32394/slurm_script: line 21: 199992 Aborted         (core dumped) ./jobreport print report
+    ```
+
+    To fix this follow these steps:
+
+    1. Open the terminal application
+    2. In the top-left corner menu select Terminal -> Settings
+    3. Select your default profile
+    4. Uncheck "Set locale environment variables on startup"
+    5. Quit and reopen the terminal and try again. This should fix the issue.
+
+## Example with slurm: batch script
+
+The `jobreport` command can be used in a batch script
+The report printing, too, can be included in the script and does not need the `srun` command.
+
+```bash title="submit script with jobreport"
+#!/bin/bash
+#SBATCH -t 5:00
+#SBATCH --nodes=2
+
+srun ./jobreport -o report -- my_command
+./jobreport print report
+```
+
+
+When used within an job script, `jobreport` will work across multiple calls to `srun`.
+Each time `srun` is called, `slurm` creates a new job step and `jobreport` records data for each one.
+Multiple job steps running simultaneously are also allowed.
+The job report generated contains sections for each `slurm` job step.
+
+```bash title="submit script with multiple steps"
+#!/bin/bash
+#SBATCH -t 5:00
+#SBATCH --nodes=2
+
+srun ${JOBREPORT} -o report -- ${COMMAND}
+srun ${JOBREPORT} -o report -- ${COMMAND}
+
+srun --nodes=1 ./jobreport -o report -- my_command_1 &
+srun --nodes=1 ./jobreport -o report -- my_command_2 &
+
+wait
+```
+
+
+## Example with uenv
+
+The following example runs a program called `burn` that computes repeated matrix multiplications to stress the GPUs.
+It was built with, and requires to run the [prgenv-gnu][ref-uenv-prgenv-gnu].
+
+```console
+$ srun --uenv=prgenv-gnu/24.2:v1 -t 5:00 --nodes=1 --ntasks-per-node=4 --gpus-per-task=1 ${JOBREPORT} -o report -- ./burn --gpu=gemm -d 30
+
+$ ./jobreport print report
+Summary of Job Statistics
++-----------------------------------------+-----------------------------------------+
+| Job Id                                  | 15923                                   |
++-----------------------------------------+-----------------------------------------+
+| Step Id                                 | 0                                       |
++-----------------------------------------+-----------------------------------------+
+| User                                    | jpcoles                                 |
++-----------------------------------------+-----------------------------------------+
+| SLURM Account                           | unknown_account                         |
++-----------------------------------------+-----------------------------------------+
+| Start Time                              | 03-07-2024 14:54:48                     |
++-----------------------------------------+-----------------------------------------+
+| End Time                                | 03-07-2024 14:55:25                     |
++-----------------------------------------+-----------------------------------------+
+| Elapsed Time                            | 36s                                     |
++-----------------------------------------+-----------------------------------------+
+| Number of Nodes                         | 1                                       |
++-----------------------------------------+-----------------------------------------+
+| Number of GPUs                          | 4                                       |
++-----------------------------------------+-----------------------------------------+
+| Total Energy Consumed                   | 18.7 Wh                                 |
++-----------------------------------------+-----------------------------------------+
+| Average Power Usage                     | 1.8 kW                                  |
++-----------------------------------------+-----------------------------------------+
+| Average SM Utilization                  | 88%                                     |
++-----------------------------------------+-----------------------------------------+
+| Average Memory Utilization              | 43%                                     |
++-----------------------------------------+-----------------------------------------+
+
+GPU Specific Values
++---------------+------+------------------+------------------+----------------------+
+| Host          | GPU  | Elapsed          | SM Utilization % | Memory Utilization % |
+|               |      |                  | (avg/min/max)    | (avg/min/max)        |
++---------------+------+------------------+------------------+----------------------+
+| nid007044     | 0    | 36s              |  83 /   0 / 100  |  39 /   0 /  50      |
+| nid007044     | 0    | 36s              |  90 /   0 / 100  |  43 /   0 /  50      |
+| nid007044     | 0    | 36s              |  90 /   0 / 100  |  43 /   0 /  48      |
+| nid007044     | 0    | 36s              |  90 /   0 / 100  |  47 /   0 /  54      |
++---------------+------+------------------+------------------+----------------------+
+```
+
+!!! note "Using `jobreport` with other uenvs"
+
+    `jobreport` works with any uenv, not just `prgenv-gnu`.
+
+## Example with container-engine (CE)
+
+Running `jobreport` with the [container-engine (CE)][ref-container-engine] requires a little more setup to allow the CE to mount the required GPU library paths inside the container.
+
+A script to set up the mount points needs to be placed in `${HOME}/.config/enroot/hooks.d/`.
+This can be generated with the `jobreport` tool, and by default, the script will be placed in `${HOME}/.config/enroot/hooks.d/cscs_jobreport.sh`.
+
+```console title="Generate DCGM hook"
+$ ./jobreport container-hook
+Writing enroot hook to "/users/myuser/.config/enroot/hooks.d/cscs_jobreport_dcgm_hook.sh"
+Add the following to your container .toml file:
+
+[annotations]
+com.hooks.dcgm.enabled = "true"
+```
+
+As indicated by the output, the hook must be added to the container `.toml` file.
+
+```toml title="Example .toml file"
+[annotations]
+com.hooks.dcgm.enabled = "true"
+```
+
+Once the CE is configured, only the EDF file (here `my-edf.toml`) needs to be specified along with a call to `jobreport`:
+
+```console title="Run jobreport in a container"
+$ srun --environment=my-edf.toml ./jobreport -- sleep 5
+```
+
+!!! note "Using `jobreport` with other container images"
+
+    `jobreport` works with any container image, as long as the hook is set up and the EDF file has the correct annotation.
diff --git a/mkdocs.yml b/mkdocs.yml
index 31129c7c..7d9596ed 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -89,6 +89,7 @@ nav:
   - 'Running Jobs':
     - running/index.md
     - 'slurm': running/slurm.md
+    - 'jobreport': running/jobreport.md
   - 'Data Management and Storage':
     - storage/index.md
     - 'File Systems': storage/filesystems.md

From 6f60f430bd15c696ff213cb886d96ba105ed3190 Mon Sep 17 00:00:00 2001
From: boeschf <48126478+boeschf@users.noreply.github.com>
Date: Tue, 20 May 2025 13:10:01 +0200
Subject: [PATCH 2/9] Update docs/running/jobreport.md

Co-authored-by: Mikael Simberg <mikael.simberg@iki.fi>
---
 docs/running/jobreport.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/running/jobreport.md b/docs/running/jobreport.md
index 7874d424..763866c1 100644
--- a/docs/running/jobreport.md
+++ b/docs/running/jobreport.md
@@ -8,7 +8,7 @@ The monitoring stage must be executed within a `slurm` environment.
 The information is recorded as `.csv` data within a directory  `jobreport_${SLURM_JOB_ID}` or a directory supplied on the command line.
 The second stage prints this information in a tabular form that can be inserted into a project proposal.
 
-## Downloading the job summary report
+## Downloading the job summary report tool
 
 Precompiled binary for the `jobreport` utility can be obtained directly from the [repository](https://github.com/eth-cscs/alps-jobreport/releases) or via the command line:
 

From bbd34917d8263f512039ff7ff5d865d9271caa43 Mon Sep 17 00:00:00 2001
From: boeschf <48126478+boeschf@users.noreply.github.com>
Date: Tue, 20 May 2025 13:10:11 +0200
Subject: [PATCH 3/9] Update docs/running/jobreport.md

Co-authored-by: Mikael Simberg <mikael.simberg@iki.fi>
---
 docs/running/jobreport.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/running/jobreport.md b/docs/running/jobreport.md
index 763866c1..d30c1dcd 100644
--- a/docs/running/jobreport.md
+++ b/docs/running/jobreport.md
@@ -10,7 +10,7 @@ The second stage prints this information in a tabular form that can be inserted
 
 ## Downloading the job summary report tool
 
-Precompiled binary for the `jobreport` utility can be obtained directly from the [repository](https://github.com/eth-cscs/alps-jobreport/releases) or via the command line:
+A precompiled binary for the `jobreport` utility can be obtained directly from the [repository](https://github.com/eth-cscs/alps-jobreport/releases) or via the command line:
 
 ```console
 $ wget https://github.com/eth-cscs/alps-jobreport/releases/download/v0.1/jobreport

From 6bf11b57a277e0501ccacc5f060c64552e174541 Mon Sep 17 00:00:00 2001
From: boeschf <48126478+boeschf@users.noreply.github.com>
Date: Tue, 20 May 2025 13:10:24 +0200
Subject: [PATCH 4/9] Update docs/running/jobreport.md

Co-authored-by: Mikael Simberg <mikael.simberg@iki.fi>
---
 docs/running/jobreport.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/running/jobreport.md b/docs/running/jobreport.md
index d30c1dcd..1894a2ae 100644
--- a/docs/running/jobreport.md
+++ b/docs/running/jobreport.md
@@ -46,7 +46,7 @@ Arguments:
   COMMAND                           The command to run as the workload
 ```
 
-## Reported Information
+## Reported information
 
 The final output from `jobreport` is a table summarizing the most important details of how your application used the compute resources during its execution.
 The report is divided into two parts: a general summary and GPU specific values.

From 28dd25e2d2307bc79f1988677a7421c2816aa811 Mon Sep 17 00:00:00 2001
From: boeschf <48126478+boeschf@users.noreply.github.com>
Date: Tue, 20 May 2025 13:10:40 +0200
Subject: [PATCH 5/9] Update docs/running/jobreport.md

Co-authored-by: Mikael Simberg <mikael.simberg@iki.fi>
---
 docs/running/jobreport.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/running/jobreport.md b/docs/running/jobreport.md
index 1894a2ae..d0a14817 100644
--- a/docs/running/jobreport.md
+++ b/docs/running/jobreport.md
@@ -134,7 +134,7 @@ GPU Specific Values
     If the application crashes or the job is killed by `slurm` prematurely, `jobreport` will not be able to write any output.
 
 !!! info "workaround known issue on macOS"
-    Currently, there is an issue when generating the report file via jobreport print on the macOS terminal:
+    Currently, there is an issue when generating the report file via `jobreport print` on the macOS terminal:
     
     ```console
     what(): locale::facet::_S_create_c_locale name not valid

From 7ad474e6ef25780b3edfd7046cb625c5ee2796cf Mon Sep 17 00:00:00 2001
From: boeschf <48126478+boeschf@users.noreply.github.com>
Date: Tue, 20 May 2025 13:10:55 +0200
Subject: [PATCH 6/9] Update docs/running/jobreport.md

Co-authored-by: Mikael Simberg <mikael.simberg@iki.fi>
---
 docs/running/jobreport.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/running/jobreport.md b/docs/running/jobreport.md
index d0a14817..98d2fcff 100644
--- a/docs/running/jobreport.md
+++ b/docs/running/jobreport.md
@@ -133,7 +133,7 @@ GPU Specific Values
     The `jobreport` tool requires the application to complete successfully.
     If the application crashes or the job is killed by `slurm` prematurely, `jobreport` will not be able to write any output.
 
-!!! info "workaround known issue on macOS"
+!!! warning "workaround known issue on macOS"
     Currently, there is an issue when generating the report file via `jobreport print` on the macOS terminal:
     
     ```console

From 3956e96b202a631252551e17e44d7a56db57b55e Mon Sep 17 00:00:00 2001
From: boeschf <48126478+boeschf@users.noreply.github.com>
Date: Tue, 20 May 2025 13:11:09 +0200
Subject: [PATCH 7/9] Update mkdocs.yml

Co-authored-by: Mikael Simberg <mikael.simberg@iki.fi>
---
 mkdocs.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mkdocs.yml b/mkdocs.yml
index 7d9596ed..ef3817c7 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -89,7 +89,7 @@ nav:
   - 'Running Jobs':
     - running/index.md
     - 'slurm': running/slurm.md
-    - 'jobreport': running/jobreport.md
+    - 'Job report': running/jobreport.md
   - 'Data Management and Storage':
     - storage/index.md
     - 'File Systems': storage/filesystems.md

From d138b9f9d2355ab9be188d8fedc6329d4579a1c4 Mon Sep 17 00:00:00 2001
From: boeschf <48126478+boeschf@users.noreply.github.com>
Date: Tue, 20 May 2025 13:13:49 +0200
Subject: [PATCH 8/9] typo

---
 docs/running/jobreport.md | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/docs/running/jobreport.md b/docs/running/jobreport.md
index 98d2fcff..1a550412 100644
--- a/docs/running/jobreport.md
+++ b/docs/running/jobreport.md
@@ -174,11 +174,11 @@ The job report generated contains sections for each `slurm` job step.
 #SBATCH -t 5:00
 #SBATCH --nodes=2
 
-srun ${JOBREPORT} -o report -- ${COMMAND}
-srun ${JOBREPORT} -o report -- ${COMMAND}
+srun ./jobreport -o report -- my_command_1
+srun ./jobreport -o report -- my_command_1
 
-srun --nodes=1 ./jobreport -o report -- my_command_1 &
-srun --nodes=1 ./jobreport -o report -- my_command_2 &
+srun --nodes=1 ./jobreport -o report -- my_command_3 &
+srun --nodes=1 ./jobreport -o report -- my_command_4 &
 
 wait
 ```

From 8fd5765d1ef8bc088f3295f2d0bc9f3d2e293e50 Mon Sep 17 00:00:00 2001
From: boeschf <48126478+boeschf@users.noreply.github.com>
Date: Tue, 20 May 2025 13:15:44 +0200
Subject: [PATCH 9/9] typo

---
 docs/running/jobreport.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/running/jobreport.md b/docs/running/jobreport.md
index 1a550412..ee8880f0 100644
--- a/docs/running/jobreport.md
+++ b/docs/running/jobreport.md
@@ -175,7 +175,7 @@ The job report generated contains sections for each `slurm` job step.
 #SBATCH --nodes=2
 
 srun ./jobreport -o report -- my_command_1
-srun ./jobreport -o report -- my_command_1
+srun ./jobreport -o report -- my_command_2
 
 srun --nodes=1 ./jobreport -o report -- my_command_3 &
 srun --nodes=1 ./jobreport -o report -- my_command_4 &