Quick Start

Quick Start

Install

Nothing to install. The tool is published as a module in a shared project directory, so a colleague needs only two module lines:

module use /g/data/gb02/modules
module load logger-rs

If your job reads or writes anything under /g/data/gb02, that project also has to be in your storage= line — see Troubleshooting.

Measure part of your job

Put two lines around the part of the script you want measured:

#!/bin/bash
#PBS -l ncpus=96,mem=384GB,walltime=02:00:00
#PBS -l storage=gdata/ab12+gdata/gb02

module use /g/data/gb02/modules
module load logger-rs
source logger-rs.sh

./stage-inputs.sh                            # setup — not measured

telemetry_start
    mpirun -np "$PBS_NCPUS" ./my_model       # measured
    ./postprocess                            # measured
telemetry_stop

./copy-results-back.sh                       # clean-up — not measured

Your script keeps its shape. Anything can go between telemetry_start and telemetry_stop — pipelines, redirections, several launches, a command line assembled in variables. None of those survive being prefixed with a launcher, which is why that approach was dropped.

Marking a region rather than the whole script also keeps data staging out of the numbers. A twenty-minute copy from /g/data to jobfs, measured, looks like your job running at nearly zero CPU efficiency.

A complete, commented job script is in examples/gadi-job.pbs — copy it and change four lines.

Where the output goes

The output path defaults to $HPC_DASHBOARD_DIR, or the working directory if that is unset, so --output is only needed when you want somewhere specific:

export HPC_DASHBOARD_DIR="/g/data/ab12/dashboard/$USER"

That directory must be on a filesystem the job requested with -l storage=.

Check it will work before you queue

logger-rs --check

Run this inside a short interactive job. It reports the cgroup version and scope, which controllers the job actually owns, whether the output path is writable, where the binary lives, and what will therefore be missing — in seconds, rather than after a queue wait.

Everything it looks for otherwise produces a plausible-looking log file that is quietly wrong, which is much harder to notice than a crash.

Options to telemetry_start

Option	Default	Notes
--output PATH	$HPC_DASHBOARD_DIR/psutil_{jobid}_{host}.log	Supports {host}, {jobid}, {user}, expanded on each node. For multi-node jobs _{host} is inserted automatically if absent, so nodes cannot overwrite each other.
--interval SECS	0.5	Sampling interval.
--gpu-interval SECS	5.0	GPU polling interval. Each poll forks nvidia-smi.
--telemetry-bin PATH	found on $PATH	The logger-rs binary.
--single-node	off	Measure only this node, even on a multi-node job.
--no-merge	off	Skip the job-level merge at the end.

---

How source logger-rs.sh finds the file

source with a bare filename searches $PATH (bash's sourcepath option, on by default), and PATH takes precedence over the working directory, so a stray local file of the same name cannot shadow it. The modulefile puts the install directory on PATH, so nothing else is needed.

logger-rs.sh therefore has to sit in the same bin/ directory as the logger-rs binary — the one the modulefile prepends. Not in /g/data/gb02/modules: that is MODULEPATH, which holds modulefiles, and module use never adds it to PATH. See examples/modulefile.

If your site disables sourcepath the bare form fails, telemetry_start is never defined, and calling it returns 127 — which under set -e would abort the job over its own instrumentation. The modulefile also exports LOGGER_RS_SH, so a script that wants to be certain can write:

source "${LOGGER_RS_SH:-logger-rs.sh}"

telemetry_stop is optional

If the script ends, exits early under set -e, or PBS terminates it at the walltime limit, an EXIT trap performs the same shutdown.

Call it explicitly where you can: it records your workload's exit status directly, and it keeps clean-up out of the measurement.

The one case neither covers is SIGKILL, which no trap can catch — a job hard-killed records an unknown outcome. Ordinary walltime expiry and qdel send SIGTERM first and are handled.

What this does that a backgrounded binary cannot

• The logger runs as a sibling of your workload and cannot see its exit status. Started by hand, every job is recorded with an unknown outcome.
• PBS starts your job script on one node only, so on a multi-node job every other node would go unmeasured. telemetry_start launches a logger on every node in the allocation and telemetry_stop merges their summaries into one job-level file. See Multi-Node Jobs.

Nothing here can fail your job

A missing binary, an unwritable output directory, a logger that dies on startup: each is reported and then ignored, and telemetry_start always returns success. Under set -e — which most production scripts use — anything else would abort the job over its instrumentation.