Collecting Telemetry

Collecting Telemetry

Do this first. The dashboard only reads files — something has to write them. That something is logger-rs, which runs alongside your job and records what it actually used.

Without it, a job produces no psutil_*.log and will never appear in the dashboard, however correct your profile is.

Add telemetry to your job script

Add the module lines, then put telemetry_start / telemetry_stop around the part you want measured:

#PBS -l storage=gdata/gb02+scratch/<your-project>

module use /g/data/gb02/modules
module load logger-rs
source logger-rs.sh

export HPC_DASHBOARD_DIR="/scratch/<your-project>/$USER/dashboard"

./stage-inputs.sh          # setup — not measured

telemetry_start
    mpirun -np "$PBS_NCPUS" ./my_model
telemetry_stop

./copy-results-back.sh     # clean-up — not measured

Replace <your-project> with your project code. The example module lives in /g/data/gb02/modules; ask your site administrator for its path if your site uses a different installation. A complete PBS script is in the logger-rs example.

Measure a region, not the whole script

Marking a region keeps data staging out of the numbers. A long copy from /g/data uses almost no CPU, so measuring it drags your efficiency down and makes the Advice tab recommend fewer cores than your science actually needs.

Anything can go between the two markers — pipelines, redirections, several launches, a command line assembled in variables.

telemetry_stop is optional

If the script ends or is terminated, the same shutdown runs automatically. Call it explicitly where you can: it records your workload's exit status directly, and keeps clean-up out of the measurement.

Nothing it does can fail your job

A missing binary, an unwritable output directory, a logger that dies on startup: each is reported and then ignored. Running unmeasured is far better than not running.

---

Check it will work before you queue

Start a short interactive job and run:

logger-rs --check

It reports what can and cannot be measured on that node — whether the job's cgroup was found, which counters PBS delegated to it, whether the output path is writable — in a few seconds instead of after a queue wait.

Most of what it looks for otherwise produces a log file that looks fine and is quietly wrong, which is much harder to notice than a crash.

The storage directive

Every filesystem your job touches must be listed in -l storage=, including wherever the telemetry is written, and wherever logger-rs itself is installed. The example above therefore includes both gdata/gb02 and your scratch project. Add any other project storage your workload uses.

Without it the filesystem simply is not mounted inside the job, and the failure looks like a permissions bug rather than a missing request.

For a multi-node job the output must also be on shared storage. Each node writes its own file and they are merged at the end, so /tmp or jobfs would leave the merge with only one node's worth.

Where to write the logs

Anywhere the job can write and the dashboard can read — typically somewhere under /scratch. The example above sets it once:

export HPC_DASHBOARD_DIR="/scratch/<your-project>/$USER/dashboard"

Whatever you choose, that is the path you give the dashboard as a log directory when you add a profile.

What gets written

psutil_<jobid>.manifest.json          written first, lists everything to expect
psutil_<jobid>_<host>.log             one per node, live NDJSON
psutil_<jobid>_<host>.log.summary.json
psutil_<jobid>.summary.json           the merged whole-job summary

The merged summary is what the Summary, Advice and Compare tabs read, and it appears once every node has stopped. A job you are watching live will not have one yet — the view picks it up on its own when the job ends.