Metrics

Metrics

Per sample (job_log)

Written every --interval seconds, default 0.5.

Metric	Source	Scope
CPU total and per-core %	cgroup cpu accounting	job
CPU efficiency vs booked cores	derived	job
Memory (current, kernel peak)	cgroup memory controller	job
Swap	cgroup memory controller	job
Block I/O bytes and ops	cgroup io/blkio controller	job
Process, thread, FD counts	/proc tree walk	job
Page faults (major/minor)	/proc tree walk	job
Summed RSS (legacy)	/proc tree walk	job
Network bytes/packets	/proc/net/dev	node
GPU util/memory/temp/power	nvidia-smi	job, or node — see below
Scheduler topology	PBS/Slurm environment	job

Summary (written on exit)

Peaks for every metric, CPU core-seconds and core-hours, walltime and memory efficiency against the booking, per-GPU peaks, network totals, and the job's exit status and reason.

---

Scope: which figures describe the job, and which the node

Every sample carries net_is_node_scoped and gpu_is_node_scoped so the dashboard never has to guess.

GPU

Readings are filtered to the job's own devices when CUDA_VISIBLE_DEVICES is set — which PBS and Slurm both do for GPU jobs — using either the index or the (possibly abbreviated) UUID form. gpu_is_node_scoped is then false, and gpu_indices carries the node index of each reading, so "GPU 1 was busy" stays distinguishable from "one of the node's GPUs was busy".

Without that variable set we cannot tell which GPUs are ours, so the readings stay node-wide and are labelled as such. A job with CUDA_VISIBLE_DEVICES="" never forks nvidia-smi at all.

Network

Always node-wide. /proc/net/dev has no notion of job ownership, and there is no cheap per-job alternative without the net_cls cgroup controller or eBPF, neither of which is available to an unprivileged job on Gadi.

Treat network figures as an upper bound: exact on an exclusive node, inflated on a shared queue.

---

Controllers that are not delegated to the job

A cgroup controller is only used if its path belongs to this job. Sites do not delegate every controller, and reading one that belongs to something else gives confidently wrong numbers.

Gadi is a live example. A job's own /proc/self/cgroup reads:

9:cpu,cpuacct:/pbspro.service/jobid/12345.gadi-pbs   <- the job
2:memory:     /pbspro.service/jobid/12345.gadi-pbs   <- the job
4:cpuset:     /pbspro.service/jobid/12345.gadi-pbs   <- the job
5:blkio:      /system.slice/pbs.service              <- the PBS daemon

blkio is not delegated. Reading it reported the PBS daemon's lifetime I/O across every job on the node — hundreds of GB for a job that wrote under a GB, and differing by 147 GB between two nodes running identical work.

The logger now notes the mismatch on stderr, reports cgroup_io_* as absent, and falls back to the per-process counters. So on Gadi today, I/O comes from /proc/{pid}/io. CPU, memory and cpuset are all properly job-scoped.

logger-rs --check reports which controllers this job actually owns.

The per-process I/O fallback is a lower bound

Because the counters are summed over live processes, a job that forks and reaps children faster than the sampling interval can lose I/O between samples.

On the validation job — 200 children each writing 4 MB, reaped in batches of 20 — it recorded 801 MB and 800 MB on the two nodes, matching the workload. That is not a guarantee, and finer-grained child churn will under-report.