Multi Node Jobs

Multi-Node Jobs

telemetry_start launches a logger on every node in the allocation. telemetry_stop merges their summaries into one job-level file.

Each node writes its own NDJSON and its own summary; nothing is sent between nodes while the job runs.

File layout

/g/data/ab12/dashboard/sam/
  psutil_12345.gadi-pbs.manifest.json                 <- written first, lists everything
  psutil_12345.gadi-pbs_gadi-cpu-clx-0123.log         <- per node
  psutil_12345.gadi-pbs_gadi-cpu-clx-0123.log.summary.json
  psutil_12345.gadi-pbs_gadi-cpu-clx-0124.log
  psutil_12345.gadi-pbs_gadi-cpu-clx-0124.log.summary.json
  psutil_12345.gadi-pbs.summary.json                  <- merged, "schema": "merged-v1"

The merged file is identified by "event": "job_summary" and "schema": "merged-v1", so a reader can tell it from a per-node summary without parsing filenames. Every per-node record also carries hostname.

To measure only the head node, pass --single-node.

How the loggers are launched

The output path may contain {host}, {jobid} and {user}, expanded on the node that does the writing — so one template is broadcast to every node and each fills in its own name.

Remote loggers are started with pbs_tmrsh (preferred), pbsdsh, or srun (Slurm). pbs_tmrsh targets a host by name, whereas pbsdsh -n N indexes PBS's vnode list, which does not necessarily line up one-to-one with unique hostnames; getting that mapping wrong would put two loggers on one node and none on another.

All three hand the command to execv(), which does not search PATH, so the launcher resolves an absolute path to the binary before launching. See Troubleshooting for what happens when that path is not visible on the other nodes.

They shut down by watching for the exit-status file the head node writes to gdata, because neither launcher reliably forwards SIGTERM to the task it started. If no launcher is available the wrapper says so and measures the head node only.

What the merge can and cannot combine

Totals — core-seconds, I/O bytes, network bytes, page faults — are exact sums, additive no matter when each node was busy.

Peaks are not. Two nodes each peaking at 90% may have done so seconds apart, so the job never used 180% at once. Rather than pick a number and pretend, the merged summary reports both bounds:

Field	Meaning
cpu_pct_sum_peak_max_node	A value some node definitely reached
cpu_pct_sum_peak_sum_of_nodes	An upper bound the job cannot have exceeded

The true simultaneous peak lies between them; recovering it exactly would need the per-node sample streams aligned on a common clock.

cpu_efficiency_pct_avg is exact — it is a ratio of two sums, not a peak.

Missing nodes

nodes_missing lists allocated nodes that produced no summary, and num_nodes_reporting says how many did.

If a logger was killed, or never started, its node's usage is absent from the totals — the merged file says so rather than silently reporting a fraction of the job as the whole. The merge prints a warning naming the missing nodes, and the launcher reports failed remote launches a few seconds into the job rather than waiting until the end.

Why not one combined file

The NDJSON is a live stream. A combined file could only be written after the job ends — which is precisely when live tracking stops being useful — so multi-node jobs would show nothing at all until they finished.

Having every node append to one shared file instead is worse: concurrent O_APPEND across nodes serialises on Lustre lock contention, and at ~1.1 KB per line there is no atomicity guarantee, so lines can interleave and tear.

Combining also would not save the reader the actual work. Samples from different nodes arrive at slightly different timestamps and each node's cpu_pct_sum is its own, so producing a job-level series means bucketing by time and summing across hosts either way. One file would save a glob(), not the aggregation. See Dashboard Integration.