Architecture

Architecture

src/
├── main.rs        # Entry point, signal handling, merge dispatch
├── lib.rs         # Library root (so integration tests share the code)
├── cli.rs         # Argument parsing, PBS env resolution, validation
├── check.rs       # Preflight (--check)
├── host.rs        # Node identity
├── manifest.rs    # Job manifest: what files to expect, and where
├── merge.rs       # Combining per-node summaries into a job total
├── types.rs       # Wire types: TelemetrySample, TelemetrySummary
├── cgroup.rs      # cgroup v1/v2 discovery and metric reads
├── cpu.rs         # CPU percentage and efficiency calculations
├── process.rs     # Process tree traversal
├── output.rs      # NDJSON writer, atomic summary write
├── logger.rs      # Main sampling loop
├── gpu.rs         # GPU metrics (rate-limited nvidia-smi)
├── network.rs     # Network statistics
└── scheduler.rs   # PBS/Slurm topology from the environment

tests/
└── schema_contract.rs   # Pins the NDJSON schema

logger-rs.sh       # The sourced shell library: launches, supervises, merges

---

cgroup discovery

Neither /proc/self/cgroup nor /proc/{tree_pid}/cgroup is reliable alone, so both are considered and ranked:

1. A candidate resolving to the root cgroup is rejected — root means the whole node, never a job.
2. A path naming the job wins. PBS writes /pbs_jobs.service/jobid/12345... and Slurm /slurm/uid_1000/job_12345.
3. Otherwise the more specific (deeper) path wins.

--cgroup DIR bypasses all of this if your site's layout is unusual.

Then, for whichever candidate won: prefer cgroup v1 cpuacct when present else the v2 unified hierarchy, resolve controller mounts from /proc/mounts (falling back to the conventional /sys/fs/cgroup/* layout), and walk up to the nearest level where the controller is actually populated.

Which processes count as the job's

By default the logger asks the kernel, reading cgroup.procs from the job's cgroup and any child cgroups. This is the authoritative membership list, and it beats walking the process tree in four ways:

• it works where the job's processes are not descendants of anything the logger started — which is every node but the head one;
• it catches processes that reparent to init;
• it cannot pick up a neighbour's processes on a shared node;
• it is cheaper.

The tree walk from --tree-pid remains as a fallback for when no usable cgroup is found. --procs-from cgroup|tree|auto forces the choice; the summary records which was used in proc_source.

This is why remote loggers are launched with no --tree-pid. An earlier version passed --tree-pid 1, which was wrong twice over: /proc/1/cgroup is the root cgroup, so CPU and memory described the whole node rather than the job, and a tree walk from init counted every process on the machine.

CPU calculation

Measurement windows are contiguous: each tick's ending counter read becomes the next tick's starting read, so CPU consumed between ticks is not lost.

Percentages use the measured elapsed time, not the configured interval — the logger records 0.511 where --interval said 0.5 — and samples report it as dt_sec. Anything deriving a rate must divide by that, per node, before summing.

Per-CPU vectors are indexed by node CPU id and sized to the node's CPU count, not to the job's booked cores.

Signal handling

Handles SIGINT, SIGTERM and SIGHUP. This matters: PBS sends SIGTERM on walltime exhaustion and qdel, and a logger that ignores it dies without writing job_end or the summary. A second signal exits immediately.

SIGKILL cannot be caught, so a hard-killed job records an unknown outcome.

Failure behaviour

Sampling errors back off and abort after 20 consecutive failures — the usual cause is the job's cgroup being torn down, which means the job is over.

The output buffer is flushed at least every 2 seconds, bounding what a SIGKILL can lose. Readers should therefore ignore a final line with no trailing newline.

Output file permissions

Files are explicitly chmoded to 0644 after creation rather than left to inherit the umask. Remote nodes launched via pbs_tmrsh got a stricter default than the head node, so one job produced -rw------- on some nodes and -rw-r--r-- on others — which reads as a corrupt job to anyone else trying to open the directory.