Troubleshooting

Troubleshooting

Run logger-rs --check inside a short interactive job first — it catches most of what follows in seconds rather than after a queue wait.

---

Could not execv logger-rs! ret=-1 errno=2

Symptom. One line per remote node in the job's stderr, and only one node's log file where there should be N. On an 8-node job you get 7 of these.

Cause. errno=2 is ENOENT. Remote loggers are started through pbs_tmrsh/pbsdsh/srun, all of which hand the command to execv(), and execv does not search PATH. The head node works because bash does search PATH, which is what makes this look like a partial failure rather than a missing binary.

Fix. Upgrade to v0.2.1 or later, which resolves an absolute path before launching. If you are already on that version, the binary exists but the other nodes cannot see it — see below.

The binary is not visible on the other nodes

An absolute path fails just as completely as a bare name if it points somewhere node-local, or somewhere the job did not request.

• Node-local storage. /tmp, /var/tmp, /dev/shm, $PBS_JOBFS are per-node. Install on /g/data or /scratch. The launcher warns about this at job start, and --check reports it.

• A storage= line that does not cover the install prefix. On Gadi a /g/data/<proj> path is only mounted inside the job if that project is in -l storage=. If the module lives in /g/data/gb02, you need:

  #PBS -l storage=gdata/ab12+gdata/gb02

  This bites people whose output directory is covered but whose module directory is not.

Only one node produced a log file

Beyond the two above: the launcher reports failed remote launches roughly five seconds into the job, naming the hosts. Look for

logger-rs: N of M remote logger(s) failed to start: <hosts>

If instead the job ran to completion and the merge said

WARNING: N allocated node(s) never reported: <hosts>

then those loggers started but died. The usual cause is the output directory not being writable from those nodes.

"Booked walltime 00:00:00" and "0.0% of booked"

PBS on Gadi exports none of PBS_RESOURCE_LIST_walltime, PBS_WALLTIME or PBS_RESOURCE_walltime, so there is no environment variable to read. Zero is not a real booking — it only ever meant "not determined".

logger-rs.sh asks qstat for it on the head node and passes it to every node's logger. If qstat is unavailable, or the job was started without the shell library, the booking stays unknown. Pass --booked-walltime explicitly if you need it.

No disk I/O, or absurdly large disk I/O

On Gadi the blkio controller is not delegated to the job — it resolves to the PBS daemon's own cgroup. Reading it reports the daemon's lifetime I/O across every job on the node.

The logger detects the mismatch, reports cgroup_io_* as absent, and falls back to per-process counters from /proc/{pid}/io. That fallback is a lower bound: a job that forks and reaps children faster than the sampling interval loses their I/O.

--check tells you which controllers this job actually owns. See Metrics.

No GPU data

• nvidia-smi is not on PATH — correct and expected on a CPU-only node. --check reports this as a note, not a failure.
• CUDA_VISIBLE_DEVICES="" — the logger never forks nvidia-smi at all.
• GPU figures are labelled node-wide (gpu_is_node_scoped: true) when CUDA_VISIBLE_DEVICES is unset, because we cannot tell which GPUs are yours.

Network numbers look too high

They are always node-wide. /proc/net/dev has no notion of job ownership. Exact on an exclusive node, inflated on a shared queue. See Metrics.

Memory looks wrong for an MPI job

Use cgroup_mem_bytes, not rss_bytes_sum. Summed RSS double-counts shared pages, which for MPI ranks on one node is badly wrong. The legacy field is kept only so archived logs still parse.

telemetry_start: command not found (exit 127)

source logger-rs.sh did not find the file. Either the module is not loaded, or your site disables bash's sourcepath option. The modulefile exports an absolute path for exactly this case:

source "${LOGGER_RS_SH:-logger-rs.sh}"

Under set -e this failure aborts the job, which is the one way the telemetry can take your job down with it — hence the belt-and-braces form above.

The job was recorded with an "unknown" outcome

Either the job was SIGKILLed — no trap can catch that — or the logger was started by hand rather than through telemetry_start. Started by hand it runs as a sibling of your workload and cannot see its exit status.

Ordinary walltime expiry and qdel send SIGTERM first and are handled.

The merged summary is missing

• The job is still running. The merge runs at telemetry_stop.
• --no-merge was passed.
• The job was SIGKILLed before telemetry_stop or the EXIT trap ran.

Per-node summaries are still on disk and can be merged after the fact:

logger-rs --merge /g/data/ab12/dashboard/$USER --job-id 12345.gadi-pbs

CPU efficiency looks terrible and the job seemed fine

Most often, data staging was measured. A twenty-minute copy from /g/data to jobfs inside the measured region reads as your job running at near-zero CPU efficiency and drags every figure down.

Move it outside telemetry_start/telemetry_stop — that is what marking a region is for.

Second most often: on Gadi NCPUS is per chunk, not for the whole allocation, so a job that booked more cores than it can use will show low efficiency honestly.