Troubleshooting

Troubleshooting

"Connection refused" / Test connection fails

• Check Host has no typo and no https:// prefix.

• Confirm you can connect from a plain terminal first:

  ssh -i KEYPATH USERNAME@HOST

  If that fails too it is a network or account issue outside the app — VPN required, account not active yet — not something the dashboard can fix.

• Off-campus, some clusters require a VPN. Connect that first.

Key permission errors

• The SSH key path field shows a red warning inline if no file exists at that path. Fix the path before touching permissions.

• If the path is right but SSH still rejects the key, its permissions are probably too open:

  chmod 600 ~/.ssh/id_gadi

• Make sure you pointed the field at the private key — no .pub suffix.

"The SSH key is passphrase-protected"

Expected for an encrypted key that is not already unlocked in your SSH agent. The app prompts you; type the passphrase and it connects. It is cached in memory for the session only, so reconnects will not re-prompt, and it is never written to disk.

Host key verification failed

The cluster presented a different host key from the one in ~/.ssh/known_hosts. The connection is refused deliberately. That means either the server was rebuilt or something is intercepting the connection — check with your system administrators before removing the old entry.

"No jobs found" for a profile that connects fine

This almost always means Log directories does not match where your logs actually live, or there are no psutil_*.log files there yet.

• Have you actually collected any telemetry? A job run without logger-rs produces no files. This is the most common cause by a wide margin.

• From a terminal on the cluster, confirm the path:

  ls -la /your/log/dir/psutil_*.log

• On a Local mount profile specifically, check the log directories are typed as paths under the local mount point, not remote-style HPC paths. See the log directory trap.

• For Local mount profiles, confirm the mount is actually attached. A stale or unmounted path can produce a missing-directory warning or an empty list.

A job shows no Summary, Advice or Compare entry

Those three read the merged whole-run summary, written once every node has stopped.

• A running job does not have one yet. The view picks it up on its own when the job ends.
• A job logged without the sourced logger-rs.sh wrapper — for instance by starting the binary by hand — has per-node logs but may have no merged summary. The wrapper's telemetry_stop (or its exit trap) runs the merge.
• If the job finished and the summary never appeared, check the job's PBS output for a merge error. The per-node summaries are still on disk and can be merged after the fact.

Only one node's worth of data on a multi-node job

Check the job's stderr for:

ERROR: Could not execv logger-rs! ret=-1 errno=2

That means the remote nodes could not find the logger binary. See logger-rs troubleshooting — the usual cause is the install path not being on storage the whole job requested.

"Booked walltime — not recorded"

Expected on Gadi, and not something you can fix in your script. PBS does not put the booked walltime in the job's environment, so unless qstat can be reached the logger has no way to learn it.

Everything else on the Summary tab is unaffected — only the "walltime used" percentage is unavailable.

It is shown as "not recorded" rather than 0.0% on purpose. A booking of zero is not a real value, and displaying it in the same shape as a measurement would state something false.

Disk I/O is missing or shows nothing

Normal on Gadi. PBS does not delegate the block-I/O counters to your job, so there is nothing job-scoped to read.

It is reported as absent rather than filled in with the numbers the counters do expose — those are the PBS daemon's lifetime totals for the whole node and have nothing to do with your job. logger-rs --check tells you this before you submit.

The Cores tab says there is no per-core breakdown

The cgroup v2 unified hierarchy does not expose per-CPU accounting at all — only a total. This view is available only for jobs recorded under cgroup v1, which is what Gadi uses today.

The dashboard seems to silently reconnect

Expected, not a bug. Before reusing a connection the app pings it, and transparently reconnects if the old one has gone stale — after your machine slept, or a VPN dropped. Nothing to do.

A download stops or fails

Use Retry download or Continue download in Jobs. Completed cache progress is kept, so you do not need to start every node log again. You can also choose View charts to stream a job directly when no download is running. If the app reports a changed remote log, reopen the charts so it can replay that file safely.

Charts look like they stopped updating

Check Last Sample on the Nodes tab. If it is climbing, the logger may have stopped writing, the job may have ended, or the connection to the filesystem may be interrupted. A stale-sample warning appears after 30 seconds on a job believed to be running. The app shows reconnection attempts separately.

The app will not open (macOS/Windows)

The builds are not yet signed by Apple or Microsoft. The macOS warning can also say the app is damaged. See Installing.

Update check failed

Usually offline, or a VPN not yet up. The check is silent on failure at launch and only reports when you ask for it explicitly. You can always download a release manually from the releases page.