Panel Diagnostics

Panel Diagnostics

Use Diagnostics to separate current health from historical crash evidence and from the support archives that you deliberately share. The native entry is /rust/diagnostics. The Diagnostics, Crash investigation, and Logs pages replace the three former classic tabs in the single native navigation. For troubleshooting beyond this page, see Troubleshooting and Panel Benchmarks and Evals.

Collect a debug kit and send it

1. Open Diagnostics and choose Saved debug kits and fleet collection, or open /rust/diagnostics/debugkit.
2. Review Upload settings (admin only). Enter an HTTPS upload URL (blank disables uploads) and Maximum stored bundles (1–100), optionally a Bearer token (leave blank to preserve existing token). Choose Save upload settings. A bearer token is never displayed back to the browser.
3. Choose Create diagnostic bundle. The collector stores a scrubbed archive, a SHA-256 digest, the creation time, size and missing-item descriptions. An archive can be incomplete; inspect the missing-item records before sharing.
4. Inspect Saved debug kits for the new bundle ID and SHA-256 checksum. Select its download link to retrieve a tar archive, or (ZIP) for a ZIP containing the archive and a checksum manifest.
5. Enter the exact Bundle ID under Upload saved bundle and choose Upload over HTTPS. The owner reports whether the transfer succeeded; an accepted request is not proof of successful delivery. The guided version at /rust/diagnostics/workflow/collect-send requires explicit confirmation before sending.
6. Check the Upload status in the saved-kit table. If no valid HTTPS destination is configured, the transfer is refused. A kit can be retained locally instead of uploaded.

Only administrators may list, create, configure, download, upload or collect debug kits. Archives are sensitive even after scrubbing. When retention prunes old kits, earlier archives can disappear. These actions do not alter instance launch parameters or restart the machine.

Fleet collection state

The debug kit page also shows Fleet collection requests and Enrolled fleet members. Enter Selected member IDs (comma separated) and Member HTTPS port, then choose Collect diagnostic bundles. Only enrolled, signed, ready members are suitable recipients; inspect the returned job states and any error details. The member collect and push routes are machine-to-machine authenticated transport; they are not manual browser forms. Failed or pending collection is not represented as a completed upload.

Investigate a crash: evidence, journal and logs

1. Open /rust/diagnostics/workflow/investigate-crash and choose Investigate recorded crash, or open Crash investigation at /rust/diagnostics/crash.
2. Read Crash analysis and Journal refresh state first. The journal snapshot is bounded to the latest 4,096 matching events. A stale or pending refresh must not be presented as a final classification.
3. Compare Correlated hard stops with Capture readiness. Netconsole, pstore, watchdog, lockup-panic and crash-dump evidence may be absent. Absence of evidence is not evidence of a power failure.
4. Select Engine and launch logs at /rust/diagnostics/logs. Choose Read size (bytes) (50,000; 200,000; 1,000,000; or 4,000,000). The browser reads a new bounded snapshot when the selection changes; administrators may also choose Read window for the no-script form fallback. Viewers receive a fixed 200 KB initial snapshot without scripting. When Follow log tail every 5 seconds is checked, the browser receives live bounded updates. Clearing the checkbox pauses updates; scrolling up holds your position. An unreadable refresh preserves previously displayed lines.
5. If needed, choose Re-analyse from the crash page. This explicitly requests a forced crash-report refresh. Recheck the journal state; the previous snapshot may remain stale while the new scan is prepared.
6. Compare the crash timing with prior parameter changes and the incident view at /rust/crash-evidence. Capture configuration commands are shown for review, never automatically executed.

See Panel Power and Hardware for current telemetry and Panel Storage, Backup and Updates for recovery and preservation of separate backups.

Local liveness and health

The /rust/diagnostics/health page distinguishes /api/health from /api/status and /api/diagnostics. The first is a small plain-text transport probe available only to appropriately connected local callers. It is not an engine readiness assessment and may refuse remote browser requests. The other two display actual instance status and diagnostic checks, including unavailable or degraded results. A healthy current instance does not explain an earlier hard stop. The Diagnostics page also shows individual check status, evidence, meaning, suggested action, and the live command line for the selected instance.

Use /rust/metrics to inspect recorded event searches and telemetry, /rust/qualification for qualification receipts and /rust/evidence for signed evidence. A missing or unverifiable receipt is not proof of qualification.

On-demand diagnostic evidence and configuration preservation

1. Open Diagnostics. Read Diagnostic checks and command line, then Boot self-test. A not-available self-test or missing diagnostic check is not a passing result.
2. As an administrator, choose Run USB checks to run the existing read-only live diagnostics owner. Inspect Live USB check result. Choose Open or save current live diagnostics JSON to save the evidence from the owner; this is a new read and may differ from the displayed result.
3. As an administrator, choose Show debug bundle JSON to inspect the selected instance's detailed report. Choose Copy debug bundle once it is displayed, or select the text manually if clipboard permissions block copying. Open raw debug bundle JSON is an alternative. Do not share command-line or environment details without reviewing them.
4. Choose Create config backup, acknowledge private storage, and inspect the verified result. Download encrypted backup archive retrieves the device-sealed file returned by the owner. The archive does not include model weights or engine builds.
5. Choose Open verified backups and portable configuration for the separate Export portable configuration (JSON), Preview portable restore, and confirmed Restore reviewed configuration controls. The portable flow requires the preview plan checksum and retains local secrets on the destination. Do not substitute a sealed local backup for the portable transfer.

Administrators see on-demand diagnostics, debug-bundle, and backup forms; viewers see no privileged form. Viewers can still open the read-only live diagnostics JSON. All reads and changes use the existing authenticated owner routes. On-demand live checks are not an installation, physical-machine, or release qualification.

Crash-history interpretation

Crash totals, Findings, Boot history, and Parameter changes are separate native sections. Boot history reports the boot end, duration, kernel, verdict, watchdog kills, OOM kills and GPU faults; parameter history shows saved context, KV, cache, batch and RAM settings. The recorded journal state and generated time are shown alongside the totals. A missing shutdown sequence supports an unclean-stop classification but cannot establish a physical cause. Inspect the expanded finding evidence, capture status, correlated stops, and journal refresh state before interpreting the timeline.

Control reference

Each help marker beside a native control has the exact text below. The registry retains mappings to the retired classic controls for parity checks.

ID	Control label	Tooltip	Native page	Routes	CLI
diagnostics.checks	Diagnostic checks	Read current instance checks; a passing check does not rule out an earlier crash.	/rust/diagnostics	GET /api/diagnostics	—
diagnostics.boot-test	Boot self-test	Inspect the last boot self-test result; missing evidence is not a pass.	/rust/diagnostics	GET /api/lexios/boot-self-test	—
diagnostics.live	Live system checks	Read live system diagnostics without changing host configuration.	/rust/diagnostics	GET /api/lexios/live-diagnostics	—
diagnostics.bundle	Instance debug bundle	Read a sensitive, instance-specific debug report; administrator access is required.	/rust/diagnostics	GET /api/debug-bundle	—
diagnostics.kit-list	Saved debug kits	List locally stored scrubbed debug kits and fleet collection state; administrators only.	/rust/diagnostics/debugkit	GET /api/debugkit	—
diagnostics.kit-create	Create diagnostic bundle	Collect a scrubbed, SHA-256-verified support archive on this machine; administrators only.	/rust/diagnostics/debugkit	POST /api/debugkit/create	lexi-debugkit
diagnostics.kit-download	Download debug kit	Download the stored archive or ZIP with its checksum manifest; administrators only.	/rust/diagnostics/debugkit	GET /api/debugkit/download, GET /api/debugkit/download.zip	—
diagnostics.kit-config	Upload settings	Set a credential-protected HTTPS destination and local retention count; administrators only.	/rust/diagnostics/debugkit	POST /api/debugkit/config	—
diagnostics.kit-upload	Upload saved bundle	Send an existing scrubbed bundle to the configured HTTPS destination; administrators only.	/rust/diagnostics/debugkit	POST /api/debugkit/upload	—
diagnostics.kit-fleet	Collect from fleet members	Request signed debug-kit collection from selected enrolled members; administrators only.	/rust/diagnostics/debugkit	POST /api/debugkit/fleet/request, POST /api/debugkit/fleet/collect, POST /api/debugkit/fleet/push	—
diagnostics.crash-report	Crash analysis	Inspect prior boot and crash evidence; a missing marker cannot prove a power failure.	/rust/diagnostics/crash	GET /api/crash-report	—
diagnostics.crash-refresh	Re-analyse	Request a refreshed journal analysis; prior evidence may remain stale until it completes.	/rust/diagnostics/crash	GET /api/crash-report	—
diagnostics.capture	Capture readiness	Read netconsole, pstore, watchdog, lockup and crash-dump readiness without executing commands.	/rust/diagnostics/crash	GET /api/crash-evidence/status	—
diagnostics.stops	Correlated stops	Compare recorded hard stops with captured evidence; absent capture leaves cause undetermined.	/rust/diagnostics/crash	GET /api/crash-evidence/stops	—
diagnostics.journal	Journal refresh state	See the journal refresh state and its bounded coverage before interpreting a crash report.	/rust/diagnostics/crash	GET /api/crash-evidence/journal-state	—
diagnostics.logs	Log window	Read bounded engine and launch logs; pause or follow the tail without losing your scroll position.	/rust/diagnostics/logs	GET /api/logs	—
diagnostics.log-filter	Filtered logs	Select a bounded log window to inspect; this does not delete or modify log files.	/rust/diagnostics/logs	GET /api/logs	—
diagnostics.observability	Events and metrics	Inspect recorded telemetry and filtered events; missing values mean unavailable, not zero.	/rust/diagnostics	GET /api/metrics/summary, GET /api/metrics/logs	—
diagnostics.qualification	Qualification evidence	Inspect qualification receipts and signed evidence without assuming unverified records are qualified.	/rust/diagnostics	GET /api/qualify, GET /api/evidence/status, GET /api/evidence/manifest, GET /api/evidence/export	—
diagnostics.bundle-copy	Copy diagnostic evidence	Show the authorized instance report for copying; its contents can include private environment details.	/rust/diagnostics	GET /api/debug-bundle	—
diagnostics.backup	Create encrypted configuration backup	Create a device-sealed configuration backup without model weights; keep the archive private.	/rust/diagnostics	POST /api/backup, GET /api/backup/download	—
diagnostics.portable	Portable configuration transfer	Open the reviewed export, preview and restore flow; secrets and model weights are not transferred.	/rust/diagnostics	GET /api/backup/config-transfer/download, POST /api/backup/config-transfer/preview, POST /api/backup/config-transfer/restore	—
diagnostics.live-run	Run live USB diagnostics	Run read-only host checks on demand; results do not qualify an installation or release.	/rust/diagnostics	GET /api/lexios/live-diagnostics	—
diagnostics.live-export	Save live USB evidence	Open the live diagnostics JSON for saving locally; it may contain private machine information.	/rust/diagnostics	GET /api/lexios/live-diagnostics	—
diagnostics.log-live	Follow log tail	Refresh the bounded log window every five seconds while enabled; scrolling back preserves your position.	/rust/diagnostics/logs	GET /api/logs	—
diagnostics.crash-timeline	Boot and parameter history	Compare boot verdicts, recorded failures and parameter saves without treating missing capture as proof.	/rust/diagnostics/crash	GET /api/crash-report	—
diagnostics.health	Local liveness	Check the local-only transport liveness route; it is not an engine readiness verdict.	/rust/diagnostics/health	GET /api/health, GET /api/status	—