Debugging and recovery

Debugging and recovery / 调试与恢复

The goal is to identify which layer failed before changing it. Preserve the evidence first; a reflash, process restart, or config rewrite can erase the most useful clue.

Identify the failing layer

Layer	Healthy evidence	Failure clues
Upstream data	New Kalshi snapshots and ESPN commentary reach the watcher	fetch warning, retry backoff, upstream event absent or late
Watcher	Startup identifies the event, mod, polling tier, and setup URLs	process exited, config error, missing ticker, delivery retry/drop
LAN	http://DEVICE-IP/api/status returns JSON from the watcher computer	timeout, wrong IP, mDNS-only failure, client isolation, firewall
Matchday mod	status has expected version/state and no relevant lastError	mod missing, TTS or celebration stays busy, command error
Host/runtime	UI, touch, Wi-Fi, and timers continue to run	debugger pause, boot loop, brownout/reset evidence in serial log

中文判断法：先看上游有没有事件，再看 watcher 有没有抓到，再看设备状态接口是否可达， 最后才进入 Mod/host 与硬件层。不要把“ESPN 还没发布”和“设备没收到命令”混成一个问题。

Preserve a useful evidence bundle

Collect this before restarting anything when possible:

• Local timestamp, timezone, match, score, and the exact observed symptom.
• Repository commit or release for the watcher, mod version from /api/status, and the host build lineage if known.
• Watcher output from startup through the incident. Include the polling-tier transition and nearby warnings, not only the last line.
• A saved /api/status response. The most useful fields include mod/version, network, mute, TTS busy/error state, celebration/light state, power state, setup state, and lastError.
• Whether xsbug, serial2xsbug, replay, an interactive client, or a second watcher was connected.
• For delayed commentary: the upstream-visible time and watcher-delivery time.
• For a real reboot: serial output across the boot boundary and whether USB power also disappeared.

Redact private LAN details if sharing publicly. Do not attach credentials, cookies, tokens, account exports, or an unreviewed configuration file.

Safe recovery sequence

1. Stop extra writers, not the evidence stream. Leave the main watcher terminal visible. Stop replay, old watchers, and interactive command loops.
2. Detach breakpoint debuggers. Disconnect xsbug/serial2xsbug; use a non-pausing trace logger if logs are required.
3. Test by numeric IP. Fetch /api/status from the watcher computer. This bypasses mDNS without changing device state.
4. If status is reachable, diagnose in place. Inspect lastError, mute, TTS, celebration, network, and setup state. Use the current Device API rather than guessed commands.
5. If status is unreachable, capture serial output and perform one controlled restart. Save both pre-restart and boot output. Recheck the numeric IP before changing configuration.
6. Repair only the confirmed layer. Update configuration for a wrong host or event; restart only the failed service; reinstall only a missing/corrupt mod; reflash the host only when the host/partition/font build is the actual mismatch.
7. Verify end to end. Confirm status, setup pending/ack, a harmless display update, watcher polling, and TTS separately before returning to live play.

Exact install and recovery commands are intentionally not duplicated here. Use Getting started (中文) and Host firmware preparation from the same revision as your code.

Common incident patterns

Freeze while a debugger is attached

xsbug can pause the whole JavaScript runtime at an exception breakpoint. A paused runtime stops Wi-Fi, touch, and timers, so first detach the debugger and retest. For unattended matches, collect traces with the headless xsbug-log workflow instead.

Mod installation stopped midway

An interrupted archive install can leave no runnable mod. Do not keep repeating the same stalled path while the device is busy. Follow the alternate recovery method documented for the current checkout in Getting started. That document, not this Wiki, owns partition addresses and build flags.

Setup request never receives an acknowledgement

Record the pending request and watcher log before clearing anything. Confirm one watcher is polling the device over HTTP, then look for apply-validation and device-sync errors. The relay accepts one pending request at a time so that an older request cannot silently overwrite a newer selection.

Repeated alerts or delivery pressure

Verify that only one watcher is running and replay is stopped. Capture delivery retry queued, delivery dropped, TTS failure, and feedback timeout lines. Delivery ordering, local-clip handling, and retry behavior are version-dependent; see the release notes before diagnosing an older or mixed watcher/mod deployment.

Delayed or missing live event

Record four times when available: real-world occurrence, first upstream ESPN or Kalshi appearance, watcher poll/log time, and device output time. This separates source latency, poll latency, queue latency, and device/TTS latency. Keep suspected market events labelled as unconfirmed until commentary confirms them.

After recovery

• Re-run the relevant tests from Development (中文).
• Add a regression test when the failure came from deterministic parsing, queueing, setup relay, or mod behavior.
• Put version-specific changes in repository docs and release notes. Add only the reusable diagnostic lesson to this Wiki.
• If opening an issue, attach the redacted evidence bundle and say which layer was proven healthy or unhealthy.

Security during debugging

The device, watcher setup service, and optional TTS endpoints are intended for a trusted LAN and may not require authentication. Do not port-forward them, publish a live device URL, or expose debug logs containing network or account data.