Skip to content

Protective Stop v1.2

Latest
Latest

Choose a tag to compare

View all tags
@iliabaranov iliabaranov released this 20 Aug 19:41
· 1 commit to main since this release
v1.2
7a72bb3
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Protective Stop v1.2

The stability-campaign release: every communication-loss failure class observed across 50+ instrumented soak runs is either fixed or precisely attributed, validated by an 8-hour zero-stop-event gate on all three bench remote classes (LAN, USB-NCM, off-site NAT).

Reliability fixes

  • Direct-path recovery (rx-silence ping-now): a safety peer silent >700 ms on its direct path gets an immediate disco ping + CallMeMaybe (~1 Hz while silent), replacing the 3 s cadence that previously turned every NAT disruption into a stop. Field-validated: a 3.1 s direct-only outage rode through with zero stops.
  • Endpoint-adoption wedge eliminated: safety-peer disco probes get a reserved slot region and a 15 s pong-accept window (late pongs behind a far-end relay reconnect previously expired unmatched, so direct paths never re-established after a NAT/WAN change); relay-bound safety links now retry direct at least every 30 s.
  • Relay-storm damping: DERP connect kicks are spaced 1.5 s apart with the safety-critical home slot and the pool-dark rescue path exempt; home reconnect causes are counted per trigger (tx-fail / rx-error / staleness reap / interface rebind / region rehome).
  • Peer-scaling hardening: over-capacity peer-add refusals are counted, rate-limited in the log, and no longer consume add-pacing slots; machine targets configured via the legacy key can no longer be dropped by a slots-table write.

Core-stall fixes (dual-core safety watchdog family)

  • Console output is budgeted (40 lines/s): measured log storms (147 lines/s) exceeded the UART's drain rate, making every log line a potential multi-millisecond blocking stall that starved the sibling safety core. The RTC crash ring still captures every line; rates and drops are exported. A/B-validated: the affected unit's lockstep-mismatch stops went from ~2/hour to zero.
  • Crash visibility: boot-time coredump summary (crash_task, crash_pc, build-identifying crash_sha) in state.json; admin-authenticated GET /api/coredump streams the raw image for host-side espcoredump decode; the panic capture path runs from IRAM so watchdog resets inside flash-cache-disabled windows are no longer traceless; controlled resets persist a one-shot cause crumb (ctrl_reset_cause, xcheck_last_detail).

ROS 2 machine node

  • Lifecycle bridge with two backends — software (hosts the certified pstop_c machine in-process on host UDP) and hardware (bridges the ESP32 machine) — validated by an 8 h gate with zero stop events across all three remotes.

Diagnostics & docs

  • Path-agnostic TX/RX worst-gap gauges, per-safety-peer silence-resume pairing, log-rate gauges, docs/API.md covering the full HTTP surface.

Safety scope unchanged: this is a protective stop (controlled preventative shutdown), not an emergency stop. The wire protocol and machine-side state logic (pstop_c) remain on their own certification track, unmodified.

Assets 2
Loading