Repository navigation
Replies: 13 comments 2 replies
|
Thanks @bilusss, this is a valuable complement to the existing aggregator. I would keep the verification kit separate as independent end-to-end evidence, and I will look at bringing the frozen- One important contract boundary: the stand-in’s 200 ms soft-stop and 2 s CPU-reset values should be labelled as experimental simulation settings. The agreed firmware bring-up policy is a 150 ms MCU-local heartbeat safe-stop, with CPU-reset supervision separate at roughly five minutes, pending physical cutoff measurements (makerspet/oomwoo-firmware#1). ROS-clock timing is useful for slow simulation, but it should not be presented as equivalent to the MCU’s own clock. I would put MCU cutoff/reset policy in the firmware or IO-board contract, while |
|
Quick follow-up: I opened health-monitor#2 with focused regressions for frozen/backward ROS time and work-path heartbeat stalls. It also withholds the MCU-facing heartbeat on a non-advancing clock and documents the simulator-versus-MCU clock boundary. The Python and ROS 2 Jazzy CI checks are green; physical cutoff timing remains untested. |
|
Thanks @xbattlax, that is clear and I've acted on it.
|
|
Soak results, first pass, with the 150 ms soft stop. Details, raw data and caveats are in the addendum at the top of FINDINGS.md. Steady state is clean. 18 cells (max_age 0.10 / 0.15 / 0.25 s, 20 / 50 Hz, idle / CPU / mixed load), a 10-minute window each: 0 false trips in all of them. Three hours in total, so this only bounds the rate below roughly one per hour. Restarts are not. In 720 kill/restart cycles there were 95 extra soft stops (13%), a median 0.34 s after the stand-in re-armed: motion is re-enabled, then stops again. They were spread evenly over load and over max_age, so neither explains them, and raising the deadline to 0.20 s or 0.30 s did not reduce them. Cause, as far as I could trace it. On Jazzy with Fast DDS 2.14.7 and its default SHM + UDP transports, each traced case (11 of 11) was a 333 ms sleep in the shared-memory port health check ( What it means here. With this middleware configuration, a heartbeat deadline under about 0.4 s is sensitive to nodes joining the graph. That is relevant to the 150 ms bring-up deadline in oomwoo-firmware#1: a restarted or late-starting node can produce a second stop right after re-arm. A longer deadline is not the fix, since the stall is longer than any deadline one would want. What I am not claiming. Not a Fast DDS bug. Rates varied from 6% to 28% between runs, so no rate is portable. Not all 95 soak stops were individually traced. Everything is one x86 PC in a container, Python nodes, no micro-ROS agent in the loop. My earlier statement of "55 kill/restart cycles, none unexplained" is superseded; I don't know why that pass saw none. Mitigation, preliminary. First attempts with a UDP-only profile and with One question: is anyone already setting a Fast DDS profile or discovery range for the robot image? I saw |
|
Transport comparison, as promised. Table, raw data and caveats are in the FINDINGS addendum ("Transport comparison"). Same setup as before (max_age 0.15 s, 20 Hz, 150 ms soft stop), four passes, 480 SIGKILL/restart cycles per profile across idle / CPU / mixed load, with each profile checked in effect through
One practical note for anyone repeating this: with Thanks for putting the draft values into SOFTWARE_INTERFACES.md. Two follow-ups from my side: I'll align the kit with the depth-1 MCU heartbeat QoS written there, and I'd suggest one sentence next to the 150 ms / "zero false trips" rows saying that the zero applies to steady state and that, with default Fast DDS transports, a node joining can stall a publisher for about 0.33 s. I can send that as a small PR if you want it. Still x86 in a container with Python nodes; the cost of UDP-only for large messages (scans, images) is not measured. |
|
@xbattlax the frozen-
Components in this test keep beating on wall time with frozen stamps, as a real node on a frozen sim clock would. 63 of 63 kit tests pass on both refs. |
|
Thanks @bilusss for the careful soak, transport comparison, and frozen-clock test. Yes, please send the small I aligned the MCU heartbeat publisher to depth 1 in health-monitor#2 ( I do not know which Fast DDS profile/discovery range the robot image will use. @makers-pet, could you confirm the intended runtime configuration so the next tests can match it? |
|
Rerun done with the measuring probe on each cell's profile too (every participant now uses the profile under test). Five passes, 200 SIGKILL/restart cycles per profile and load, all 45 cells valid, UDP-only cells with no SHM files at all:
@xbattlax thanks. The kit passes 63 of 63 against your |
|
A correction and an update, after recomputing every figure from the raw files. Correction. For the first soak pass I reported "95 extra stops in 720 restart cycles, spread evenly over load". The stand-in's cumulative counter gives 103 (14%); 95 is the number that have a detail record. And the load split is not even: idle 25, CPU 36, mixed 42. The larger runs since then say the same thing more clearly: with default transports the extra stops happen idle too, but about twice as often under load (6% idle, 13-15% under load). They still occur at every max_age from 0.10 to 0.25 s. FINDINGS is corrected and says what changed. Wide grid for the 60 ms health check (every participant on the profile, max_age 0.10 / 0.15 / 0.25 s, 20 and 50 Hz, idle / CPU / mixed, 54 cells, all valid):
Together with the earlier rerun: default 196 / 1800, 60 ms health check 0 / 1560, UDP only 0 / 600. Steady state remains at 0 false trips in every window. Zero in 1560 bounds the rate below roughly 0.2%; the limits are unchanged (one x86 PC, Python nodes, no large topics, no micro-ROS agent). The SOFTWARE_INTERFACES.md change is up as draft #86 with these counts. |
|
Final soak numbers and what I'm starting next. Soak, final. With every participant on the profile under test: default transports 286 extra stops in 2680 restart cycles (10.7%; about 5% idle, 13% under CPU load), Next: the health loop on the real sim stack. The board row says real nodes are not instrumented and task launches don't publish rosters, and everything I measured so far used fake components. I'm starting on that, in the kit, without touching upstream packages:
Things a first look at the running stack showed, which I'd like your view on before I bake them in:
Nav2 runs as one composed process in the sim, so fault injection there is per process, not per node. |
|
Results from running the health loop on the real sim stack (Gazebo + Nav2 + slam_toolbox, RTF 1.0, one PC, up to four parallel instances). Full tables, method and limits are in the M6 section of FINDINGS.md; code and raw data are in the repo. Setup. The unmodified aggregator ( Rates on the running stack. Supervising motion instead of following jobs. A job-following roster (idle / driving from Nav2 action status) stopped the robot twice per goal for 0.6-0.8 s, because the aggregator restarts arming when the roster changes, and it left motion commanded outside those actions unsupervised. I use a static roster instead, with one component, Navigation, no faults injected. 68 sim hours, about 21,300 goals, about 36.7 km. One stand-in soft stop in valid sessions: Faults in real processes (5 trials each, robot driving, from the signal):
Two things that need care when reading this:
Limits. Simulation only, five trials per fault, Nav2 as one composed process (process faults, not per-node), health nodes on wall time (no header-stamp checks), no AMCL, docking, cleaning jobs or behaviour-tree-triggered recoveries. A watcher proves that a node produces output, not that the output is right: the local-costmap watch stayed alive with The four questions from my previous post still stand, in particular whether per-component |
|
Results for the tasks the previous run did not cover: map-based navigation (AMCL), coverage jobs (both executors), sim docking ( Without faults (33 sessions, 16.5 sim hours)
With faults (SIGKILL and 5 s SIGSTOP, 5 trials each, nodes in separate processes)
What I take from the three "no" rows:
Limits: simulation, one x86 PC, small samples (zero stops in 2 h bounds the rate only to about 1.5 per hour), per-node faults use a non-composed launch, three sessions ran with the host overloaded and are flagged in the tables. The two Fast DDS profiles did not differ in this run. Built with AI assistance (Claude Code); numbers were recomputed from the raw files in a separate review pass before posting. |
|
Thank you, @bilusss - this is impressive work and the Fast DDS finding is valuable! I'm merging #86. On your questions: (1) yes, max_age is per component; I'll fix the draft row so 0.15 s applies to the drive path only. (2) Your static roster with drive_supervision is better than job-following rosters; the arming stalls at every goal show why. (3) A topic watcher is fine as the fallback for third-party nodes; our own nodes should send in-node work-path heartbeats. (4) On the robot the MCU bridge is the last hop for cmd_vel, so yes, pplease gate there. |
Uh oh!
There was an error while loading. Please reload this page.
Hi all. I'd like to work on the health-monitor RFC, but not as a second aggregator. @xbattlax's health-monitor already implements the aggregator and the contract. What the RFC's acceptance criteria still need is the other half of the loop and the evidence, so that is what I built:
Repo: https://github.com/bilusss/oomwoo-health-kit (Apache-2.0, ROS 2 Jazzy, runs in
makerspet/oomwoo:jazzy-dev)oomwoo/health/mcu_heartbeatand gatescmd_vel(unarmed / run / soft stop / CPU reset), so the whole loop is testable before firmware exists.b2b9796: critical node killed, process frozen, work loop stalled, node never started, advisory node killed, aggregator killed, no roster, frozen/clock.What I measured (details, method and limits in docs/FINDINGS.md):
cmd_velin every injected fault. No aggregator bug found.max_age_sec0.15 s under load; 576-684 ms with the 0.5 s default. Aggregator max_age and the MCU timeout add in series.Things the contract in
SOFTWARE_INTERFACES.mdleaves open, which I ran into: the payload ofmcu_heartbeat, the start value of the heartbeatsequence, QoS depth for the heartbeat topics, the aggregator publish period (50 ms, fixed in code), and the MCU soft-stop / reset timeouts.Limits: simulation only, one machine, small sample sizes, and the stand-in is Python, not firmware.
Questions:
mcu-io-firmwareas the behavioural reference for the watchdog)?This was built with substantial AI assistance (Claude Code). Every number comes from the raw files in
results/and can be reproduced with the commands in the findings. I'll follow up with a pointer PR undercontributions/health-monitor/bilusss/.All reactions