turtlebot3: stop the debounce profile from hiding the goal-status faults - #74
turtlebot3: stop the debounce profile from hiding the goal-status faults#74bburda wants to merge 2 commits into
Conversation
The debounce profile sets confirmation_threshold -3, which suits the no-progress check: it repeats every five seconds while the robot is stuck, so three reports arrive and the profile filters roughly fifteen seconds of it. That is the contrast with the storm profile. The other two faults do not repeat. NAVIGATION_GOAL_ABORTED and NAVIGATION_GOAL_CANCELED are raised from a status change, guarded so they fire once, and cleared by one PASSED when a later goal succeeds. Under -3 the counter reaches -1 and stops, so both stayed PREFAILED for good and the default CONFIRMED-only fault list never showed them. Measured against a running fault_manager: one FAILED leaves PREFAILED at -1, silence changes nothing, and the PASSED only moves the counter to 0. They now report under their own source_id and take their thresholds from a per-source file, -1 to confirm on the single event and 0 to heal on the single clear. The same measurement with that file in place gives CONFIRMED on the raise and HEALED on the recovery, while the no-progress fault still needs its three reports. healing_threshold was 3 with a comment saying three PASSED events heal. Healing costs healing_threshold minus the counter at recovery, so from -3 it would have taken six, and the detector sends one. It is 0. The storm profile set confirmation_threshold 0, which the fault manager rejects and replaces with -1 while logging a warning. It now says -1.
Per-topic snapshots are written on the fault manager capture thread pool. They land a bit after the fault reports CONFIRMED, and CONFIRMED is the status the previous step waits for. The check read the fault detail once at that moment, so it sometimes saw an empty snapshot list while the capture was still running. In one run the check ran 124 ms before the fault manager logged "Captured 3/3 snapshots" for the same fault code. The rosbag check below already polls for the same reason. This makes the snapshot check poll too, with a 30s budget.
| # prefix, so the entry below applies only to the goal-status reporter and the | ||
| # global thresholds still cover the no-progress one. | ||
|
|
||
| /anomaly_detector/goal_status: |
There was a problem hiding this comment.
This key never matches. launch/demo.launch.py:208-209 starts the detector as name="anomaly_detector", namespace="bridge", so get_fully_qualified_name() is /bridge/anomaly_detector and anomaly_detector.py:299 reports source_id = "/bridge/anomaly_detector/goal_status". The resolver is an anchored prefix compare (entity_threshold_resolver.cpp:37-39), so resolve() falls back to the global -3 and the goal-status faults stay PREFAILED exactly as before; the replay table must have used the bare key. /bridge/anomaly_detector/goal_status: fixes it; worth a comment tying the key to the launch namespace.
| # Healing costs healing_threshold minus the counter at recovery, so from -3 | ||
| # a value of 3 would need six PASSED events. The detector sends one per | ||
| # recovery, so anything above 0 never heals. | ||
| healing_threshold: 0 |
There was a problem hiding this comment.
The comment reads as if 0 makes the no-progress fault heal on its one PASSED. It does not: a CONFIRMED no-progress fault sits at counter -3 (clamped), one PASSED takes it to -2, and HEALED needs counter >= 0, so three PASSED (compute_debounce_status). The detector sends exactly one and active_faults.discard() (anomaly_detector.py:305) stops further ones, so a confirmed no-progress fault still never heals; only one caught at PREFAILED -1 does. Either keep sending PASSED (throttled) while progress continues until healed, or say here that 0 is the minimum cost and the confirmed fault still needs three.
| # so they are written slightly AFTER the fault reports CONFIRMED - the status | ||
| # the previous step polled for. Checking once races that write and reads an | ||
| # empty snapshot list from a fault whose capture is still in flight. | ||
| if poll_until "/${NAV_ENTITY}/faults/${NAV_CODE}" \ |
There was a problem hiding this comment.
This is the only test change in the PR and it is the OTA demo's snapshot race (NAV_ENTITY=apps/bt-navigator, ACTION_NAVIGATE_TO_POSE_ABORTED), nothing to do with the title. Nothing in tests/ or .github/ starts docker-compose.debounce.yml or asserts a goal-status fault confirming and healing, which is how the key above slipped through. One smoke run of the debounce profile with one FAILED and one PASSED goal transition, asserting CONFIRMED then HEALED, would pin the behaviour this PR is about.
2 participants
Summary
Closes #73.
The debounce profile's
confirmation_threshold: -3suits the no-progress check, which repeatsevery five seconds while the robot is stuck. It does not suit the other two faults:
NAVIGATION_GOAL_ABORTEDandNAVIGATION_GOAL_CANCELEDcome from a goal status change, fire once,and are cleared by one PASSED on a later success. Under -3 the counter reached -1 and stopped, so
both stayed PREFAILED and the default CONFIRMED-only fault list never showed them.
They now report under their own
source_idand take their thresholds from a per-source file: -1 toconfirm on the single event, 0 to heal on the single clear. The no-progress fault keeps the global
-3, so the storm-versus-filtered contrast is unchanged.
Two smaller fixes on the same files.
healing_thresholdwas 3 under a comment saying three PASSEDevents heal; the cost is
healing_thresholdminus the counter at recovery, so from -3 it was sixand the detector sends one. And the storm profile set
confirmation_threshold: 0, which the faultmanager rejects and replaces with -1.
Testing
The demo needs Gazebo and a TurtleBot, so this was measured against a running
fault_managerwiththe profiles' own values, replaying the event pattern each reporter produces.
Before, global -3 only:
After, with the per-source file:
The last two lines are the check that the filtering the demo exists to show still works.
Also in this branch: a flaky check in the OTA demo smoke test
Not related to the TurtleBot profile, but it fails on this branch and on main, so it is fixed
here.
ota-demo-narrativefails on and off since 25 August, always on the same check:fault detail has >=1 environment_data snapshot. Per-topic snapshots are captured on the faultmanager capture thread pool, so they are written a bit after the fault reports CONFIRMED, and
CONFIRMED is the status the step before it waits for. The check read the fault detail once at
that moment. In the failing run on this branch it ran 124 ms before the fault manager logged
Captured 3/3 snapshotsfor the same fault code, so it read an empty list from a capture thatwas still running.
The rosbag check right below it already polls, and for the same reason. The snapshot check now
polls too, for up to 30s.
Checked against a stub API that serves the fault detail, to make sure the new form still fails
when the snapshots are genuinely missing:
The middle line is the one that matters: polling did not turn the check into one that always
passes.