Skip to content

v0.7.9 — auto-recovery can no longer reload a config entry forever

Choose a tag to compare

@petergCA petergCA released this 07 Aug 17:35
· 9 commits to main since this release

A field bug, found because Sonos looked broken and a manual reload didn't fix it.

What happened

Two battery Sonos speakers were powered off, so their entities sat permanently
unavailable. Auto-recovery responded the only way it can for a non-Z-Wave
entity — by reloading the owning config entry — which restarts every speaker
that entry owns. It repeated every ~15 minutes, all night. Manual reloads
appeared not to work because Sentinel simply broke them again minutes later.

RECOVERY_MAX_ATTEMPTS = 3 was supposed to stop this after three tries. It could
never fire: reloading an entry removes its entities from the state machine,
Sentinel read that removal as a recovery, and a recovery refunds the attempt
budget. The reload reset the counter meant to stop it reloading again. As a
side effect it also fired entity_recovered and inflated
sensor.sentinel_recovered_today for entities that never came back.

This affected every config-entry reload recovery, not just Sonos. Only the
Z-Wave ping path — which removes nothing — had a cap that actually held.

The fix

Attempts are now budgeted properly: spent on failure, refunded the moment the
entity is seen healthy again.

  • Entity removal is no longer treated as recovery. The incident is dropped, but
    nothing is banked and the budget survives.
  • The budget is refunded when the entity next reports healthy, including when
    it is no longer tracked
    — which is exactly how it comes back after a reload.
    Without this, an integration that a reload genuinely fixes (GE Home) would get
    one successful reload and never another.
  • The budget scales to blast radius: 3 attempts normally, 1 when the entry
    still owns working entities.
    Deliberately not a veto — reloading a shared
    entry to fix one entity is legitimate, and blocking it outright would break
    the integrations this feature exists for. Doing it 25 times a night is not.
  • A declined attempt no longer burns budget.
  • Sentinel now logs when it gives up, and why, at info. "Gave up" and "never
    tried" were previously indistinguishable in the log.

Behaviour change worth knowing

If you rely on auto-recovery reloading a shared config entry, it now gets one
attempt instead of three before backing off until the entity recovers on its own.
A reload that works needs only one attempt, so integrations it actually fixes are
unaffected.

Auto-recovery remains opt-in and off by default. If you have it on and an
entity is offline because it is simply switched off — a battery speaker on its
charger, a seasonal device — an exclusion rule is a better fit than leaving it
for auto-recovery to discover.

Tests

New tests/test_recovery.py covers both directions, including the case that
must keep working. Against the pre-fix coordinator the runaway test reloads 10
times in 10 passes (unbounded, matching the field behaviour) and a removal
inflates recovered_today; after the fix, 3 and 0.

Full changelog: v0.7.8...v0.7.9