fix(healthcheck): give broadcast-orphan-recovery a staleness window - #36
Merged
Merged
Conversation
The first real healthcheck run on prod (2026-07-29, right after installing the
systemd timer) reported:
! beat:broadcast-orphan-recovery no staleness window configured
which is exactly the ride-along case the DEFAULT_STALENESS_HOURS comment warns
about: a task seeded into the schedule but never added to the dict, so its
staleness can never fail. The task is seeded by broadcast/0009 and runs
`0 */6 * * *`, so the window is 6h + 1h grace = 7, matching the +1h pattern the
daily entries already use.
Because the dict doubles as the must-exist set in _check_periodic_tasks, adding
the key also makes a missing schedule entry a FAIL. That is safe here: the task
is created by a migration, so any migrated database has it.
Tests cover the window's boundaries in both directions — one late interval is
still OK, two missed intervals FAIL — so a future retune can't silently widen it
into uselessness.
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by running the healthcheck on prod for the first time, immediately after installing the systemd timer (suite 35, step 5).
The finding
broadcast-orphan-recoveryis seeded into the beat schedule bybroadcast/0009_seed_orphan_recovery_beat.pybut was never added toDEFAULT_STALENESS_HOURS— so its staleness could never fail. That is precisely the ride-along case the dict's own comment warns about:Given the outage this suite exists to fix was a scheduled task quietly not running, a scheduled task that structurally cannot report staleness is worth closing.
The change
Schedule is
0 */6 * * *, so the window is 6h + 1h grace = 7, matching the+1hpattern the daily entries already use (24 → 25).Note:
DEFAULT_STALENESS_HOURSdoubles as the must-exist set in_check_periodic_tasks, so adding the key also makes a missing schedule entry a FAIL. That's safe — the task is created by a migration, so any migrated database has it. I verified this rather than assuming it.Tests
Boundaries in both directions, so a future retune can't silently widen the window into uselessness:
OKFAILNot addressed here
The
weekly-digest-sundayFAIL above is real and separate: beat was dead through Sunday 2026-07-26, so that week's digest was never sent. It is not a healthcheck bug — the check is correctly reporting missed work. Next run is Sunday 2026-08-02, which will clear it.🤖 Generated with Claude Code