fix: stop the flatline check crying wolf, and clear it on recovery - #158
Conversation
Two CodeRabbit findings on #149 that landed after it merged. Both defeat the restraint the feature was built around. A broken collector read as a flatline. get_flatlined_services queried all historical rows, and a collector that fails simply stops writing new ones - its earlier readings are still there and, naturally, unchanged. A removed service that kept its history behaved the same way. Both are different faults with different fixes, and reporting them as "running but not earning" is exactly the crying wolf that makes a report worth ignoring. The query now requires a reading from today, which a failing collector does not produce. A recovered service stayed in cooldown forever. record_alert suppresses a repeat within its quiet window and nothing cleared the stored flatline alert when a balance started moving again, so a service that recovered and then went flat again inside that window was silently swallowed - the alert nobody gets. _flatline_check now clears the cooldown for every subject that is no longer flat. Regression tests cover both: a collector with seven unchanged days but nothing today, a removed service with only old history, and that the collection hook clears exactly the recovered subject and leaves the still-flat one alone.
|
Warning Review limit reached
Next review available in: 31 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #158 +/- ##
==========================================
+ Coverage 94.23% 94.24% +0.01%
==========================================
Files 36 36
Lines 4475 4486 +11
==========================================
+ Hits 4217 4228 +11
Misses 258 258
🚀 New features to boost your workflow:
|
Two CodeRabbit findings on #149 that landed after it merged. I'd missed them; both defeat the restraint the feature was built around.
1. A broken collector read as a flatline
get_flatlined_servicesqueried all historical rows. A collector that fails stops writing new ones — its earlier readings are still there and, naturally, unchanged. A removed service that kept its history behaved identically.Those are different faults with different fixes, and reporting them as "running but not earning" is exactly the crying wolf that makes a report worth ignoring.
The query now requires a reading from today, which a failing collector does not produce.
2. A recovered service stayed in cooldown forever
record_alertsuppresses a repeat within its quiet window, and nothing cleared the storedflatlinealert when a balance started moving again. So a service that recovered and then went flat again inside that window was silently swallowed — the alert nobody gets._flatline_checknow clears the cooldown for every subject that is no longer flat.Regression tests
Verification
ruff check . && ruff format --check .— cleanpytest --cov=app --cov-fail-under=90— 1459 passed, coverage 94.25%