A standing fact repeated every fifteen minutes, and a page that said nothing - #272
Merged
Conversation
…ge that said nothing Two defects found by reading the chat the notifications actually land in, then reconstructing the volume from OpenObserve rather than from the transition count I used before. Those are different numbers: a transition is a state change, but a notification goes out every silence window for as long as the condition holds. Over 103 hours the fleet sent 352 messages from 116 episodes, 82 a day, where counting transitions alone had said 54. Half of it was two conditions that repetition cannot change. kernel_slab_unreclaimable held one alerting episode for 23.6 hours and announced it 95 times; host_standard_updates_available sent 82 for a fact that changed four times. Neither was going to stop until a member rebooted or the hosts were patched, and saying so every fifteen minutes does not make either happen sooner -- it teaches the reader to skim the channel that also carries the pages. The repeat cadence was two constants, ten minutes for a page and fifteen for a ticket, applied to everything. It is now a per-rule statement, because the cost of widening it is not uniform: OpenObserve pauses outcome evaluation while silenced, so recovery is observed that much later, which is cheap for a state whose recovery is a reboot you performed on purpose and expensive for a queue that drains by itself. Absent, the old constants still apply. Four human-cleared conditions declare four to twelve hours; the validator refuses a page quieter than an hour and anything quieter than a day. The test that guarded this asserted the two constants, so any deliberate cadence would have read as a regression while the flood read as correct. It now asserts what its name always claimed -- that stale recovery during silence is bounded -- and matches alerts by name, since RenderOpenObserve sorts its output and the positional comparison was right only by accident. Second, the observer. snapshot.Healthy has five inputs and the unhealthy warning printed three. On 2026-08-28 at 23:24 fleet_platform_unhealthy paged while its own log read collection_errors 0, orphan_instances 0, missing_instances 0 and an empty error list. Confirmed from the metric store: queue.uncovered_running was 1 at 23:25, and that field was not in the line. An inactive service would have been just as invisible. Both are logged now, and the failing service is named rather than counted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by reading the chat the notifications actually land in, then reconstructing the volume from OpenObserve — which is a different number than I reported this morning.
A transition is a state change; a notification goes out every silence window for as long as the condition holds. Counting transitions said 54 messages a day. Reconstructing notifications says 82:
The model was checked against the raw chat log:
kernel_slab_unreclaimableappears at :09:57, :24:57, :39:57, :54:57 — every fifteen minutes without a break, from a single alerting episode that began 2026-08-28 12:25:37 and is still open.Half the traffic is two facts repetition cannot change
Neither stops until a member reboots or the hosts are patched. Saying so 96 times a day does not make either happen sooner; it teaches the reader to skim the channel that also carries the pages.
The repeat cadence was two constants — 10 minutes for a page, 15 for a ticket — applied to everything. It is now a per-rule statement, because the cost of widening it is not uniform: OpenObserve pauses outcome evaluation while silenced, so recovery is observed that much later. That is cheap for a state whose recovery is a reboot you performed on purpose, and expensive for a queue that drains by itself.
kernel_slab_unreclaimablehost_standard_updates_availablehost_reboot_requiredhost_package_inventory_staleAbsent, the old constants still apply. The validator refuses a page quieter than an hour, and anything quieter than a day.
The test that guarded this asserted the two constants — so any deliberate cadence would have read as a regression while the flood read as correct. Same shape as the test that required
or vector(0)in #270. It now asserts what its name always claimed (stale recovery during silence is bounded), and matches alerts by name, sinceRenderOpenObservesorts its output and the positional comparison was right only by accident of the bundle already being alphabetical.A page whose own diagnostic said nothing was wrong
snapshot.Healthyhas five inputs. The unhealthy warning printed three.On 2026-08-28 at 23:24
fleet_platform_unhealthyfired while the observer logged:Confirmed from the metric store:
gha_fleet_queue_uncovered_runningwas 1 at 23:25 — a running job with no execution lease covering it — and that field was not in the line. An inactive service would have been just as invisible.Both are logged now, and the failing service is named rather than counted.
make verifygreen.https://claude.ai/code/session_01NpzpgiRaxi5mHVTMoRYndt