fix(observability): replace an alert whose stream moved instead of editing it - #313
Merged
Merged
Conversation
…iting it OpenObserve accepts a PUT that changes an alert's stream_name and keeps the stream it already had. Everything else in the document lands. So a rule whose metric moves plans an update that can never converge: the expression and the threshold change, the stream does not, and every subsequent reconcile reports the identical drift. Observed live moving lifecycle_queued_delivery_stall and queue_wait_slow_burn off the state clock. Apply returned "OpenObserve read-back did not converge", replanning produced the same two updates, and reading the alert back showed the new expression and threshold 1800 sitting on gha_fleet_queue_intent_oldest_state_age_seconds. The alerts delivered to the owner still named the old stream while evaluating the new query. A moved stream is now planned as delete plus create. The fleet was converged by hand this way before the fix, and the reconcile then reported managed with zero actions. The test fake was wrong in the same direction as the code: it accepted the whole document on PUT, so it could not reproduce the failure it was meant to cover. It now keeps the previous stream_name the way the real store does. Verified by mutation -- with the replacement branch disabled the plan falls back to a single update and the test fails.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OpenObserve accepts a PUT that changes an alert's
stream_nameand keeps the stream it already had. Everything else in the document lands.So a rule whose metric moves plans an
updatethat can never converge: expression and threshold change, the stream does not, and every subsequent reconcile reports the identical drift.Observed live
Moving
lifecycle_queued_delivery_stallandqueue_wait_slow_burnoff the state clock:OpenObserve read-back did not convergeupdateactionsgha_fleet_queue_intent_oldest_state_age_secondsThe alerts still delivered to the owner named the old stream while evaluating the new query — which is the worst version of this, because the message contradicts the rule.
The change
A moved stream is planned as
delete+create. The fleet was converged by hand that way before this fix, after which the reconcile reportedstate: managed, actions: 0.The fake was wrong in the same direction as the code
The test double accepted the whole document on PUT, so it could not reproduce the failure it existed to cover — a green suite over a store that behaves differently from the real one. It now keeps the previous
stream_name, as OpenObserve does.Verified by mutation: with the replacement branch disabled the plan falls back to one
updateandTestMovingAStreamReplacesTheAlertRatherThanEditingItfails.