rule: fix panic when calling API /api/v1/rules?type=alert #6189
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fpetkovski
previously approved these changes
Mar 6, 2023
saswatamcode
previously approved these changes
Mar 7, 2023
There was a problem hiding this comment.
Thanks for fixing! 🙂
In the rules proto spec, we have four stdtime fields, the last_evaluation of RuleGroup, RecordingRule, and Alert and this active_at in AlertInstance. The gogo/protobuf issue does specify that the field needs to be non-nullable, but no harm in trying this out.
pkg/rules/manager.go
Outdated
| @@ -97,12 +97,14 @@ func ActiveAlertsToProto(s storepb.PartialResponseStrategy, a *rules.AlertingRul | |||
| active := a.ActiveAlerts() | |||
| ret := make([]*rulespb.AlertInstance, len(active)) | |||
| for i, ruleAlert := range active { | |||
| // https://github.com/gogo/protobuf/issues/519 | |||
There was a problem hiding this comment.
Let's keep comments consistent with the other workarounds as well.
Suggested change
| // https://github.com/gogo/protobuf/issues/519 | |
| // UTC needed due to https://github.com/gogo/protobuf/issues/519. |
There was a problem hiding this comment.
Hello,
I updated the comment (+ another similar one found in the codebase)
Signed-off-by: Thibaut Ackermann <thibaut.ackermann@al-enterprise.com>
saswatamcode
approved these changes
Mar 7, 2023
junotx
pushed a commit
to junotx/thanos
that referenced
this pull request
Aug 3, 2023
…6189) Signed-off-by: Thibaut Ackermann <thibaut.ackermann@al-enterprise.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hello,
Sometimes, when I open the
/alertspage on Ruler web UI, I get an error. When it happens, the situation is irremediable and I have to restart the Ruler component completely.. As you can see in the stacktrace linked below, the error look like and old issue fixed in #2925 . After digging, I think this is linked to theAlertInstance.ActiveAtdate, which is not 'formatted' to UTC() before protobuf encoding like the ones in the old PR.I tried to produce a testcase to reproduce this, but unfortunately I failed to find the exact combination of events...
I think this has someting to do with the
reloadprocess of Ruler (either SIGHUP or POST /-/reload) which might be copying the alerts and losing/changing thetime.TimeLocations (which is the real problem with protobuf encoding)..Here is the full stacktrace from server:
Changes
Verification