-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vmalert: recording rules produce unexpected spikes #4768
Comments
vmagent
recording rules
@mbrancato do you have non-default settings for vmalert? Have you read https://docs.victoriametrics.com/vmalert.html#troubleshooting ? Could you please elaborate on how data gets into the VictoriaMetrics? How exactly |
@hagen1778 We're using the VM operator to deploy and configure everything. I do want to clarify, I've switched to the Again, if I run the |
@mbrancato thanks for details! Do you use one or multiple vmagents? |
@mbrancato I've been running a test on dev environment to calculate recording rules based on aggregates generated by vmagent. I've used the following config for rules:
For the streaming aggregation I used the following config:
I ran the test for more than 7d and I see no discrepancy: Screen.Recording.2023-08-24.at.09.47.29.movCould you help to identify what in your config could be causing the issue you mentioned? |
@hagen1778 sorry I haven't had much time to get back to this, but today I did. I made two changes: I doubled the evaluation interval to 1 minute, and I added a lookback parameter. This was following guidance at: https://docs.victoriametrics.com/vmalert.html#data-delay. But unfortunately, it did not seem to help. evaluationInterval: "1m"
extraArgs:
# attempt to solve blank spaces in recorded rule graphs due to delayed data
datasource.lookback: "3m" my recording rules look like this: groups:
- name: my-rules
interval: 1m
rules:
- record: process_time:1m_quantiles
expr: |
histogram_quantile(0.5, sum(rate(process_time_bucket:30s_without_instance_pod_total[1m])) by (le))
labels:
quantile: "0.50" However, recorded data still has gaps. I did find sometimes there source data has a gap, but not normally, and it doesnt' correspond with the recording rule gaps (green below). I inverted the recording rule to make it easier to see... |
I just realized the gaps were not really what this issue was about, but since the lookback change, I've seen fewer spike, but they still happen. |
@mbrancato could it be that vmagent is struggling to deliver the mentioned metric |
@hagen1778 There are no restarts at that time, and kubernetes shows no container restarts for those pods. The persistent queue graph is empty - I see its filtered to only values I do now think this this is mainly related to #4966. At least a lot of recording rule spikes are correlated with streaming aggregation spikes. |
Describe the bug
I am seeing frequent outliers in the form of delta functions for recording rules by
vmagent
.I'm using recording rules for histogram quantiles. The rules looks like this:
Any while I first attributed this to starts / restarts of
vmagent
, over time I identified these delta functions appear at random across different recorded metrics.If I run the expression manually, the spike is not there.
I'd rather the data not be there, than be wrong and mess up the scale.
To Reproduce
create a
histogram_quantile
recording ruleVersion
1.90.0
Logs
No response
Screenshots
No response
Used command-line flags
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: