-
Notifications
You must be signed in to change notification settings - Fork 183
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
optimize remote write settings #250
Conversation
It also makes the configuration simpler by having a 1:1 remote write block per source/target, so its easier to add things (and exclude them) |
regex: 'fluentbit.*' | ||
sourceLabels: [__name__] | ||
# kube state metrics | ||
- url: http://collection-sumologic.sumologic.svc.cluster.local:9888/prometheus.metrics.state |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will you need to change the plugin as tags are changed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't believe so, but need @samjsong to confirm. In my testing, all things worked like I expected, data volume diffs were accounted for. I do not believe we depend on on the tag in the plugins as we use a label to handle the datapoint stuff.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we do depend on the tag in the plugins, we no longer do the retag based on job to make it clearer to the user where the metrics were going. In this case the tags are correct since we check for <match prometheus.metrics.state**>
etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This PR simplfies the remote write configuration and also imporves memory and CPU performance by reducing the number of remote_write blocks, which are all seperate queues that read from the WAL. Testing shows roughly a 3x reduction in memory and CPU usage.