optimize remote write settings #250
Conversation
|
It also makes the configuration simpler by having a 1:1 remote write block per source/target, so its easier to add things (and exclude them) |
| regex: 'fluentbit.*' | ||
| sourceLabels: [__name__] | ||
| # kube state metrics | ||
| - url: http://collection-sumologic.sumologic.svc.cluster.local:9888/prometheus.metrics.state |
There was a problem hiding this comment.
Will you need to change the plugin as tags are changed?
There was a problem hiding this comment.
Don't believe so, but need @samjsong to confirm. In my testing, all things worked like I expected, data volume diffs were accounted for. I do not believe we depend on on the tag in the plugins as we use a label to handle the datapoint stuff.
There was a problem hiding this comment.
we do depend on the tag in the plugins, we no longer do the retag based on job to make it clearer to the user where the metrics were going. In this case the tags are correct since we check for <match prometheus.metrics.state**> etc.
This PR simplfies the remote write configuration and also imporves memory and CPU performance by reducing the number of remote_write blocks, which are all seperate queues that read from the WAL. Testing shows roughly a 3x reduction in memory and CPU usage.