how to handle aggregated sums from federated prometheus

[filippog]

hi,
I'm testing prometheus federation and aggregation, namely I have a "global" prometheus at https://prometheus.wmflabs.org pulling all metrics matching '^project:' from several per-project prometheus servers, e.g. https://graphite-prometheus.wmflabs.org/status. In this context a "project" is a simply collection of VMs, following the job: example in the federation documentation. Currently project: metrics are generated via rules on the federated servers e.g. project:node_network_receive_bytes{} = sum(node_network_receive_bytes{device!="lo"}) BY (device) and the idea being to pull only aggregated metrics from federated servers.

Normally this works but with some caveats e.g. rate(project:node_network_receive_bytes[5m]) on the global prometheus will artificially jump when a new target is added to the federated servers since that's now rate of sums instead of sum of rates. Jump on the global server at the same time a new target came up

What's the recommendation in this case? Having the 5m rate as a recording rule would achieve that I think? thanks!

[brian-brazil]

You should be aggregating up on the project prometheus servers:

project_device:node_network_receive_bytes:rate5m = sum by (project, device) (rate(node_network_receive_bytes{job="node"}[5m])))

Never take the rate of a sum, as it causes the problem you've encountered.

[filippog]

thanks @brian-brazil ! btw I'm happy to turn this issue into a documentation improvement, e.g. linking "best practices -> recording rules" from "federation", I'm fine to close it too

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.