[WIP] Prometheus remote read support

[DanCech]

This PR adds a new endpoint (currently /render/prometheus but could move to /prometheus/read or similar) that implements the Prometheus remote read API, allowing Prometheus to query data out of Graphite.

It accepts remote read requests (as snappy-compressed protobuf objects), converts them into native seriesByTag requests and executes the queries, then encodes and returns the results to Prometheus.

In order to use this you need to pip install python-snappy protobuf (for that to work you'll likely need to sudo apt install libsnappy-dev first), I didn't add them to requirements.txt due to the non-trivial installation for python-snappy.

This is still a work in progress, I intend to add tests and docs, and I'm working on a matching patch for carbon to add native remote write support. Right now the simplest way to get Prometheus metrics into Graphite is to use prometheus/prometheus#3533 but you can also use this PR to read regular Graphite data from within Prometheus.

[deniszh]

Cool, nice feature!

[iksaif]

Note that we plan to add tags support to https://github.com/criteo/graphite-remote-adapter very soon, at least for writes. That would work nicely with this.

Our graphite-remote-adapter should hopefully be more up to date than the one in prometheus/documentation/example. I'll try to write more doc for it and advertise it a little bit more.

[DanCech]

Once I add native support in carbon we won't need an adapter at all for the standard use-case, though looking at graphite-remote-adapter it seems like it would be useful for cases where more control over the final series names etc are needed.

[iksaif]

Exactly, also when you have multiple Prometheus but a single Graphite cluster you want to be able to control the prefix (as a way to namespace the metrics and not pollute every prometheus instance)

[deniszh]

More is better in this case. 👍

[codecov-io]

Codecov Report

Merging #2195 into master will decrease coverage by 3.56%.
The diff coverage is 7.95%.

@@            Coverage Diff             @@
##           master    #2195      +/-   ##
==========================================
- Coverage   79.63%   76.07%   -3.57%     
==========================================
  Files          85       90       +5     
  Lines        8862     9315     +453     
  Branches     1899     1916      +17     
==========================================
+ Hits         7057     7086      +29     
- Misses       1542     1965     +423     
- Partials      263      264       +1

Impacted Files	Coverage Δ
webapp/graphite/urls.py	80% <ø> (ø)	⬆️
webapp/graphite/prometheus/gogo_pb2.py	0% <0%> (ø)
webapp/graphite/prometheus/types_pb2.py	0% <0%> (ø)
webapp/graphite/prometheus/urls.py	100% <100%> (ø)
webapp/graphite/prometheus/views.py	24% <24%> (ø)
webapp/graphite/prometheus/remote_pb2.py	5.26% <5.26%> (ø)
webapp/graphite/render/views.py	71.02% <77.77%> (+0.13%)	⬆️
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d483c17...658a6df. Read the comment docs.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.