Add streaming to Remote Read endpoint

[gouthamve]

Remote read is quite inefficient right now and could be made much better through streaming. This has been discussed in the dev-summit: https://docs.google.com/document/d/1-C5PycocOZEVIPrmM1hn8fBelShqtqiAmFptoG4yK70/edit#heading=h.5d2mc4i0bnnp

[bwplotka]

I can take it if no one started working on this.

Design doc: https://docs.google.com/document/d/1JqrU3NjM9HoGLSTPYOvR217f5HBKBiJTqikEB9UiJL0/edit?usp=sharing

Dev-mailing topic: https://groups.google.com/forum/#!topic/prometheus-developers/79b6j_6F_Vc

[harshal-shah]

According to #4655 it seems this has been resolved in prometheus 2.5.0

[bwplotka]

Well, the memory was reduced (not sure how largely), but there still is a need for streamed remote read, potentially with chunk encoding to reduce used memory on query to minium.

[rshivane]

I would like to add that, as a vendor implementing the server-side of remote read API, the lack of streaming adds similar memory pressure on the marshaling side.

@bwplotka I notice that your already have a PR for streaming support in #4591. Whenever this is approved for merge, we would be happy to implement it on our storage and validate.

[bwplotka]

Yes. This is working code for streaming with raw samples, but we have some internal discussion how it should be done. I closed it to reduce confusion for now: #4591

TL;DR: It would be nice to have remote read stream with Prometheus chunk encoded samples (instead of row samples), to avoid unnecessary decoding & encoding. Does it
The plan is to:

1. Change the doc to propose this idea (right now it proposes something in between)
2. Setup a call to gather feedback on this idea.

Did not have much time to jump into it yet, but I am happy to hear that you are ok to help with this!

[rshivane]

It would be nice to have remote read stream with Prometheus chunk encoded samples (instead of row samples), to avoid unnecessary decoding & encoding.

Is that essentially "Alternative 1" in the design doc ?

[bwplotka]

Exactly, but need to reorder ;p