Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Prometheus to use ReadRequest_STREAMED_XOR_CHUNKS when calling remote read API. #5926

Open
bwplotka opened this issue Aug 19, 2019 · 5 comments · May be fixed by #6042

Comments

@bwplotka
Copy link
Member

commented Aug 19, 2019

#5703 extended remote read protocol and added ReadRequest_STREAMED_XOR_CHUNKS

Prometheus implements remote read API, but it can also be configured to use/call other remote read API. Enable streaming remote read if the server supports it.

AC:

  • Prometheus remote read client is capable to use ReadRequest_STREAMED_XOR_CHUNKS

Similar client code for inspiration: thanos-io/thanos#1268

@YaoZengzeng

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2019

I'd like to fix this issue 😃 @bwplotka

Maybe we can solve this problem by following steps:

  1. Extend the Read method of Client in storage/remote/client.go to support both SAMPLE and STREAM, the return type changed to *prompb.QueryResult to *http.Response would be better.

  2. Refactor FromQueryResult to decode both type of response and expose unified storage.SeriesSet.

  3. Encapsulate the chunks into some struct to make it implement storage.SeriesSet. Actually the implementation method is similar to chunkSeries stuff in tsdb/query.go. In addition, this step may easy to be implemented if #5882 being merged :)
    ( At first I‘d encapsulate the chunks to make the code run)

My initial idea is probably like this, any comments? @bwplotka

Test is also needed and now I'm trying to figure out how to do it properly.

@bwplotka

This comment has been minimized.

Copy link
Member Author

commented Aug 29, 2019

Sorry for lag and thanks @YaoZengzeng

Yes that makes sense to me 👍

I think step 2 would be the most work, especially packing SAMPLED method to SeriesSet, but I guess we can't really avoid this.

For Streaming be careful as we might need to implement something like this we did in Thanos in SeriesSet.

@YaoZengzeng

This comment has been minimized.

Copy link
Contributor

commented Aug 30, 2019

@bwplotka Thanks for your comments and I'll fix this issue step by step 😃

@YaoZengzeng

This comment has been minimized.

Copy link
Contributor

commented Sep 4, 2019

@bwplotka Please review #5974 again, otherwise the fixing of this issue can't move on 😂

@bwplotka

This comment has been minimized.

Copy link
Member Author

commented Sep 11, 2019

Sorry for massive lag, proposed solution (: Also reach me on IRC bwplotka or CNCF slack channel -> bwplotka if you need to ping me - bit spammed on email.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.