Remote read should handle down or misbehaving backends gracefully

[brian-brazil]

I'm playing a bit with remote read, and it looks like any failure on the part of the remote read endpoint (e.g. down, malformed response, wrong number of result series) causes the query as a whole to fail.

When any remote read backend fails, the query should continue to work on the rest of the data available.

We may also in future want to report this out to the user via the query api, metrics etc.

FYI @tomwilkie as you mentioned this being relevant for Cortex.

[tomwilkie]

Yes we have the same problem in Cortex, in that we want to return partial results in the even of partial failures (for instance, if S3 is down we still have recent data in the ingesters). Cortex issue: cortexproject/cortex#309

There doesn't seem to be a particularly appropriate HTTP status code for this - some references imply 207 (http://www.restpatterns.org/HTTP_Status_Codes/207_-_Multi-Status) is appropriate, others that it should be 500 with a body. WDYT?

[brian-brazil]

I'd have a standard 200, and the extra information in an warning field in the response.

[krasi-georgiev]

DIBS

[krasi-georgiev]

@brian-brazil just to make sure I understand the issue here is what I did
set to write and read from influxdb , than I stop the influxdb server and when running any query I am getting

{"status":"error","errorType":"internal","error":"error sending request: Post http://localhost:9201/read: dial tcp [::1]:9201: getsockopt: connection refused"}

this shouldn't happen as the local tsdb should be able to provide some samples if available and the the http header should be:
HTTP/1.1 206 Partial Content

[brian-brazil]

The http response code should be 200. A 206 is something different.

[gouthamve]

This should also handle broken checksums in TSDB.

[brian-brazil]

That's a bit of a different problem, but may share the warning plumbing.

[sipian]

@brian-brazil @krasi-georgiev

Hi,
I read about this on GSOC'18 ideas page and would like to contribute to it as part of GSOC.

After going through all references, I came to the following conclusions. Please do correct me if my interpretation is wrong.

1. From this, @brian-brazil proposed to handle this in fanout storage, using the primary and secondary storage .

After going through the code, I came to the following conclusions about it's implementation

1. In fanout Querier, if subsequent Querier request are to remote storage, then
   • If primary Querier fails , report ERROR
   • If primary Querier succeeds but secondary fails, this is the special case.
     In this special case, return an error with the data collected so far & notify user that such a case has occurred. (This case needs to be discussed further)
   • If both Querier succed, its a success

Is this interpretation of the issue correct ?
And if yes, is the proposed implemetation a way to solve this issue ?

[krasi-georgiev]

@brian-brazil, @tomwilkie isn't this solved by #3963

[brian-brazil]

@sipian Your understanding it correct.

@krasi-georgiev No, that's a bug in PromQL where it wasn't propagating errors correctly.

[pmb311]

This issue is a major issue for us where we're trying to scale out while preserving user experience...it would be amazing if we could get it in a tagged release soon.

[giusyfruity]

In order for us to move Prometheus into production, it is critical that this issue be fixed first!

[foosinn]

Seems like it is being worked on #4832

[brian-brazil]

Fixed by #4832