Returns no data when the metric in right side of `or` operator does not exist on InfluxDB

[hirose31]

Bug Report

What did you do?

send following query

sum(irate(node_cpu_seconds_total{mode="system",instance=~'u183:9100'}[5m])) or sum(irate(node_cpu{mode="system",instance=~'u183:9100'}[5m]))

What did you expect to see?

return metrics reading from local storage and remote storage (InfluxDB).

What did you see instead? Under which circumstances?

return only metrics reading from local storage.

Environment

• System information:

  Linux 4.15.0-29-generic x86_64

• Prometheus version:

prometheus, version 2.8.0 (branch: HEAD, revision: 59369491cfdfe8dcb325723d6d28a837887a07b9)
  build user:       root@4c4d5c29b71f
  build date:       20190312-07:46:58
  go version:       go1.11.5

• InfluxDB version:

  1.7.4

Detail

• query using or operator like VECTOR1 or VECTOR2 to Prometheus.

• metrics of VECOTR1 exist in both Prometheus and InfluxDB

• metrics of VECOTR2 does not exist in both Prometheus and InfluxDB

• Prometheus returns metrics (VECTOR1) in Prometheus (within --storage.tsdb.retention.time)

• Prometheus query to InfluxDB metrics of VECTOR1 and VECTOR2

• InfluxDB returns metrics of VECTOR1

• InfluxDB returns 500 Internal Server Error as response of query VECTOR2

• Prometheus returns no data in InfluxDB (over --storage.tsdb.retention.time)

{"status":"success","data":{"resultType":"matrix","result":[]},"warnings":["server returned HTTP status 500 Internal Server Error","server returned HTTP status 500 Internal Server Error"]}

The following graphs:

• Prometheus has metrics 4/14 - 4/16
• InfluxDB has metrics 4/13
• But in the first graph metrics in 4/13 does not drawed

[brian-brazil]

I don't see how InfluxDB returning a 500 is an issue with Prometheus.

[hirose31]

Is this Prometheus's expected behavior?

Prometheus returns no data when the query right hand of VECTOR1 or VECTOR2 gets an error, even if the query left hand is successful.

[brian-brazil]

If a remote storage returns an error, we ignore all data from it. It'd be too hard to reason about otherwise.

[hirose31]

I understand. Thanks a lot for very quick response! 😀