Gaps with `histogram_quantile` if no count change

[Dominik-K]

What did you do?
Our service gets only sometimes some requests via gRPC. To check the request counts, I run this query:
histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket[5m])) by (le))

What did you expect to see?
A solid line at 0 were there are gaps now.

What did you see instead? Under which circumstances?
Gaps between the request. There are no 0 values pointing with the mouse over it.

I checked that the service was scraped and the counter grpc_server_handling_seconds_bucket is in the metrics (manually "scraped"). The service was up the whole time.

I got the gaps at the same timeframes by doing a division by zero:

Environment

• System information:
  Official latest prom/prometheus Docker image.

• Prometheus version:
  prometheus, version 1.6.1 (branch: master, revision: 4666df5)

[juliusv]

When all rate buckets are 0, histogram_quantile() returns NaN (can you check in the returned AJAX data that this is the case?). The graph will not show 0 values for NaN, but should just show a gap, as you mention. This makes sense to me, as there were not any requests at all in the considered time window, so no quantiles can be calculated.

[Dominik-K]

@juliusv You're right. It actually makes sense to have a gap if there are no requests. We just were confused as we are trained/familiar that a gap in Prometheus/Grafana meant a downtime (until we've got that little requests). But for downtime there's the up metric.

How can I check the Ajax return data?

[juliusv]

How can I check the Ajax return data?

In Chrome: open Chrome inspector -> "Network" tab -> (refresh a graph) -> click on one of the query_range requests -> "Response"

[juliusv]

Cool, closing here then.

[Dominik-K]

They are NaN. Thanks for your answer.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.