Using a query that returns "no data points found" in an expression

[rich-youngkin]

I've created an expression that is intended to display percent-success for a given metric.

( sum(rate(rio_dashorigin_serve_manifest_duration_millis_count{Success="Succeeded"}[5m]) )  / 
( sum(rate(rio_dashorigin_serve_manifest_duration_millis_count{Success="Succeeded"}[5m])) +   
  sum(rate(rio_dashorigin_serve_manifest_duration_millis_count{Success="Failed"}[5m])) ) 
) * 100

This works fine when there are data points for all queries in the expression. However when one of the expressions returns no data points found the result of the entire expression is no data points found. In my case there haven't been any failures so rio_dashorigin_serve_manifest_duration_millis_count{Success="Failed"} returns no data points found. Is there a way to write the query so that a default value can be used if there are no data points - e.g., 0.

Thanks,
Rich

[brian-brazil]

It's recommended not to expose data in this way, partially for this reason. Separate metrics for total and failure will work as expected.

[rich-youngkin]

Good to know, thanks for the quick response!

[rich-youngkin]

Finally getting back to this. I made the changes per the recommendation (as I understood it) and defined separate success and fail metrics. A simple request for the count (e.g., rio_dashorigin_memsql_request_fail_duration_millis_count) returns no datapoints). So I still can't use that metric in calculations ( e.g., success / (success + fail) ) as those calculations will return no datapoints. So it seems like I'm back to square one.

Here's the full expression:

( sum(rate(rio_dashorigin_memsql_request_success_duration_millis_count[5m]))  / 
  ( sum(rate(rio_dashorigin_memsql_request_success_duration_millis_count[5m])) + 
    sum(rate(rio_dashorigin_memsql_request_fail_duration_millis_count[5m])) ) 
) * 100

Perhaps I misunderstood, but it looks like any defined metrics that hasn't yet recorded any values can be used in a larger expression. I.e., there's no way to coerce no datapoints to 0 (zero)?

[juliusv]

The idea is that if done as @brian-brazil mentioned, there would always be a fail and success metric, because they are not distinguished by a label, but always are exposed. Are you not exposing the fail metric when there hasn't been a failure yet? Or do you have some other label on it, so that the metric still only gets exposed when you record the first failued request it?

[rich-youngkin]

I'm not sure what you mean by exposing a metric. I am always registering the metric as defined (in the Go client library) by prometheus.MustRegister().

To your second question regarding whether I have some other label on it, the answer is yes I do. So perhaps the behavior I'm running into applies to any metric with a label, whereas a metric without any labels would behave as @brian-brazil indicated?

[juliusv]

@rich-youngkin Yes, the general problem is non-existent series. When you add dimensionality (via labels to a metric), you either have to pre-initialize all the possible label combinations, which is not always possible, or live with missing metrics (then your PromQL computations become more cumbersome). This is one argument for not overusing labels, but often it cannot be avoided. One thing you could do though to ensure at least the existence of failure series for the same series which have had successes, you could just reference the failure metric in the same code path without actually incrementing it, like so:

successes.WithLabelValues("labelvalue").Inc()
failures.WithLabelValues("labelvalue")

That way, the counter for that label value will get created and initialized to 0.

[rich-youngkin]

@juliusv Thanks for clarifying that. This makes a bit more sense with your explanation. Is what you did above (failures.WithLabelValues...) an example of "exposing"?

[juliusv]

@rich-youngkin Yeah, what I originally meant with "exposing" a metric is whether it appears in your /metrics endpoint at all (for a given set of labels). The thing with a metric vector (a metric which has dimensions) is that only the series for it actually get exposed on /metrics which have been explicitly initialized. This is in contrast to a metric without any dimensions, which always gets exposed as exactly one present series and is initialized to 0.

[rich-youngkin]

Nice, thanks again for clarifying.

[rich-youngkin]

Hmmm, upon further reflection, I'm wondering if this will throw the metrics off. For example, I'm using the metric to record durations for quantile reporting. Will this approach record 0 durations on every success?

successes.WithLabelValues("labelvalue").Inc()
failures.WithLabelValues("labelvalue")

If so it seems like this will skew the results of the query (e.g., quantiles). Is that correct? If so I'll need to figure out a way to pre-initialize the metric which may be difficult since the label values may not be known a priori.

[juliusv]

No, only calling Observe() on a Summary or Histogram metric will add any observations (and only calling Inc() on a counter metric will increment it). So just calling WithLabelValues() should make a metric appear, but only at its initial value (0 for normal counters and histogram bucket counters, NaN for summary quantiles).

[rich-youngkin]

perfect, thanks again.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.