chore(pubsub): additional buckets to consume_latency prom metrics

[juanamari94]

Summary

We're increasing the amount of buckets in
gcloud_aio_pubsub_subscriber_consume_latency_seconds_bucket to enable better tracking of tail end latencies.

[TheKevJames]

Can probably trim off some of these smaller ones, that level of granularity is going to be dominated just in transit time anyway. Maybe just do something like:

buckets=(.01, .1, .25, .5, 1.0, 2.5, 5.0, 7.5, 10.0, 20.0, 30.0, float('inf'))

[juanamari94]

Done!

[shaundialpad]

nit: while you're here, float('inf') is unnecessary

[juanamari94]

Is it? I've seen it used in the prometheus libraries themselves as well

[shaundialpad]

Yeah we leave it out elsewhere and still receive values for those buckets. Interesting though, do you have a link? I only mention to avoid confusion in the places we've omitted including this, but it's fine if we want to keep it to be explicit here.

edit: for reference https://github.com/prometheus/client_python/blob/4535ce0f43097aa48e44a65747d82064f2aadaf5/prometheus_client/metrics.py#L618-L619

But I also see they explicitly mention inf here: https://github.com/prometheus/client_python/blob/4535ce0f43097aa48e44a65747d82064f2aadaf5/prometheus_client/metrics.py#L586

I conclude with no conclusion.

[juanamari94]

The second link is the one I saw

[TheKevJames]

I believe they always add an inf bucket at the end if it's missing, so including it ourselves is a noop/stylistic choice.

[cphoward]

The defaullt buckets seem to be:

from prometheus_client import Histogram

# Create a Histogram without specifying buckets to see the default ones
histogram = Histogram('request_duration_seconds', 'Description of histogram', unit='seconds')

# Access the default buckets
default_buckets = histogram._upper_bounds
default_buckets

Results:

[0.005,
 0.01,
 0.025,
 0.05,
 0.075,
 0.1,
 0.25,
 0.5,
 0.75,
 1.0,
 2.5,
 5.0,
 7.5,
 10.0,
 inf]

Given the 2 plus minutes we're being reported, we might even need to have bigger buckets, say, 60, 90 and 120.

[shaundialpad]

I'd be fine going up to 2 minutes as well. I'm not sure the 90 second bucket is necessarily useful. For our purposes, if we're something like 30 seconds behind then we're definitely way too far behind already, I think we just wanted to have a better chance at capturing more accurate upper bounds (or perhaps more accurately, non-infinite upper bounds!). Also good to keep in mind this applies to all of our apps, plus the apps of our users who choose to make use of the metrics.