store-gateway: add postings shortcutting for LabelValues

[dimitarvdimitrov]

What this PR does

In a related fashion to what #4698 did for Series this PR adds some heuristics to reduce the resources used in LabelValues calls. The general flow is as follows

1. Read all label values and their offsets from the index header. (unchanged)
2. If there are no matchers, return all the label values. (unchanged)
3. Create a new index reader which has a specially initiated postings strategy - labelValuesPostingsStrategy. This strategy makes a decision between three options
   • A) fetch complete expanded postings + series
   • B) fetch complete expanded postings + postings for each possible label value (as is the current behaviour)
   • C) fetch partial expanded postings + series
4. Call ExpandedPostings and get a list of postings + list of pending matchers
5. If there are pending matchers or if the size of postings from step 1. is larger than the series for the postings from step 4., then fetch series, apply filters, record unique label values seen.
6. Otherwise, fall back to the current implementation of intersecting posting lists.

The strategy uses the configured regular strategy for Series calls to decide between A) and C). Deciding between (A or C) and B happens in a similar fashion to the decision in worstCaseFetchedDataStrategy: assuming 1 byte of series == 1 byte of postings.

Benchmarks

I expected that decision C will be the most significant one in making a difference, but it seems that most of the optimizations happen because of choosing decision A over B. In fact none of the existing benchmarks showed an improvement when I eliminated decision A as an option.

BenchmarkBucketStoreLabelValues with options to choose from A, B or C

goos: darwin
goarch: arm64
pkg: github.com/grafana/mimir/pkg/storegateway
                                                                                                                    │ before-ec33b862e34d7ad2cd61396d86813746b514cfa5.txt │ after-c5ce4d73147075e2a413b140db9f47d5fa0231e1.txt │
                                                                                                                    │                       sec/op                        │           sec/op             vs base               │
BucketStoreLabelValues/no_cache/10-series-matched-with-10-label-values-10                                                                                   7011.3µ ± ∞ ¹                  839.0µ ± ∞ ¹  -88.03% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1000-series-matched-with-1000-label-values-10                                                                                26.69m ± ∞ ¹                  16.24m ± ∞ ¹  -39.14% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1_000_000-series-matched-with-10-label-values-10                                                                             28.77m ± ∞ ¹                  29.47m ± ∞ ¹   +2.43% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1_000_000-series-matched-with-1-label-values-10                                                                             13.208m ± ∞ ¹                  7.790m ± ∞ ¹  -41.02% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/10-series-matched-with-10-label-values-10                                                1144.6µ ± ∞ ¹                  101.9µ ± ∞ ¹  -91.09% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1000-series-matched-with-1000-label-values-10                                           24612.8µ ± ∞ ¹                  983.1µ ± ∞ ¹  -96.01% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1_000_000-series-matched-with-10-label-values-10                                          28.66m ± ∞ ¹                  28.95m ± ∞ ¹        ~ (p=0.114 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1_000_000-series-matched-with-1-label-values-10                                          603.96µ ± ∞ ¹                  79.74µ ± ∞ ¹  -86.80% (p=0.029 n=4)
geomean                                                                                                                                                      8.760m                        2.278m        -74.00%
¹ need >= 6 samples for confidence interval at level 0.95

                                                                                                                    │ before-ec33b862e34d7ad2cd61396d86813746b514cfa5.txt │ after-c5ce4d73147075e2a413b140db9f47d5fa0231e1.txt │
                                                                                                                    │                        B/op                         │            B/op              vs base               │
BucketStoreLabelValues/no_cache/10-series-matched-with-10-label-values-10                                                                                  19.056Mi ± ∞ ¹                 2.251Mi ± ∞ ¹  -88.19% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1000-series-matched-with-1000-label-values-10                                                                               39.98Mi ± ∞ ¹                 26.13Mi ± ∞ ¹  -34.64% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1_000_000-series-matched-with-10-label-values-10                                                                            186.3Mi ± ∞ ¹                 186.3Mi ± ∞ ¹   +0.00% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1_000_000-series-matched-with-1-label-values-10                                                                             36.86Mi ± ∞ ¹                 18.68Mi ± ∞ ¹  -49.33% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/10-series-matched-with-10-label-values-10                                              3171.70Ki ± ∞ ¹                 50.02Ki ± ∞ ¹  -98.42% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1000-series-matched-with-1000-label-values-10                                           26.905Mi ± ∞ ¹                 1.531Mi ± ∞ ¹  -94.31% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1_000_000-series-matched-with-10-label-values-10                                         169.6Mi ± ∞ ¹                 169.6Mi ± ∞ ¹        ~ (p=0.343 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1_000_000-series-matched-with-1-label-values-10                                        2986.42Ki ± ∞ ¹                 31.82Ki ± ∞ ¹  -98.93% (p=0.029 n=4)
geomean                                                                                                                                                     26.11Mi                       4.105Mi        -84.27%
¹ need >= 6 samples for confidence interval at level 0.95

                                                                                                                    │ before-ec33b862e34d7ad2cd61396d86813746b514cfa5.txt │ after-c5ce4d73147075e2a413b140db9f47d5fa0231e1.txt │
                                                                                                                    │                      allocs/op                      │          allocs/op           vs base               │
BucketStoreLabelValues/no_cache/10-series-matched-with-10-label-values-10                                                                                    1.704k ± ∞ ¹                  1.738k ± ∞ ¹   +1.97% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1000-series-matched-with-1000-label-values-10                                                                                49.51k ± ∞ ¹                  26.63k ± ∞ ¹  -46.21% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1_000_000-series-matched-with-10-label-values-10                                                                             1.595k ± ∞ ¹                  1.631k ± ∞ ¹   +2.26% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1_000_000-series-matched-with-1-label-values-10                                                                               872.5 ± ∞ ¹                   919.5 ± ∞ ¹   +5.39% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/10-series-matched-with-10-label-values-10                                                  902.0 ± ∞ ¹                   693.0 ± ∞ ¹  -23.17% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1000-series-matched-with-1000-label-values-10                                             70.08k ± ∞ ¹                  11.91k ± ∞ ¹  -83.00% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1_000_000-series-matched-with-10-label-values-10                                          2.042k ± ∞ ¹                  2.075k ± ∞ ¹   +1.62% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1_000_000-series-matched-with-1-label-values-10                                            517.0 ± ∞ ¹                   539.0 ± ∞ ¹   +4.26% (p=0.029 n=4)
geomean                                                                                                                                                      3.067k                        2.243k        -26.87%
¹ need >= 6 samples for confidence interval at level 0.95

BenchmarkBucketStoreLabelValues with options to choose from B or C

goos: darwin
goarch: arm64
pkg: github.com/grafana/mimir/pkg/storegateway
                                                                                                                    │ before-ec33b862e34d7ad2cd61396d86813746b514cfa5.txt │ after-ff93ed82f51fd0f940699c86db7ee66a99f63d6c.txt │
                                                                                                                    │                       sec/op                        │           sec/op             vs base               │
BucketStoreLabelValues/no_cache/10-series-matched-with-10-label-values-10                                                                                    7.011m ± ∞ ¹                  7.586m ± ∞ ¹        ~ (p=0.200 n=4)
BucketStoreLabelValues/no_cache/1000-series-matched-with-1000-label-values-10                                                                                26.69m ± ∞ ¹                  27.37m ± ∞ ¹        ~ (p=0.200 n=4)
BucketStoreLabelValues/no_cache/1_000_000-series-matched-with-10-label-values-10                                                                             28.77m ± ∞ ¹                  32.12m ± ∞ ¹  +11.64% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1_000_000-series-matched-with-1-label-values-10                                                                            13208.2µ ± ∞ ¹                  230.0µ ± ∞ ¹  -98.26% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/10-series-matched-with-10-label-values-10                                                 1.145m ± ∞ ¹                  1.191m ± ∞ ¹        ~ (p=0.343 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1000-series-matched-with-1000-label-values-10                                             24.61m ± ∞ ¹                  25.16m ± ∞ ¹        ~ (p=0.114 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1_000_000-series-matched-with-10-label-values-10                                          28.66m ± ∞ ¹                  28.89m ± ∞ ¹        ~ (p=0.343 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1_000_000-series-matched-with-1-label-values-10                                          603.96µ ± ∞ ¹                  82.16µ ± ∞ ¹  -86.40% (p=0.029 n=4)
geomean                                                                                                                                                      8.760m                        4.263m        -51.33%
¹ need >= 6 samples for confidence interval at level 0.95

                                                                                                                    │ before-ec33b862e34d7ad2cd61396d86813746b514cfa5.txt │ after-ff93ed82f51fd0f940699c86db7ee66a99f63d6c.txt │
                                                                                                                    │                        B/op                         │            B/op              vs base               │
BucketStoreLabelValues/no_cache/10-series-matched-with-10-label-values-10                                                                                   19.06Mi ± ∞ ¹                 19.06Mi ± ∞ ¹   +0.03% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1000-series-matched-with-1000-label-values-10                                                                               39.98Mi ± ∞ ¹                 40.05Mi ± ∞ ¹   +0.17% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1_000_000-series-matched-with-10-label-values-10                                                                            186.3Mi ± ∞ ¹                 186.3Mi ± ∞ ¹   +0.00% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1_000_000-series-matched-with-1-label-values-10                                                                           37746.5Ki ± ∞ ¹                 274.7Ki ± ∞ ¹  -99.27% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/10-series-matched-with-10-label-values-10                                                3.097Mi ± ∞ ¹                 3.103Mi ± ∞ ¹   +0.18% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1000-series-matched-with-1000-label-values-10                                            26.91Mi ± ∞ ¹                 26.96Mi ± ∞ ¹        ~ (p=0.057 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1_000_000-series-matched-with-10-label-values-10                                         169.6Mi ± ∞ ¹                 169.6Mi ± ∞ ¹        ~ (p=1.000 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1_000_000-series-matched-with-1-label-values-10                                        2986.42Ki ± ∞ ¹                 32.53Ki ± ∞ ¹  -98.91% (p=0.029 n=4)
geomean                                                                                                                                                     26.11Mi                       8.025Mi        -69.26%
¹ need >= 6 samples for confidence interval at level 0.95

                                                                                                                    │ before-ec33b862e34d7ad2cd61396d86813746b514cfa5.txt │ after-ff93ed82f51fd0f940699c86db7ee66a99f63d6c.txt │
                                                                                                                    │                      allocs/op                      │          allocs/op            vs base              │
BucketStoreLabelValues/no_cache/10-series-matched-with-10-label-values-10                                                                                    1.704k ± ∞ ¹                   1.780k ± ∞ ¹  +4.46% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1000-series-matched-with-1000-label-values-10                                                                                49.51k ± ∞ ¹                   49.59k ± ∞ ¹  +0.16% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1_000_000-series-matched-with-10-label-values-10                                                                             1.595k ± ∞ ¹                   1.661k ± ∞ ¹  +4.11% (p=0.029 n=4)
BucketStoreLabelValues/no_cache/1_000_000-series-matched-with-1-label-values-10                                                                               872.5 ± ∞ ¹                    897.0 ± ∞ ¹  +2.81% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/10-series-matched-with-10-label-values-10                                                  902.0 ± ∞ ¹                    945.0 ± ∞ ¹  +4.77% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1000-series-matched-with-1000-label-values-10                                             70.08k ± ∞ ¹                   70.15k ± ∞ ¹  +0.10% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1_000_000-series-matched-with-10-label-values-10                                          2.042k ± ∞ ¹                   2.103k ± ∞ ¹  +3.01% (p=0.029 n=4)
BucketStoreLabelValues/inmemory_cache_(without_label_values_cache)/1_000_000-series-matched-with-1-label-values-10                                            517.0 ± ∞ ¹                    563.0 ± ∞ ¹  +8.90% (p=0.029 n=4)
geomean                                                                                                                                                      3.067k                         3.175k        +3.51%
¹ need >= 6 samples for confidence interval at level 0.95

Which issue(s) this PR fixes or relates to

related to #4593

Checklist

• Tests updated
• [n/a] Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[dimitarvdimitrov]

This PR also makes the existing indexheader.Reader#LabelValues method unused. I will delete all the code related to that in a follow-up PR

[pracucci]

Fantastic job, as usual! You set the bar very high 🚀 I found a bit difficult to read the code because we have both "label values postings" and "matchers postings". I left some nit comments to try to clarify it with a bit of renaming.

[dimitarvdimitrov]

maybe something which isn't directly obvious - this lv- prefix means that we will cache expanded postings for LabelValues and Series under different keys. The premise is that the tradeoff for the two may be different for the same request matchers - Series may have pending matchers, LabelValues may not (see TestLabelValuesPostingsStrategy).

We can potentially cache the expanded postings when they are fully resolved from LabelValues under the Series entry as well. I'm not sure if this is worth pursuing. I'm also not sure how to measure its potential impact.

[pracucci]

I didn't realise, but I don't think it's a problem. I think it's fine caching them separately.

[pracucci]

Still LGTM ;)