store-gateway: report touched postings & series instead of fetched (#…

…4671)

* store-gateway: report touched postings & series instead of fetched

The store-gateway sends statistics about each request in its response to
 the querier. My understanding is that the purpose of these stats is to
 help gauge the cost of a query. Currently, the store-gateway reports
 its fetched bytes. I propose to report touched bytes instead.

__Fetched index bytes__ are based on the number of bytes fetched from
the bucket. This excludes bytes fetched from the cache and includes bytes overfetched
because of an incorrect size estimation or when joining adjacent ranges
from the index object.

__Touched bytes__ are the sum of bytes that were
directly necessary in order to serve the request. This means that they
exclude bytes that we overfetched (e.g. when joining adjacent regions of
an object in the object store or when incorrectly estimating the size of
a series).

The number of touched bytes should not generally change between query
executions and will give a better picture of how expensive the query was
and not depend on cache hit rates or the adjacency of requested regions
in the index.

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Add changelog entry

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

---------

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

CHANGELOG.md

@@ -84,6 +84,7 @@
 * [BUGFIX] OTLP: Do not drop exemplars of the OTLP Monotonic Sum metric. #4063
 * [BUGFIX] Packaging: flag `/etc/default/mimir` and `/etc/sysconfig/mimir` as config to prevent overwrite. #4587
 * [BUGFIX] Query-frontend: don't retry queries which error inside PromQL. #4643
+* [BUGFIX] Store-gateway & query-frontend: report more consistent statistics for fetched index bytes. #4671
 
 ### Mixin
 

pkg/storegateway/bucket.go

@@ -713,7 +713,7 @@ func (s *BucketStore) Series(req *storepb.SeriesRequest, srv storepb.Store_Serie
 	}
 
 	unsafeStats := stats.export()
-	if err = srv.Send(storepb.NewStatsResponse(unsafeStats.postingsFetchedSizeSum + unsafeStats.seriesFetchedSizeSum)); err != nil {
+	if err = srv.Send(storepb.NewStatsResponse(unsafeStats.postingsTouchedSizeSum + unsafeStats.seriesTouchedSizeSum)); err != nil {
 		err = status.Error(codes.Unknown, errors.Wrap(err, "sends series response stats").Error())
 		return
 	}