SelectMerge latency is high

[kolesnikovae]

Most of the queries spend significant time after all the samples were fetched and deduplicated. We need to figure out and fix what causes the latency.

For example, phlare-querier SelectMergeStacktraces:

Solving the problem may result in a very significant decrease in the overall query latencies (up to 50%).

It’s very likely that this is caused by pulling and merging resolved stacks from the ingesters: the size of the payload may be quite big. If this is the case, we may want to find a way to reduce it, e.g. by stack truncation:

We may want to optimize the representation of results in this API: notice that stacks take appx. 50MB (encoded in protobuf). We should consider building a truncated tree (w/o insignificant nodes) as close to the storage as possible, instead of passing an array of stack traces along the way. That would also decrease CPU time of the query execution, and allocations as well:

[kolesnikovae]

Also related: grafana/pyroscope#2107

[cyriltovena]

Based on our discussions I think this is a great idea to use an opaque format for the internal API. We should use the Pyroscope tree package https://github.com/grafana/pyroscope/blob/254c5759900d3a2da6e6dfaf4ef2767c05cb45bb/pkg/storage/tree/tree.go between query-frontend - querier - ingester and store-gateway.

[kolesnikovae]

#702 reduces the query duration 2-3 times in certain cases:

As a bonus, resource consumption also decreased:

Closing the issue for now