TSDB doc value for skipping documents per tsid

[iverase]

Time series indices are always sorted first by the _tsid field which identifies uniquely a time series and secondly by the timestamp in reverse order. A typical query on this type of data is to return the last metrics for each unique time series. Because data is ordered in the segments, we only need to read the first document for each _tsid. Unfortunately we currently don't have the information to skip the documents to the next _tsid so we are still reading all documents in the index to answer that question. _tsid fields are stored as lucene sorted doc values.

This PR tries to provide a mechanism to efficiently skip documents from a _tsid. It does it by creating a new doc value format for _tsid fields. This doc value format is just a wrapper on top of lucene sorted doc values but adds an extra file to the current format that provides a jump table between _tsids. This size of the new file depends on the number of _tsids in the segment and it will have a maximum size of 4 * number of _tsid but typically will be significantly smaller.

I tried to make the PR as compatible as possible to #92045

[martijnvg]

This looks great!
Do you think that as part of this pr, we already want to make use of TSIDSortedDocValues? E.g. in top_metric agg when sorting by timestamp field?

[martijnvg]

Maybe leave this method abstract? I don't think want this implementation to run in production?
The ES87TSDBDocValuesFormat.java implementation at line 132 should always be used.

[iverase]

The idea of this implementation is that we can use this approach even for old indices?

[martijnvg]

Good point, even with this implementation aggregations can benefit from just skipping to the next tsid.

[martijnvg]

I think that mapperService should never be null?

[iverase]

yes, this is a leftover

[iverase]

Do you think that as part of this pr, we already want to make use of TSIDSortedDocValues?

My idea is that TSIDSortedDocValues might be used by the leaf walker to skip documents if we only need to visit the first documents for each tsid. Still we need a mechanism that tells the walker that we are in such a mode. Here is @jpountz suggestion on how we can achieve it: #87564 (comment).

wdyt?

[martijnvg]

My idea is that TSIDSortedDocValues might be used by the leaf walker to skip documents if we only need to visit the first documents for each tsid. Still we need a mechanism that tells the walker that we are in such a mode. Here is @jpountz suggestion on how we can achieve it: #87564 (comment).

I see, that makes sense to me. Maybe aggregation implementation can just check the AggregationExecutionContext to check whether they can utilise the mode that allows them to skip to the next tsid (similar to the change made to here).

[StephanErb]

This still sounds quite useful for many TSDB query usecases! Are there any plans to bring this forward? Or is the idea discarded in favor of the more generic #95701?