Use doc values skipper for _tsid in synthetic _id postings

[tlrx]

Instead of scanning all documents to find the first document that has a _tsid less than, or equal to, a given ordinal we can use a doc values skipper to skip as much as possible documents, and only then scan. the remaining docs.

When seeking a synthetic _id, we look up the _tsid ordinal, then use DV skipper to find a starting doc ID, then scan each doc to find the first doc ID matching the exact _tsid ordinal. Then we finally scan remaining docs to find the one matching the timestamp.

I wonder if that also makes sense to use DV skipper for the timestamp too?

Relates ES-13604

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[elasticsearchmachine]

Hi @tlrx, I've created a changelog YAML for you.

[romseygeek]

Does this need to be gated by an IndexVersion?

[tlrx]

I don't think it needs but it's better to be safe indeed. So I reverted the change which uses the USE_DOC_VALUES_SKIPPER index setting.

[tlrx]

See 9ff33a7

[fcofdez]

Just for my understanding, if we don't find the tsIdOrd at level 0, this will return NO_MORE_DOCS? I think that I might be missing something here.

[tlrx]

If the ordinal is not found in the first level 0, then it skips to the next levels until it finds a level that includes the ordinal or exhaust the iterator, in which case the Javadoc indicates that minDocs returns NO_MORE_DOCS.

[tlrx]

My understanding is that a DocValuesSkipper is kind of a skip list on top of docs values blocks of data.

If that helps, here is a representation of such skipper levels:

minValue: 0, maxValue: 0, [minDocID: 0, maxDocID: 31], docCount: 32, level: 0/3
minValue: 0, maxValue: 7, [minDocID: 0, maxDocID: 718], docCount: 719,  level: 1/3
minValue: 0, maxValue: 64, [minDocID: 0, maxDocID: 4732], docCount: 4733,  level: 2/3
minValue: 0, maxValue: 0, [minDocID: -1, maxDocID: -1], docCount: 0,  level: 3/3
minValue: 1, maxValue: 1, [minDocID: 32, maxDocID: 178], docCount: 147, level: 0/3
minValue: 0, maxValue: 7, [minDocID: 0, maxDocID: 718], docCount: 719,  level: 1/3
minValue: 0, maxValue: 64, [minDocID: 0, maxDocID: 4732], docCount: 4733,  level: 2/3
minValue: 0, maxValue: 0, [minDocID: -1, maxDocID: -1], docCount: 0,  level: 3/3
minValue: 2, maxValue: 2, [minDocID: 179, maxDocID: 269], docCount: 91, level: 0/3
minValue: 0, maxValue: 7, [minDocID: 0, maxDocID: 718], docCount: 719,  level: 1/3
minValue: 0, maxValue: 64, [minDocID: 0, maxDocID: 4732], docCount: 4733,  level: 2/3
minValue: 0, maxValue: 0, [minDocID: -1, maxDocID: -1], docCount: 0,  level: 3/3
...
minValue: 8, maxValue: 8, [minDocID: 719, maxDocID: 765], docCount: 47, level: 0/3
minValue: 8, maxValue: 15, [minDocID: 719, maxDocID: 1440], docCount: 722,  level: 1/3
minValue: 0, maxValue: 64, [minDocID: 0, maxDocID: 4732], docCount: 4733,  level: 2/3
minValue: 0, maxValue: 0, [minDocID: -1, maxDocID: -1], docCount: 0,  level: 3/3
...

For example, when looking for tsIdOrd == 9 the advance(min, max) method executes:

• the first level minValue: 0, maxValue: 0, [minDocID: 0, maxDocID: 31] has max value 0 below 9 so we can skip to maxDocID + 1 = 32
• while there we can check if we can skip even more docs so we look up the next level 1 which is minValue: 0, maxValue: 7, [minDocID: 0, maxDocID: 718] which also has a max value 7 < 9 so we can in fact skip to maxDocID + 1 = 718 + 1 = 719
• next level 2 has a max value of 64 so we cannot skip more
• we advance the iterator to 719
• our new level 0 is now minValue: 8, maxValue: 8, [minDocID: 719, maxDocID: 765], with max value of 8 we can skip all docs until 765 +1
• while there we check if we can skip more in the next level 1, which is minValue: 8, maxValue: 15, [minDocID: 719, maxDocID: 1440] and has max value of 15, so tsIdOrd == 9 is between docs ids [766, 1440]
• the while loop ends with minDocs(0) == 766

I hope it helps. It took me some time to understand all of this 🫠

[fcofdez]

Thanks for the detailed explanation, this makes sense 👍

[romseygeek]

If the skipper minValue is greater than your requested tsid then that means the tsid isn't present in the segment, so this should probably also return NO_MORE_DOCS? Or maybe trigger an assertion?

[tlrx]

Right, I pushed b3fc3b9

[fcofdez]

LGTM

[tlrx]

Thanks Francisco & Alan!