New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LUCENE-9539: Use more compact datastructures for sorting doc-values #1908
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sparse sorting, nice! The change looks great to me.
I was wondering if we could also compact the offsets array but that would require an extra sort per field. Although, that's probably not needed since we can also remove the caching of the doc values in order to ensure that we load only one field at a time during a merge.
} | ||
if (startOffset != ordOffset) { // do we have any values? | ||
offsets[newDocID] = startOffset; | ||
builder.add(0); // 0 ord marks next value |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Technically you could also have a sign switch as boundary (~value marks first value) but I don't think it'd result in any savings and would only complicate the value stream.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agreed, I didn't do it since it would have impacted readability too much IMO
Nice. +1. |
…1908) This change cuts over from object based data-structures to primitive / compressed data-structures.
This change cuts over from object based datastructures to primitive / compressed datastructures.