Speed up the sort when building forward index

[gf2121]

Based on the idea mentioned here:

1. If we use a stable sorter, we can only compare docIds because termIds are already in order.
2. If we take the maxDoc into consideration, we can save 1 round of reorder when maxDoc < (1 << 24).
3. We may even purely use an offline version of radix sorter to sort the whole file, since all we need is just 3 or 4 times reorder based on point 1 and 2.

Some thoughts / todos:

• I have not finished the performance benchmark.
• The radix sorter can not take advantage of ForkJoinPool by now.
• Do we need to turn to a pure memory radix sorter if memory budget is enough?
• This offline radix sorter is a bit different as i can not write to random address in a file. I have to play some trick here if I do not want to create 256 tmp files...

[gf2121]

To get an quick insight, i make a naive benchmark on the sorter, showing generally 5x faster than baseline.

• JVM 8G (result in the ram budget of OfflineSorter = 800MB)
• No ForkJoinPool, parallelism = 1.

max doc bits	term count	doc per term	baseline	candidate	diff
24	1000	1000	647	137	-78.83%
24	1000	10000	6709	1058	-84.23%
24	10000	1000	5858	1460	-75.08%
24	10000	10000	90835	12002	-86.79%
32	1000	1000	649	180	-72.27%
32	1000	10000	7219	1801	-75.05%
32	10000	1000	7368	1774	-75.92%
32	10000	10000	89758	12981	-85.54%

[gf2121]

I forked the LSBRadixSorter to sort longs and use it when ram budget is enough. Generally 5x faster than candidate, 25x faster than baseline.

max doc bits	term count	doc per term	baseline	candidate	candidate memory
24	1000	1000	647	137	27
24	1000	10000	6709	1058	242
24	10000	1000	5858	1460	248
24	10000	10000	90835	12002	/
32	1000	1000	649	180	21
32	1000	10000	7219	1801	261
32	10000	1000	7368	1774	220
32	10000	10000	89758	12981	/

[gf2121]

I indexed wikimidumall with:

• BPIndexReorder monfig mentioned here.
• BPMergePolicy on this commit (most recent commit passed CI).
• without docvalues and facets
• 8GB JVM heap.

The index time reduced around 7%:

	baselline	candidate	diff
index took	13852262	12785804	-7.70%

[jpountz]

Very nice, thanks for helping making recursive graph bisection faster! This is an interesting approach to running radix sort in an offline fashion. Initially I was a bit worried about random access, but it doesn't look too bad, especially as postings would generally fit in the page cache.

Maybe we can keep the "everything in memory" approach for later. It sounds like a good idea to me, it just feels like this 5x speedup would be enough for building the forward index to no longer be a bottleneck, and I worry that the in-memory optimization would decrease test coverage for the offline approach as most tests/benchmarks would work on small datasets that always go in memory.

[jpountz]

Nit: It would read better to me if the IndexOutput was an argument of reset rather than of addEntry

[gf2121]

I like it too!

[jpountz]

Add some comments that it's ok to only compare doc IDs since this is a LSD radix sort which is stable and term IDs are already sorted?

[jpountz]

LGTM

[jpountz]

Maybe add some comments about the fact that we expect this sorter to need ~16MB of heap (I think?)

[jpountz]

Remove the IndexOutput from flush as well?

[jpountz]

Unrelated to this change: I sent you an email to your Apache address, can you check it out? (Sorry for the noise on this PR, I don't know how else to contact you).

[gf2121]

I sent you an email to your Apache address, can you check it out?

Sorry for missing the email. Thank you so much for reminding me here!