Update sortedness in metadata if disagree with ColumnStatistics

[J-HowHuang]

Description

Before #17965, we might see a raw column is unsorted according to metadata.properties but is actually sorted (false negative). For example:

column.xxx.cardinality = -2147483648
column.xxx.totalDocs = 48
column.xxx.dataType = LONG
column.xxx.bitsPerElement = 31
column.xxx.lengthOfEachEntry = 0
column.xxx.columnType = DATE_TIME
column.xxx.isSorted = false
column.xxx.hasDictionary = false
column.xxx.isSingleValues = true
column.xxx.totalNumberOfEntries = 48
column.xxx.minValue = 0
column.xxx.maxValue = 0

So when the table config enables dictionary on this column, during the segment load/reload, the forward index rebuild in updateIndicies. Yet the forward index creator is constructed using an AbstractColumnStatisticsCollector, which actually read through the column and report sorted.

Thus a SingleValueSortedForwardIndexCreator would be created here, write index to a file with a different file extension .sv.sorted.fwd. Later segment writer would try to read from .sv.unsorted.fwd because it determines the extension by metadata's sortedness.

So we see

java.io.FileNotFoundException: 
.../xxx.sv.unsorted.fwd (No such file or directory)
	at java.base/java.io.RandomAccessFile.open0(Native Method)
	at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:356)
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:273)
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:223)
	at org.apache.pinot.segment.spi.memory.PinotByteBuffer.readFrom(PinotByteBuffer.java:295)
	at org.apache.pinot.segment.local.segment.index.loader.LoaderUtils.writeIndexToV3Format(LoaderUtils.java:57)

Change

We should follow the metadata as the source of truth. IMO the easiest change here is to override the sortedness in IndexCreationContext with the value in segment metadata.

Test

Add an unit test to reproduce the failure scenario and verified it fails without the change, and pass with the change.

[Jackie-Jiang]

(deleted)

[Jackie-Jiang]

When a column is sorted, we should rewrite the column metadata so that it can correctly reflect the stats. After ForwardIndexHandler, the segment should be the same as directly creating the dictionary

[Jackie-Jiang]

If column is sorted, we should update the column metadata and make isSorted true. Essentially it should be the same as directly generating a dictionary encoded column

[J-HowHuang]

Got it. So now

1. the file extension is determined based on what statsCollector.isSorted() report, instead of from metadata
2. update the isSorted in metadata to statsCollector.isSorted()

[codecov-commenter]

Codecov Report

❌ Patch coverage is 95.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 65.28%. Comparing base (545210b) to head (0cb8d66).
⚠️ Report is 77 commits behind head on master.

Files with missing lines	Patch %	Lines
...ocal/segment/index/loader/ForwardIndexHandler.java	95.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18998      +/-   ##
============================================
+ Coverage     64.82%   65.28%   +0.46%     
- Complexity     1347     1405      +58     
============================================
  Files          3392     3418      +26     
  Lines        211647   214652    +3005     
  Branches      33302    33922     +620     
============================================
+ Hits         137199   140136    +2937     
+ Misses        63388    63209     -179     
- Partials      11060    11307     +247     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-21	65.28% <95.00%> (+0.46%)	⬆️
temurin	65.28% <95.00%> (+0.46%)	⬆️
unittests	65.28% <95.00%> (+0.46%)	⬆️
unittests1	56.87% <30.00%> (-0.14%)	⬇️
unittests2	37.80% <95.00%> (+0.62%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.