[Arrow] Typed zero-boxing column-major build for single-value primitive columns

[real-mj-song]

TL;DR: The column-major build added in #18638 removed per-row GenericRow materialization, but it still consumes every value through the generic Object path — so each primitive is boxed twice, once on the stats pass and once on the index pass. This PR adds a typed, boxing-free fast path for single-value INT/LONG/FLOAT/DOUBLE columns: ColumnReader.nextInt() → collect(int) / indexOfSV(int) → IndexCreator.addInt(value, dictId). Purely additive; nulls, multi-value, and every other type fall back to the unchanged Object path. No SPI changes, no new public API. It also logs a one-line fast-path/Object-path column summary per build.

Tracks #18629. Builds on #18638 (column-major build) and the ColumnReader / ColumnReaderFactory SPI (#16727).

Problem

#18629 set out to build a segment from a columnar source without "the per-row GenericRow allocation and per-primitive boxing." #18638 delivered the column-major path and removed the GenericRow allocation, but it routes values through the generic Object API:

columnReader.next()                         // Object — boxes the primitive
  -> ColumnarValueNormalizer.normalize(..)  // Object
  -> dictionaryCreator.indexOfSV(Object)    // boxed dictionary lookup
  -> creator.add(Object, dictId)            // unboxes again to write

So the per-primitive boxing #18629 aimed to remove is still on the hot path — and twice over, since buildColumnar() reads every column once for stats and once for indexing. (This is the follow-up requested in the #18638 review.)

What this PR does

The ColumnReader SPI (#16727) already exposes typed accessors (nextInt(), isInt(), …) and the downstream sinks already have primitive overloads — AbstractColumnStatisticsCollector.collect(int), SegmentDictionaryCreator.indexOfSV(int), and addInt(value, dictId) on ForwardIndexCreator / CombinedInvertedIndexCreator / DictionaryBasedInvertedIndexCreator. They were simply never wired to the column-major build. This PR wires them:

• Stats pass (ColumnarSegmentPreIndexStatsContainer) — when the reader can serve a single-value column as a primitive (isInt()/isLong()/…), read with nextInt()/… and feed collect(int)/… directly.
• Index pass (SegmentColumnarIndexCreator) — same dispatch: nextInt() → indexOfSV(int) → creator.addInt(value, dictId) across the column's index creators (creators with no typed override keep the boxing default and stay correct).

Key choices:

• Gated on the logical FieldSpec type (INT/LONG/FLOAT/DOUBLE), so BOOLEAN (stored INT) and TIMESTAMP (stored LONG) — which require value coercion the typed read would skip — correctly stay on the Object path.
• Null docs route through the existing Object path for byte-for-byte parity, including null-value-vector marking and default substitution. The typed primitive accessor is only called when isNextNull() is false.
• Multi-value and non-primitive types are unchanged.

Net effect: for single-value primitive columns, neither build pass boxes — removing the remaining per-primitive boxing on both column reads.

Potential saving

Each column is read twice (stats pass + index pass), so the Object path boxes every single-value primitive value 2 × N times for an N-row column. The typed path avoids, per such column:

Type	Garbage avoided
INT / FLOAT	2 × N × 16 B
LONG / DOUBLE	2 × N × 24 B

Wrapper sizes assume a 64-bit JVM with compressed oops (Integer / Long skip the −128..127 cache). For example a 1M-row column eliminates ~32 MB (INT / FLOAT) or ~48 MB (LONG / DOUBLE) of short-lived allocation and 2 × N boxing ops — allocation/GC pressure only; retained heap and segment size are unchanged.

Scope

Single-value INT/LONG/FLOAT/DOUBLE only. The typed multi-value sinks (addIntMV, nextIntMV / MultiValueResult) already exist but are intentionally not wired here — the multi-value element-null contract is a distinct concern and is better handled on its own.

Compatibility

Additive and behavior-preserving: no SPI changes, no new public API; the row-major path and the Object-path column-major fallback are untouched. The only non-fast-path edit documents the existing two-pass rewind() precondition on the init(config, ColumnReaderFactory) overload (the consumer that imposes it), leaving the ColumnReaderFactory SPI itself generic.

Observability

Emits one INFO summary per column-major build — how many single-value primitive columns took the typed fast path vs. the Object-path fallback (multi-value, non-primitive single-value, or a fast-path primitive whose reader served a different physical type than the schema). Counts are incremented once per column where the path is chosen, so the summary can't drift from the path taken. No SPI or public-API change.

Testing

The de-box is behavior-preserving (output-identical), so it's largely covered by the existing column-major equivalence suites; one new test is added for a previously-unexercised fallback:

• ColumnarRowMajorEquivalenceTest — column-major vs. row-major over single-value INT/LONG/FLOAT/DOUBLE with ~10% nulls, asserting per-doc value and min/max/cardinality equality (typed dispatch + null branch).
• The pinot-arrow column-major suite (e.g. testRichMultiBatchEquivalence, INT with nulls across batches) exercises the typed path via an Arrow source.
• New — testSourceSchemaTypeMismatchInt64ToIntEquivalence (pinot-arrow): an INT64 Arrow source declared as INT in the schema declines the fast path and coerces via the Object path; asserts byte-for-byte parity with row-major.

References

Tracks #18629 · builds on #18638 · ColumnReader SPI #16727

[codecov-commenter]

Codecov Report

❌ Patch coverage is 54.71698% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.80%. Comparing base (6d319a5) to head (6f1c180).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
...ment/creator/impl/SegmentColumnarIndexCreator.java	58.82%	16 Missing and 12 partials ⚠️
...l/stats/ColumnarSegmentPreIndexStatsContainer.java	47.22%	11 Missing and 8 partials ⚠️
...t/creator/impl/SegmentIndexCreationDriverImpl.java	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             master   #18797    +/-   ##
==========================================
  Coverage     64.80%   64.80%            
  Complexity     1347     1347            
==========================================
  Files          3392     3392            
  Lines        211648   211754   +106     
  Branches      33302    33341    +39     
==========================================
+ Hits         137152   137222    +70     
- Misses        63436    63442     +6     
- Partials      11060    11090    +30     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-21	64.80% <54.71%> (+<0.01%)	⬆️
temurin	64.80% <54.71%> (+<0.01%)	⬆️
unittests	64.79% <54.71%> (+<0.01%)	⬆️
unittests1	56.94% <0.94%> (-0.06%)	⬇️
unittests2	37.17% <54.71%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[siddharthteotia]

When will we run into this ?

[real-mj-song]

If Arrow producers write a wider/narrower int than the Pinot schema (Arrow int64 or INT8/INT16 for an INT column), this condition kicks in

[siddharthteotia]

Thanks for addressing the original ask of optimizing away the boxing overhead. Couple of point

• Can we put this behind a flag ? I mean the new path generates segments identical to previous path but just in case.

• I don't see any log or metric to indicate which path fired. You may need it initially at least as we enable this and debug (if there are any issues)

• Do we not need to add new tests to get coverage on the new path or is this already covered ?

[real-mj-song]

Thanks @siddharthteotia. Correctness is already tested in existing tests, but let me add specific tests for internal type mismatches. Flag is not necessary because columnar builder is already opt-in only in SegmentIndexCreationDriverImpl (with 0 caller now)

pinot/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java

Lines 159 to 162 in 6d319a5

	public void init(SegmentGeneratorConfig config, ColumnReaderFactory columnReaderFactory)
	throws Exception {
	// Initialize the column reader factory with target schema
	columnReaderFactory.init(config.getSchema());

Also adding a summary log on which path was fired.