Add end-to-end Parquet VARIANT support

[xiangfu0]

Summary

Add an end-to-end VARIANT user journey for Apache Pinot:

• define a single-value VARIANT dimension that is retained in Pinot's raw forward index
• ingest top-level, non-repeated Apache Parquet VARIANT(1) values, including unshredded and shredded layouts
• materialize frequently queried paths during ingestion while retaining the original Variant value
• expose Spark-style parse, extraction, type, null, and JSON-rendering functions in both query engines
• carry Variant through Pinot's schema, data block, DDL, JDBC, and response layers
• add a packaged quickstart and an integration suite covering table creation, Parquet ingestion, and queries
• add a reproducible 2M-row JSON-versus-VARIANT benchmark covering ingestion, storage, and query performance

Raw Variant values are deliberately not orderable, comparable, or generally aggregatable. Pinot allows COUNT(raw_variant) but rejects raw comparison, grouping, distinct, ordering, join-key, set-operation, and other aggregate uses with guidance to extract a typed scalar first.

The persisted format, null semantics, ownership boundaries, compatibility
contract, and activation/rollback gates are recorded in
pinot-spi/VARIANT_DESIGN.md.

Try it

Build Pinot and launch the dedicated batch quickstart:

./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar
build/bin/quick-start-variant-batch.sh

The quickstart:

1. registers the variantEvents schema and offline table
2. ingests the committed Parquet VARIANT(1) fixture
3. uploads a five-row segment
4. runs the representative queries below

SET enableNullHandling=true;

SELECT eventType, COUNT(*)
FROM variantEvents
GROUP BY eventType
ORDER BY eventType;

SELECT
  eventId,
  variant_get(payload, '$.user.id', 'STRING') AS userId,
  variant_get(payload, '$.amount', 'DOUBLE') AS amount
FROM variantEvents
WHERE eventType = 'checkout'
ORDER BY eventId;

SELECT eventId, variantToJson(payload)
FROM variantEvents
ORDER BY eventId;

The complete sample is under
pinot-tools/src/main/resources/examples/batch/variantEvents.

Supported scope

• top-level, non-repeated Parquet VARIANT(1) columns
• a single-value Pinot VARIANT dimension stored without a dictionary
• raw forward-index retention plus optional ingestion-time scalar materialization
• storage null handling enabled by the schema or table
• query-time typed extraction and JSON rendering with SET enableNullHandling=true
• both single-stage and multi-stage query execution

Nested/repeated Variant columns, streaming ingestion, quoted path keys, and mixed-version queries over active Variant tables are outside this initial verified scope.

The automatic Parquet reader selection keeps the existing Avro-metadata
precedence. Files that contain Avro metadata must explicitly select the native
Parquet reader to ingest a Variant column.

Production safety

• centralizes Pinot's PVAR framing and validation in VariantEnvelope, freezes the version-1 bytes with a golden test, and assigns a new protobuf wire value without renumbering existing types
• reports an actionable full-upgrade error when an unknown query-wire type is received
• rejects schemas/tables without effective storage null handling and rejects VARIANT SQL functions when query null handling is disabled
• rejects unsupported index configurations and unsafe raw Variant query operations, including window partition/order keys and non-COUNT raw aggregates
• preserves historical Parquet reader selection, uses locale-independent type canonicalization, and converts Parquet dates to UTC epoch-day timestamps
• initializes Parquet converters atomically, publishes replacement readers transactionally, caches immutable Variant schema indexes, preserves progress when a malformed row is skipped, and tests terminal malformed-row behavior
• validates Parquet decimal precision, scale, byte bounds, and hostile exponents before allocation
• writes array-backed, direct, and read-only Parquet Binary values into the final PVAR envelope without a full-payload intermediate copy and without changing source buffer positions or limits
• compiles query paths and target types once, reuses allocation-free cursors, vectorizes existence checks, and shares cache/null lifecycle across the single-stage Variant functions
• parses literal parseJson/tryParseJson inputs once per query in both engines and specializes ingestion-time Variant functions with constant paths/types into compiled, cursor-reusing evaluators
• keeps tolerant extraction mismatch and numeric overflow on a non-throwing path, while preserving strict conversion behavior, and avoids BigInteger allocation for JSON integers that fit primitive ranges
• restores the Java client's legacy textual JSON-null contract while distinguishing SQL null, encoded Variant null, and the Variant string "null" across Arrow, JSON, HTTP JDBC, and gRPC JDBC results
• keeps only parquet-variant in pinot-common; Parquet column/schema dependencies remain isolated to the input plugin
• adds approximately 124 KiB to the shaded common artifact, with no Parquet column/schema classes
• covers raw forward-index min/max behavior and direct-buffer inputs with regressions
• documents the activation contract: upgrade the complete Pinot fleet and external ingestion jobs before registering a Variant schema, and do not roll back while a Variant table is active

japicmp reports no binary-incompatible changes for pinot-spi or
pinot-segment-spi. The published Java 11 artifacts retain classfile major
version 55.

Compatibility and rollout

Existing schemas, segments, and data types keep their current behavior. Older
components reject an unknown Variant wire type deterministically, but they
cannot operate on an active Variant table. Upgrade controllers, brokers,
servers, clients, and external ingestion jobs/plugins before creating a Variant
schema; do not activate Variant during a rolling mixed-version window.

Verification

• all 13 GitHub checks passed at final head acfba945dd1d, covering unit, integration, quickstart, linter, dependency, compatibility, binary compatibility, and container-security jobs
• focused wire, utility, transform, validation, planner, runtime, client, DDL, and Parquet reader suites
• 19 VariantTypeTest integration scenarios across both query engines
• 63-module integration-test reactor build
• 83-module clean shaded binary distribution build on JDK 25 (BUILD SUCCESS, 11m24s)
• 64-module pinot-perf package/test reactor with benchmark unit coverage (BUILD SUCCESS, 6m38s)
• packaged benchmark launcher integration test, including legacy JSON-path execution and CSV output validation
• 2M-row, four-segment JSON/VARIANT comparison completed across four alternating ingestion rounds; every query result matched independently generated ground truth
• Java 11 compatibility scan: classfile major version 55 for SPI, segment SPI, common, time-series SPI, Java client, and JDBC client
• packaged quickstart: table creation, five-row Parquet ingestion, materialized-field grouping, typed nested extraction, JSON rendering, SQL/Variant null semantics, and clean shutdown
• local JMH smoke: compiled reusable extraction is 90.953 ns/op with 0.001 B/op versus 92.328 ns/op and 112.001 B/op for the object-returning path
• local JMH smoke: unshredded Parquet conversion is 27.682, 29.470, and 30.850 ns/op for array-backed, direct, and read-only buffers respectively; all allocate the same 328 B/op final result
• spotless, checkstyle, license formatting, and license checks across all 16 affected modules
• git diff --check

Performance: 2M-row JSON versus VARIANT

Commit acfba945dd1d adds a reproducible end-to-end benchmark under pinot-perf. It generates paired Parquet files from the same deterministic logical events, builds four local Pinot segments per representation, validates every query against independently generated ground truth, and alternates build order across four rounds. Source generation is excluded from ingestion timing.

Environment: Apple M2 Max (12 cores), 32 GB RAM, macOS 26.5.2, Temurin 25+36, AC power. Each query result below is the median of the four round-level p50 values; every round uses two warmups followed by 10 measured sequential executions.

Ingestion and storage

Representation	Median build	Median rows/s	Paired ingestion result	Source Parquet	Pinot segment	Segment bytes/row
JSON raw (ZSTD)	32.942 s	60,760	baseline	285.52 MiB	294.16 MiB	154.23
VARIANT raw (ZSTD)	14.315 s	139,714	2.29x faster	279.18 MiB	298.89 MiB	156.70
JSON + JSON index	194.873 s	10,263	index-assisted, not format-only	285.52 MiB	1,253.70 MiB	657.30

The raw VARIANT Parquet source is 2.22% smaller than JSON. The resulting raw Pinot VARIANT segment is 1.61% larger than raw JSON. Adding the JSON index makes the JSON segment 4.26x the raw JSON size and is intentionally excluded from the raw-format ingestion conclusion.

Query latency

Workload	JSON p50	VARIANT p50	Paired result
Materialized eventType count (control)	0.326 ms	0.349 ms	0.96x; effectively neutral/noise floor
Nested numeric sum	2,674.153 ms	592.766 ms	VARIANT 4.49x faster
Nested tier filter	2,704.314 ms	681.794 ms	VARIANT 3.98x faster
Four-path numeric aggregate	9,037.254 ms	1,051.095 ms	VARIANT 8.60x faster
Selective country filter	JSON index: 6.553 ms	VARIANT raw scan: 645.687 ms	JSON index 98.69x faster; not a format-only comparison

The neutral materialized-column control and the stable paired round ratios support attributing the raw nested-query gap to parsing/extraction work rather than general broker/server overhead.

Run after packaging pinot-perf:

/path/to/jdk-25/bin/java -Xms4G -Xmx8G \
  -Dpinot.perf.jsonVariant.rows=2000000 \
  -Dpinot.perf.jsonVariant.segments=4 \
  -Dpinot.perf.jsonVariant.rounds=4 \
  -Dpinot.perf.jsonVariant.warmups=2 \
  -Dpinot.perf.jsonVariant.samples=10 \
  -Dpinot.perf.jsonVariant.includeLegacy=false \
  -Dpinot.perf.jsonVariant.cleanup=false \
  -Dpinot.perf.jsonVariant.workDir=/tmp/pinot-json-variant-benchmark \
  -cp 'pinot-perf/target/pinot-perf-pkg/lib/*' \
  org.apache.pinot.perf.BenchmarkJsonVariantComparison

Limitations: this is a local, in-process, single-stage, sequential warm-cache server/broker comparison over four segments. It does not model network transfer, multi-stage execution, concurrent clients, distributed scheduling, or production hardware. The workstation was interactive and had a long uptime, so paired alternating rounds and the materialized control are used to bound environmental noise.

Review notes

This is a design and implementation draft, not a merge-ready ownership
approval. It changes public SPI (FieldSpec.DataType.VARIANT,
VariantEnvelope, protobuf/DataSchema mappings) and requires explicit
compatibility sign-off before merge:

• SPI, schema, and query-wire owners
• Parquet and input-format plugin owners
• single-stage and multi-stage query-engine owners
• Java, JDBC, Arrow, JSON, and gRPC client/response owners

The draft is kept end-to-end so those owners can review one complete activation
contract: the public type, its only supported ingestion path, the raw-value
safety guards, and the executable acceptance test. No partial subset is safe to
activate: the public type without ingestion is unusable, ingestion without all
query guards exposes byte-layout semantics, and any subset without mixed-version
gates is unsafe to roll out. The design document records the dependency order
and the rule that VARIANT remains unactivatable until all safety pieces land.
If maintainers prefer smaller rollback units after design review, this draft can
be converted into that documented dependency-ordered stack: foundation and wire
contract; segment and Parquet ingestion; query functions and engines; clients
and DDL; then quickstart, integration, and benchmarks.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 79.32203% with 427 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.70%. Comparing base (ff1981c) to head (ca98faf).

Files with missing lines	Patch %	Lines
...va/org/apache/pinot/common/utils/VariantUtils.java	73.49%	118 Missing and 71 partials ⚠️
...ransform/function/VariantGetTransformFunction.java	67.92%	22 Missing and 12 partials ⚠️
...n/inputformat/parquet/ParquetVariantConverter.java	80.98%	14 Missing and 13 partials ⚠️
...uery/runtime/operator/operands/VariantOperand.java	82.95%	5 Missing and 10 partials ⚠️
...pinot/query/runtime/operator/HashJoinOperator.java	57.57%	9 Missing and 5 partials ⚠️
...not/common/evaluator/InbuiltFunctionEvaluator.java	82.35%	4 Missing and 8 partials ⚠️
...pinot/common/function/scalar/VariantFunctions.java	8.33%	11 Missing ⚠️
...java/org/apache/pinot/common/utils/DataSchema.java	62.96%	4 Missing and 6 partials ⚠️
...n/inputformat/parquet/ParquetAvroRecordReader.java	75.00%	9 Missing and 1 partial ⚠️
...anner/validation/VariantTypeValidationVisitor.java	88.15%	1 Missing and 8 partials ⚠️
... and 33 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19101      +/-   ##
============================================
+ Coverage     65.53%   65.70%   +0.16%     
+ Complexity     1423     1322     -101     
============================================
  Files          3432     3445      +13     
  Lines        218106   220086    +1980     
  Branches      34665    35021     +356     
============================================
+ Hits         142941   144597    +1656     
- Misses        63603    63747     +144     
- Partials      11562    11742     +180     

Flag	Coverage Δ
custom-integration1	?
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	?
java-25	65.70% <79.32%> (+0.16%)	⬆️
temurin	65.70% <79.32%> (+0.16%)	⬆️
unittests	65.69% <79.32%> (+0.16%)	⬆️
unittests1	57.09% <75.02%> (+0.21%)	⬆️
unittests2	38.51% <43.43%> (+0.60%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.