[VL] feat(iceberg): Add V3 initial value reads - #12537
Merged
Merged
Conversation
|
Run Gluten Clickhouse CI on x86 |
infvg
force-pushed
the
iceberginitvalread
branch
from
July 16, 2026 16:53
a0f00c0 to
0191abd
Compare
|
Run Gluten Clickhouse CI on x86 |
infvg
force-pushed
the
iceberginitvalread
branch
from
July 16, 2026 17:39
0191abd to
d42bf74
Compare
|
Run Gluten Clickhouse CI on x86 |
infvg
force-pushed
the
iceberginitvalread
branch
from
July 20, 2026 08:50
d42bf74 to
ca3eab5
Compare
|
Run Gluten Clickhouse CI on x86 |
infvg
force-pushed
the
iceberginitvalread
branch
from
July 20, 2026 09:10
ca3eab5 to
103561f
Compare
|
Run Gluten Clickhouse CI on x86 |
infvg
marked this pull request as ready for review
July 20, 2026 13:44
Member
|
@infvg could you please help to update on the doc(https://github.com/apache/gluten/blob/main/docs/get-started/VeloxIceberg.md) as well? Please also help to do a rebase as the Velox code has been advanced recently |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds end-to-end plumbing for Iceberg V3 “initial default” values in the Velox backend by propagating Iceberg field IDs + initial-default metadata through Substrait LocalFiles advanced extensions, then consuming that metadata when constructing Velox Iceberg column handles.
Changes:
- Introduces a backend capability flag and a Velox-specific read extension packer for Iceberg initial defaults.
- Extends Iceberg LocalFiles nodes to carry and serialize initial-default metadata via Substrait
advanced_extension. - Adds Velox-side parsing of the new extension and a Velox Iceberg V3 initial-default unit test.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/TransformerApi.scala | Adds a default hook for packing Iceberg initial-default metadata into a backend-specific extension. |
| gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/BackendSettingsApi.scala | Adds a backend feature flag for Iceberg initial-default reads (default: false). |
| gluten-iceberg/src/main/scala/org/apache/iceberg/spark/source/GlutenIcebergSourceUtil.scala | Collects Iceberg field IDs + initial defaults from the scan and attaches them to split info. |
| gluten-iceberg/src/main/scala/org/apache/gluten/execution/IcebergScanTransformer.scala | Validates backend support for initial defaults and passes scan context into split generation. |
| gluten-iceberg/src/main/java/org/apache/gluten/substrait/rel/IcebergLocalFilesNode.java | Emits the new Iceberg read extension in LocalFiles.advanced_extension when defaults exist. |
| gluten-iceberg/src/main/java/org/apache/gluten/substrait/rel/IcebergLocalFilesBuilder.java | Wires new constructor parameters (field IDs, initial defaults) into Iceberg LocalFiles node creation. |
| gluten-iceberg/src-iceberg5/main/java/org/apache/gluten/IcebergDefaultValueUtil.java | Introduces Iceberg-version-specific helper for reading initial defaults (stubbed to null here). |
| gluten-iceberg/src-iceberg3/main/java/org/apache/gluten/IcebergDefaultValueUtil.java | Introduces Iceberg-version-specific helper for reading initial defaults (stubbed to null here). |
| gluten-iceberg/src-iceberg10/main/java/org/apache/gluten/IcebergDefaultValueUtil.java | Implements initial default extraction via field.initialDefault(). |
| cpp/velox/substrait/SubstraitToVeloxPlan.cc | Uses parsed Iceberg default metadata to select IcebergColumnHandle when needed. |
| cpp/velox/compute/VeloxPlanConverter.cc | Passes LocalFiles.advanced_extension into Iceberg split parsing. |
| cpp/velox/compute/iceberg/IcebergPlanConverter.h | Adds column metadata (fieldId + initialDefault) to Iceberg split info and updates parsing signature. |
| cpp/velox/compute/iceberg/IcebergPlanConverter.cc | Unpacks and stores the new Iceberg read extension from advanced_extension. |
| backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxTransformerApi.scala | Implements packing of IcebergReadExtension protobuf into a Substrait Any. |
| backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala | Enables Iceberg initial-default reads for Velox backend settings. |
| backends-velox/src/main/resources/org/apache/gluten/proto/IcebergReadExtension.proto | Defines the protobuf payload carried in LocalFiles.advanced_extension. |
| backends-velox/src-iceberg/test/scala/org/apache/gluten/execution/VeloxIcebergSuite.scala | Adds a test validating Iceberg V3 initial default values are returned under Velox. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
216
to
218
| case p: SparkDataSourceRDDPartition => | ||
| GlutenIcebergSourceUtil.genSplitInfo(p, getPartitionSchema, metadataColumnNames) | ||
| GlutenIcebergSourceUtil.genSplitInfo(p, scan, getPartitionSchema, metadataColumnNames) | ||
| case _ => throw new GlutenNotSupportException() |
Comment on lines
+108
to
111
| metadataColumns, | ||
| getFieldIds(sparkScan), | ||
| getInitialDefaults(sparkScan) | ||
| ) |
infvg
force-pushed
the
iceberginitvalread
branch
from
August 13, 2026 13:30
103561f to
bbb37b0
Compare
|
Run Gluten Clickhouse CI on x86 |
infvg
force-pushed
the
iceberginitvalread
branch
from
August 13, 2026 18:48
bbb37b0 to
9e167a1
Compare
|
Run Gluten Clickhouse CI on x86 |
zhouyuan
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Iceberg V3 supports initial default values for newly added columns, but the velox backend did not previously propagate the required metadata to return those defaults.
Changes
This PR carries Iceberg field IDs and initial-default metadata to Velox.
Impact
Velox can now correctly return configured initial default values instead of
NULLwhen reading older rows from Iceberg V3 tables.Feature
https://iceberg.apache.org/spec/#default-values