You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Native-write eligibility detection — feat: detect Iceberg V2 writes and emit fall-back reasons #5298. An allowlist over the write's effective configuration decides whether a native write would produce the same outcome as iceberg-java, and reports a fall-back reason via extended EXPLAIN when it would not. The supported configuration and the accepted divergences are documented in iceberg-writes.md.
This issue tracks part 3: actually performing the per-task Parquet data-file write via iceberg-rust when detection accepts the write.
Describe the potential solution
High-level approach:
Conversion: CometIcebergNativeWrite.convert (currently a stub) produces a native write operator. The JVM resolves all write settings and passes them to the native side via protobuf; the vetted write.parquet.* configuration (compression codec/level, row-group/page/dictionary sizes, page row limit) is translated into parquet-rs WriterProperties. Footer statistics are written untruncated to match parquet-mr.
Transport: the native side serialises the resulting DataFiles as an in-memory Iceberg manifest via iceberg-rust's ManifestWriter; the JVM decodes the bytes with Iceberg's own ManifestFiles.read. Both ends use their library's spec codec, so no hand-rolled Avro.
Commit: unchanged from part 1 — the recovered DataFile[] goes into SparkWrite$TaskCommit and BatchWrite.commit, preserving Iceberg's overwrite semantics, copy-on-write validation, WAP, and snapshot summaries.
Manifest metrics: assembled on the JVM at commit time using Iceberg's own MetricsConfig / ParquetUtil code against the written file's footer (plus NaN counts from the native writer, which the footer cannot provide). Java makes every metadata decision — metrics modes, the inferred-column cap, bound truncation, list/map bounds suppression — so the counts / none metrics restrictions from part 2 can be lifted.
Testing: parity tests comparing native output against iceberg-java per data type (including a targeted float/double NaN metrics test), plus end-to-end tests showing compaction / sort / z-order rewrites running fully native.
The guiding rule from part 2 carries over: if the native write cannot reproduce iceberg-java's outcome for the effective configuration, the write falls back — the documented accepted divergences (row-group/file-rolling cadence, compressed page bytes, footer key-value metadata, partition-path escaping) are the only exceptions.
Additional context
Detection (#5298) is deliberately conservative so this part can land incrementally: individual restrictions can be relaxed as the native writer proves each behaviour byte- or semantics-identical.
What is the problem the feature request solves?
This is the third and final part of #4322 (accelerating Iceberg V2 write operations), which was split into three deliverables:
iceberg-writes.md.This issue tracks part 3: actually performing the per-task Parquet data-file write via iceberg-rust when detection accepts the write.
Describe the potential solution
High-level approach:
CometIcebergNativeWrite.convert(currently a stub) produces a native write operator. The JVM resolves all write settings and passes them to the native side via protobuf; the vettedwrite.parquet.*configuration (compression codec/level, row-group/page/dictionary sizes, page row limit) is translated into parquet-rsWriterProperties. Footer statistics are written untruncated to match parquet-mr.DataFiles as an in-memory Iceberg manifest via iceberg-rust'sManifestWriter; the JVM decodes the bytes with Iceberg's ownManifestFiles.read. Both ends use their library's spec codec, so no hand-rolled Avro.DataFile[]goes intoSparkWrite$TaskCommitandBatchWrite.commit, preserving Iceberg's overwrite semantics, copy-on-write validation, WAP, and snapshot summaries.MetricsConfig/ParquetUtilcode against the written file's footer (plus NaN counts from the native writer, which the footer cannot provide). Java makes every metadata decision — metrics modes, the inferred-column cap, bound truncation, list/map bounds suppression — so thecounts/nonemetrics restrictions from part 2 can be lifted.The guiding rule from part 2 carries over: if the native write cannot reproduce iceberg-java's outcome for the effective configuration, the write falls back — the documented accepted divergences (row-group/file-rolling cadence, compressed page bytes, footer key-value metadata, partition-path escaping) are the only exceptions.
Additional context
Detection (#5298) is deliberately conservative so this part can land incrementally: individual restrictions can be relaxed as the native writer proves each behaviour byte- or semantics-identical.