Skip to content

feat: Write Iceberg Parquet data files natively via iceberg-rust (part 3 of 3) #5308

Description

@jordepic

What is the problem the feature request solves?

This is the third and final part of #4322 (accelerating Iceberg V2 write operations), which was split into three deliverables:

  1. Split-operator planfeat: Optionally split the Iceberg V2 write operator into distinct writer and committer operations #4658 (merged). Iceberg's single V2 write command is rewritten into a writer operator (inside AQE) and a committer operator, with file writing still running through iceberg-java.
  2. Native-write eligibility detectionfeat: detect Iceberg V2 writes and emit fall-back reasons #5298. An allowlist over the write's effective configuration decides whether a native write would produce the same outcome as iceberg-java, and reports a fall-back reason via extended EXPLAIN when it would not. The supported configuration and the accepted divergences are documented in iceberg-writes.md.

This issue tracks part 3: actually performing the per-task Parquet data-file write via iceberg-rust when detection accepts the write.

Describe the potential solution

High-level approach:

  • Conversion: CometIcebergNativeWrite.convert (currently a stub) produces a native write operator. The JVM resolves all write settings and passes them to the native side via protobuf; the vetted write.parquet.* configuration (compression codec/level, row-group/page/dictionary sizes, page row limit) is translated into parquet-rs WriterProperties. Footer statistics are written untruncated to match parquet-mr.
  • Writer modes: unpartitioned, clustered, and fanout writers, mirroring iceberg-java's writer selection.
  • Transport: the native side serialises the resulting DataFiles as an in-memory Iceberg manifest via iceberg-rust's ManifestWriter; the JVM decodes the bytes with Iceberg's own ManifestFiles.read. Both ends use their library's spec codec, so no hand-rolled Avro.
  • Commit: unchanged from part 1 — the recovered DataFile[] goes into SparkWrite$TaskCommit and BatchWrite.commit, preserving Iceberg's overwrite semantics, copy-on-write validation, WAP, and snapshot summaries.
  • Manifest metrics: assembled on the JVM at commit time using Iceberg's own MetricsConfig / ParquetUtil code against the written file's footer (plus NaN counts from the native writer, which the footer cannot provide). Java makes every metadata decision — metrics modes, the inferred-column cap, bound truncation, list/map bounds suppression — so the counts / none metrics restrictions from part 2 can be lifted.
  • Testing: parity tests comparing native output against iceberg-java per data type (including a targeted float/double NaN metrics test), plus end-to-end tests showing compaction / sort / z-order rewrites running fully native.

The guiding rule from part 2 carries over: if the native write cannot reproduce iceberg-java's outcome for the effective configuration, the write falls back — the documented accepted divergences (row-group/file-rolling cadence, compressed page bytes, footer key-value metadata, partition-path escaping) are the only exceptions.

Additional context

Detection (#5298) is deliberately conservative so this part can land incrementally: individual restrictions can be relaxed as the native writer proves each behaviour byte- or semantics-identical.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions