-
Notifications
You must be signed in to change notification settings - Fork 437
Description
iceberg-rust currently supports FastAppendAction for simple appends but is missing several transaction actions available in the Java Iceberg API that are needed for complete write support.
| Java API (api/src/main/java/org/apache/iceberg/) | iceberg-rust (crates/iceberg/src/transaction/) | Status |
|---|---|---|
| AppendFiles.java | append.rs (FastAppendAction) | Available |
| OverwriteFiles.java | - | Missing |
| ReplacePartitions.java | - | Missing |
| DeleteFiles.java | - | Missing |
| RowDelta.java | - | Missing |
| RewriteFiles.java | - | Missing |
TODOs:
- Implement OverwriteAction - needed for INSERT OVERWRITE (static partition overwrite)
- Implement ReplacePartitionsAction - needed for dynamic partition overwrite
- Implement DeleteFilesAction - needed for DELETE FROM table operations
- Implement RowDeltaAction - needed for row-level DELETE, UPDATE, MERGE INTO (CoW/MoR)
- Implement RewriteFilesAction - needed for compaction and file rewriting
Related open issues:
#1104 - Support RowDeltaAction
#2202 - Implement RowDeltaAction for row-level modifications (CoW)
#2201 - Add MERGE INTO support for DataFusion integration
#2205 - SQL UPDATE support for DataFusion integration
#2186 - Copy-on-Write and Merge-on-Read support
#1382 - [EPIC] Support for appending data to iceberg table
#2218 - Support for a DeltaWriter
Motivation:
These transaction actions are required for apache/datafusion-comet to implement a fully native Iceberg write path (datafusion-comet#3595, datafusion-comet#2967). Comet currently writes Parquet data files natively via Rust/DataFusion but still relies on the Java Iceberg layer for table commits. Completing these actions in iceberg-rust would enable end-to-end native Iceberg writes - from Arrow batches through DataFusion execution to Iceberg table commits - eliminating JVM dependency for the entire write path.