fix: Delete data files left by a failed native Iceberg write task - #5652
fix: Delete data files left by a failed native Iceberg write task#5652andygrove wants to merge 1 commit into
Conversation
iceberg-java's writer abort deletes the files a failed task attempt wrote; the native path left them for remove_orphan_files. Close the gap in both places a task can fail. Inside the native writer, a TrackingLocationGenerator records every location handed to a file writer, since iceberg-rust's writers keep finalized files private until close and have no abort hook. The task deletes the recorded locations when a write fails, and an AbortOnDrop guard does the same when the task future is dropped without ever seeing an error, which is what happens when the JVM input iterator throws: executePlan returns that error from its JNI batch pull and the JVM releases the plan. After the native writer has returned, CometIcebergWriteExec registers a task failure listener that deletes the decoded manifest's files through the table FileIO, via a new best-effort IcebergReflection helper. Both deletions log failures rather than raising them, so the original task failure is the one Spark reports. Closes apache#5618
| if outcome.is_err() { | ||
| delete_task_files(&file_io, location_generator.locations()).await; | ||
| } | ||
| abort_guard.disarm(); |
There was a problem hiding this comment.
We disarm the cleanup guard here as soon as run_write_task completes, but the outer IcebergWriteExec::execute still performs fallible encode_data_files_as_manifest(...).await? and build_output_batch(...)? operations afterward. If either fails, the data files have already been written, this guard is disarmed, and the JVM never receives a manifest from which it could recover their locations. That leaves the task's files orphaned.
Could we keep cleanup ownership alive through manifest encoding/output-batch construction and disarm only once the native result is successfully materialized? One option would be for run_write_task to return the cleanup token alongside the DataFiles and let the outer task disarm it after packaging succeeds.
| // serialization), delete them the way iceberg-java's `DataWriter.abort()` would; failures | ||
| // inside the native writer itself are cleaned up on the native side. | ||
| Option(TaskContext.get()).foreach { tc => | ||
| tc.addTaskFailureListener(new TaskFailureListener { |
There was a problem hiding this comment.
By the time this listener is registered, both drainAvroPayload(batches) and decodeManifestToDataFiles(manifestBytes, specId) have already run. A failure in either operation happens after the native writer has successfully produced its data files, but before this listener owns cleanup. Since the native guard has also already been disarmed on successful writer close, neither side can remove those files.
This is especially important for decodeManifestToDataFiles: the cleanup paths are currently recoverable only by successfully decoding the same manifest whose decode may fail. I think cleanup ownership/locations need to cross this boundary independently of successful manifest decoding (or native cleanup needs to remain armed until the JVM acknowledges successful decode).
sunchao
left a comment
sunchao
left a comment
There was a problem hiding this comment.
[P2] Adding evidence to the existing JVM handoff thread: a valid 864,547-byte manifest from 4,096 real Parquet files decoded successfully with more heap available, but the same Iceberg 1.11 decoder failed under controlled heap pressure. In a 128 MiB standalone JVM, the low-headroom run had about 6.7 MiB free before decoding and raised OutOfMemoryError inside the actual Avro reader. All task files remained.
At this boundary, drainAvroPayload has already released the native plan, while the new failure listener is not yet registered. Could cleanup ownership remain armed until decoding and listener registration succeed? A component ownership model retaining the exact native guard across that decoder failure subsequently deleted all 4,096 files. This was isolated component validation, not a full Spark/JNI run. The decoder behavior predates this PR. The concern is the uncovered handoff in this cleanup fix, not a newly introduced decoder regression.
Current CI has 53 successful, 12 running and 7 skipped checks. I did not run the full Comet or Spark suites.
3 participants
Which issue does this PR close?
Closes #5618.
Rationale for this change
When a native Iceberg write task fails partway through, the data files it had already finalized stay in the table's data location. They are invisible to readers and
remove_orphan_fileseventually reclaims them, but iceberg-java deletes them synchronously (DataWriter.abort()callsSparkCleanupUtil.deleteTaskFiles), so on spot-heavy or preemption-prone clusters the native path silently accumulates orphans that the JVM path does not. This is the first phase-2 item of the native Iceberg writes epic (#5649).What changes are included in this PR?
Two halves, matching the two places a task can fail.
Inside the native writer. iceberg-rust's writers keep the
DataFiles they have finalized private untilcloseand have no abort hook, so the task cannot ask a failed writer what it wrote. Instead, theRollingFileWriterBuilderis given aTrackingLocationGenerator, a wrapper aroundDefaultLocationGeneratorthat records every location it hands to a file writer. If the input stream, a write, or the final close fails,run_write_taskdeletes every recorded location through the task'sFileIObefore propagating the original error.That explicit path is not enough on its own, and the new end-to-end test proved it: when the JVM-side input iterator throws (a UDF failure upstream of the write, the most common shape),
executePlanreturns the error straight from its JNI batch pull and the JVM releases the plan, so the write task's future is dropped without ever seeing an error. AnAbortOnDropguard covers that case. It is disarmed once the task completes; if it is dropped while still armed it deletes the tracked files, synchronously on a throwaway current-thread runtime when dropped from a plain JVM thread (releasePlan), or spawned onto the current runtime when dropped from inside one. Deletion is best-effort in both paths: failures are logged, never returned, so the task failure Spark reports is still the real one. The location that was open at the time of the failure is included; deleting a path that was never materialized is a no-op.After the native writer has returned. Once the manifest is decoded, the JVM knows the files.
CometIcebergWriteExecregisters aTaskFailureListenerthat deletes them through the tableFileIOalready in the task closure if the metrics rebuild,TaskCommitconstruction, or serialization fails.IcebergReflection.deleteFilesQuietlyprefersSupportsBulkOperations.deleteFilesand falls back toFileIO.deleteFile(String)per path, and likewise never throws.The "Failure handling" section of
iceberg-writes.mdis updated to describe the cleanup and drop the reference to this issue.How are these changes tested?
FileIOincluding a recorded location that was never written, and for the drop guard (disarmed guard deletes nothing; armed guard dropped outside a runtime deletes synchronously).CometIcebergWriteActionSuite: a failure-injection test writes a single-task source with a two-row Comet batch size into a table with a one-bytewrite.target-file-size-bytes, so the rolling writer finalizes a file per batch, and a UDF throws on the seventh row. A control run with the same source and settings first proves the writer rolls into several files; the failing run then asserts the write planned natively (CometIcebergWriteExecin the failed plan), no snapshot was created, the pre-existing data file is untouched, and no other parquet file remains under the table's data location.deleteFilesQuietlythrough a real table'sHadoopFileIO: written files are removed, a nonexistent path is tolerated, and a second call is a no-op.