Skip to content

GH-3706: Mark writer aborted on end() failure to avoid flushing incomplete files - #3707

Open
arg3t wants to merge 1 commit into
apache:masterfrom
arg3t:footer-corruption-on-write-fix
Open

GH-3706: Mark writer aborted on end() failure to avoid flushing incomplete files#3707
arg3t wants to merge 1 commit into
apache:masterfrom
arg3t:footer-corruption-on-write-fix

Conversation

@arg3t

@arg3t arg3t commented Aug 4, 2026

Copy link
Copy Markdown

Rationale for this change

Makes ParquetFileWriter#end() mark the writer aborted before flushing, so an incomplete file is never committed on failure. Previously close() ran before the catch block in withAbortOnFailure and withAbortOnFailure only caught IOException, causing:

  • an incomplete file being flushed to storage when serializing indexes/stats/footer throws any exception
  • a RuntimeException (e.g. OOM) to be missed, leaving the writer un-aborted and still flushing

What changes are included in this PR?

  • withAbortOnFailure catches Throwable (not just IOException) and marks the writer aborted.
  • end() runs close() in an outer finally, so abort happens before close() and the flush is skipped on failure.

Are these changes tested?

Yes. 2 new tests in TestParquetFileWriter inject an IOException/RuntimeException during end() and assert the stream is never flushed.

Are there any user-facing changes?

No. Successful writes are unchanged; only the failure path is affected.

Closes #3706

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ParquetFileWriter.end() flushes an incomplete file to storage when finalizing fails

1 participant