Spark: Broadcast slim rewritable delete metadata in position delta wr…#17281
Open
Neuw84 wants to merge 1 commit into
Open
Spark: Broadcast slim rewritable delete metadata in position delta wr…#17281Neuw84 wants to merge 1 commit into
Neuw84 wants to merge 1 commit into
Conversation
Neuw84
force-pushed
the
slim-rewritable-deletes-broadcast
branch
from
July 17, 2026 10:50
da52fda to
fa52135
Compare
3 tasks
…ites Applies to Spark 3.5, 4.0 and 4.1. Position delta writes (MERGE/UPDATE/DELETE in merge-on-read mode) broadcast a map of every scanned data file with position deletes to its full DeleteFile metadata. Executors only need enough of it to locate and read the previous delete content (location, format, sizes, key metadata, DV offsets) and to identify files back to the driver; partition data, column bounds and counts ride along only to be echoed back for RowDelta.removeDeletes at commit time. Under streaming upserts against v3 DV tables this map converges to one entry per data file in the table and is rebuilt and re-broadcast every micro-batch, adding driver and executor memory pressure that grows with table age. Keep the full map driver-local, broadcast slim copies built via FileMetadata.deleteFileBuilder (with referencedDataFile normalized so file-scoped checks keep working without the file_path column bounds), and resolve reported rewritten files back to their full metadata at commit time before calling removeDeletes. The resolution step is required for correctness of the slimming: manifest filtering uses the removed file's spec ID and partition, which slim copies do not carry, so reported rewritten files must never reach the RowDelta directly. Measured serialized size of the broadcast map (1000 entries, Java serialization - Spark's default broadcast serializer): map contents full slim factor DV entries (v3) 661 B/entry 466 B/entry 1.4x v2 file-scoped entries with stats 1069 B/entry 581 B/entry 1.8x The serialized win is bounded by what slimming cannot remove (the data file location key and the referenced data file location dominate each entry); the deserialized heap saving per executor is larger because the per-entry object graph (partition data, stats maps, split offsets) is gone. Commit messages shrink correspondingly, since executors echo the slim copies back as rewritten files.
Neuw84
force-pushed
the
slim-rewritable-deletes-broadcast
branch
from
July 17, 2026 11:05
fa52135 to
ca1899d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Position delta writes (MERGE/UPDATE/DELETE in merge-on-read mode) broadcast a map of every scanned data file with position deletes to its full DeleteFile metadata.
Executors only need enough of it to locate and read the previous delete content (location, format, sizes, key
metadata, DV offsets) and to identify files back to the driver; partition data, column bounds and counts ride along only to be echoed back for RowDelta.removeDeletes at commit time.
Under streaming upserts against v3 DV tables this map converges to one entry per data file in the table and is rebuilt and re-broadcast every micro-batch, adding driver and executor memory pressure that grows with
table age.
Keep the full map driver-local, broadcast slim copies built via FileMetadata.deleteFileBuilder (with referencedDataFile normalized so file-scoped checks keep working without the file_path column bounds),
and resolve reported rewritten files back to their full metadata at commit time before calling removeDeletes. The resolution step is required for correctness of the slimming: manifest filtering uses the
removed file's spec ID and partition, which slim copies do not carry, so reported rewritten files must never reach the RowDelta directly.
Measured serialized size of the broadcast map (1000 entries, Java serialization - Spark's default broadcast serializer):
The serialized win is bounded by what slimming cannot remove (the data file location key and the referenced data file location dominate each entry); the deserialized heap saving per executor is larger because the
per-entry object graph (partition data, stats maps, split offsets) is gone. Commit messages shrink correspondingly, since executors echo the slim copies back as rewritten files.
The code can be easily backported to Spark 3.4.
AI Disclosure
Model: Claude Fable
Platform/Tool: Kiro
Human Oversight: reviewed
Relates to #17241