Fix: AddFiles() should respect commit.manifest-merge.enabled#806
Open
RSP22 wants to merge 3 commits intoapache:mainfrom
Open
Fix: AddFiles() should respect commit.manifest-merge.enabled#806RSP22 wants to merge 3 commits intoapache:mainfrom
RSP22 wants to merge 3 commits intoapache:mainfrom
Conversation
Author
|
Added
Happy to move it into |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #805
AddFiles()hardcodes.fastAppend()at line 780, bypassingappendSnapshotProducer()which reads
commit.manifest-merge.enabled. Setting this property has no effect whenusing
AddFiles()— manifests accumulate one-per-commit indefinitely regardless of theproperty value.
Root Cause
appendSnapshotProducer()already exists at line 126 and correctly reads the property.It is used by
AddDataFiles()andAppend()but was missed inAddFiles():appendSnapshotProducer()AddDataFiles()Append()AddFiles().fastAppend()— this bugJava Reference
BaseTable.newAppend()always returnsMergeAppend.MANIFEST_MERGE_ENABLED_DEFAULT = truein the Java implementation. The Go implementation should behave consistently with the
reference implementation.
Test Gap
Three existing tests set
commit.manifest-merge.enabled=truebut none callAddFiles():AddFiles()?TestCommitV3RowLineageDeltaIncludesExistingRowssnapshot_producers_test.gonewMergeAppendFilesProducer()directlyTestMergeManifeststable_test.goAppendTable()→Append()TestSetPropertiestransaction_test.goThe new regression test
table/addfiles_merge_regression_test.goexercisesAddFiles()directly with
commit.manifest-merge.enabled=trueand verifies the merge fires.It can be moved into
transaction_test.goif preferred by reviewers.Changes
table/transaction.go— 1-line fix: useappendSnapshotProducer()instead of hardcoded.fastAppend()table/addfiles_merge_regression_test.go— regression test proving the bug (fails before fix, passes after)Verification
Regression test:
AddFiles()commits withminCountToMerge=2Production verification on a table with 1 commit/min:
Note on AI Assistance
This PR was developed with AI assistance (Claude). The author has verified the
implementation, Java reference behavior, test coverage gap, and production impact end-to-end.