Core: Fix for deleting files when commiting transactions with multiple branches #6634
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jackye1995
added this to In progress
in [Priority 2] Spec: Snapshot tagging and branching
via automation
amogh-jahagirdar
force-pushed
the
branch-txns-fix
branch
from
5c18843
to
4cf2ced
Compare
rdblue
reviewed
Jan 23, 2023
amogh-jahagirdar
force-pushed
the
branch-txns-fix
branch
2 times, most recently
from
e383f32
to
870de44
Compare
amogh-jahagirdar
force-pushed
the
branch-txns-fix
branch
from
870de44
to
3eb6f3c
Compare
amogh-jahagirdar
changed the title
amogh-jahagirdar
changed the title
rdblue
reviewed
Jan 23, 2023
rdblue
reviewed
Jan 23, 2023
|
@amogh-jahagirdar, this looks great! I think it's about ready to merge but there are a couple of naming nits. |
rdblue
approved these changes
Jan 23, 2023
[Priority 2] Spec: Snapshot tagging and branching
automation
moved this from In progress
to Reviewer approved
Jan 23, 2023
amogh-jahagirdar
force-pushed
the
branch-txns-fix
branch
from
3eb6f3c
to
6186cf5
Compare
|
Thanks @rdblue for the review! since the focus of this PR was properly determining the set of committed files when committing a transaction I think we should be good. Regardless I think we probably want some more coverage on transactions spanning multiple branches, I can raise that in a separate PR? |
jackye1995
approved these changes
Jan 23, 2023
[Priority 2] Spec: Snapshot tagging and branching
automation
moved this from Reviewer approved
to Done
Jan 23, 2023
|
Thanks, @amogh-jahagirdar! |
krvikash
pushed a commit
to krvikash/iceberg
that referenced
this pull request
Mar 16, 2023
bartash
added a commit
to bartash/iceberg
that referenced
this pull request
Nov 29, 2023
When a snapshot is expired as part of a transaction, the snapshot file(s) should be deleted when the transaction commits. A recent change (apache#6634) ensured that files are not deleted when they have also been committed as part of a transaction, but this breaks the simple case where no new files are committed. Fix this be not skipping deletion when the list of committed files is empty. TESTING: Extended a unit test to ensure that snapshot files are deleted. Ran the test without the fix on a branch where apache#6634 was reverted to show that this is a regression.
bartash
added a commit
to bartash/iceberg
that referenced
this pull request
Nov 30, 2023
When a snapshot is expired as part of a transaction, the manifest list file(s) should be deleted when the transaction commits. A recent change (apache#6634) ensured that these files are not deleted when they have also been committed as part of a transaction, but this breaks the simple case where no new files are committed. Fix this by not skipping deletion when the list of committed files is empty. TESTING: Extended a unit test to ensure that manifest list files are deleted. Ran the test without the fix on a branch where apache#6634 was reverted to show that this is a regression.
bartash
added a commit
to bartash/iceberg
that referenced
this pull request
Dec 4, 2023
When a snapshot is expired as part of a transaction, the manifest list file(s) should be deleted when the transaction commits. A recent change (apache#6634) ensured that these files are not deleted when they have also been committed as part of a transaction, but this breaks the simple case where no new files are committed. Fix this by not skipping deletion when the list of committed files is empty. TESTING: Extended a unit test to ensure that manifest list files are deleted. Ran the test without the fix on a branch where apache#6634 was reverted to show that this is a regression.
amogh-jahagirdar
pushed a commit
that referenced
this pull request
Dec 4, 2023
When a snapshot is expired as part of a transaction, the manifest list file(s) should be deleted when the transaction commits. A recent change (#6634) ensured that these files are not deleted when they have also been committed as part of a transaction, but this breaks the simple case where no new files are committed. Fix this by not skipping deletion when the list of committed files is empty. TESTING: Extended a unit test to ensure that manifest list files are deleted. Ran the test without the fix on a branch where #6634 was reverted to show that this is a regression.
bartash
added a commit
to bartash/iceberg
that referenced
this pull request
Dec 5, 2023
…che#9183) When a snapshot is expired as part of a transaction, the manifest list file(s) should be deleted when the transaction commits. A recent change (apache#6634) ensured that these files are not deleted when they have also been committed as part of a transaction, but this breaks the simple case where no new files are committed. Fix this by not skipping deletion when the list of committed files is empty. TESTING: Extended a unit test to ensure that manifest list files are deleted. Ran the test without the fix on a branch where apache#6634 was reverted to show that this is a regression.
amogh-jahagirdar
added a commit
that referenced
this pull request
Dec 19, 2023
…ed (#9223) * Core: Expired Snapshot files in a transaction should be deleted. (#9183) When a snapshot is expired as part of a transaction, the manifest list file(s) should be deleted when the transaction commits. A recent change (#6634) ensured that these files are not deleted when they have also been committed as part of a transaction, but this breaks the simple case where no new files are committed. Fix this by not skipping deletion when the list of committed files is empty. * Core: Fix logic for determining set of committed files in BaseTransaction when there are no new snapshots (#9221) --------- Co-authored-by: Amogh Jahagirdar <amogh@tabular.io>
lisirrx
pushed a commit
to lisirrx/iceberg
that referenced
this pull request
Jan 4, 2024
…che#9183) When a snapshot is expired as part of a transaction, the manifest list file(s) should be deleted when the transaction commits. A recent change (apache#6634) ensured that these files are not deleted when they have also been committed as part of a transaction, but this breaks the simple case where no new files are committed. Fix this by not skipping deletion when the list of committed files is empty. TESTING: Extended a unit test to ensure that manifest list files are deleted. Ran the test without the fix on a branch where apache#6634 was reverted to show that this is a regression.
devangjhabakh
pushed a commit
to cdouglas/iceberg
that referenced
this pull request
Apr 22, 2024
…che#9183) When a snapshot is expired as part of a transaction, the manifest list file(s) should be deleted when the transaction commits. A recent change (apache#6634) ensured that these files are not deleted when they have also been committed as part of a transaction, but this breaks the simple case where no new files are committed. Fix this by not skipping deletion when the list of committed files is empty. TESTING: Extended a unit test to ensure that manifest list files are deleted. Ran the test without the fix on a branch where apache#6634 was reverted to show that this is a regression.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix for #6632
The way intermediate snapshots in transactions are tracked is incorrect in the presence of branching.
Moving to draft as I read more of the code to make sure this handles different failure cases properly and will add tests if determined this is the right way to fix this issue.