Diff speedup #2598
Merged
Diff speedup #2598
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MichaelEischer
force-pushed
the
diff-speedup
branch
2 times, most recently
from
d87a569
to
7eee19e
Compare
|
Confirmed, this gives a big speedup in a quick test on two ~120GB snapshots with nearly equal data. Master: This PR: |
MichaelEischer
force-pushed
the
diff-speedup
branch
from
7eee19e
to
0d0a89a
Compare
MichaelEischer
force-pushed
the
diff-speedup
branch
from
cb8ce59
to
89f65bc
Compare
MichaelEischer
force-pushed
the
diff-speedup
branch
from
89f65bc
to
d8d71f8
Compare
When the diff calculation compares two trees with identical id then no differences between them can ever show up. Optimize for that case by simply traversing the tree only once to collect all referenced blobs for a proper calculation of added and removed blobs. Just skipping the common subtrees is not possible as this would skew the results if the added or removed blobs are shared with one of the subtrees.
MichaelEischer
force-pushed
the
diff-speedup
branch
from
d8d71f8
to
4f221c4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of this change? What does it change?
Optimize diff calculation for shared subtrees.
When the diff calculation compares two trees with identical id then no
differences between them can ever show up. Optimize for that case by
simply traversing the tree only once to collect all referenced blobs for
a proper calculation of added and removed blobs.
Just skipping the common subtrees is not possible as this would skew the
results if the added or removed blobs are shared with one of the
subtrees.
This change can speed-up the diff calculation by up to a factor two. I'm not sure whether that's worth a changelog entry or not.
Was the change discussed in an issue or in the forum before?
No.
Checklist
[ ] I have added documentation for the changes (in the manual)changelog/unreleased/that describes the changes for our users (template here)gofmton the code in all commits