Split ApplyPruningPointProof to multiple small database transactions #1937
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## dev #1937 +/- ##
==========================================
- Coverage 61.26% 61.24% -0.03%
==========================================
Files 651 651
Lines 31114 31112 -2
==========================================
- Hits 19063 19055 -8
- Misses 9276 9277 +1
- Partials 2775 2780 +5
Continue to review full report at Codecov.
|
someone235
requested changes
Jan 31, 2022
| @@ -607,13 +608,15 @@ func (ppm *pruningProofManager) dagProcesses( | |||
| return reachabilityManagers, dagTopologyManagers, ghostdagManagers | |||
| } | |||
|
|
|||
| func (ppm *pruningProofManager) ApplyPruningPointProof(stagingArea *model.StagingArea, pruningPointProof *externalapi.PruningPointProof) error { | |||
| func (ppm *pruningProofManager) ApplyPruningPointProof(pruningPointProof *externalapi.PruningPointProof) error { | |||
There was a problem hiding this comment.
Can you add a comment that this function is not atomic and if it fails the outside scope should probably delete the database?
…oof-database-transaction
someone235
approved these changes
Feb 6, 2022
someone235
added a commit
that referenced
this pull request
Feb 18, 2022
* Make kaspawallet store the utxos sorted by amount, so that the bigger utxos are spent first - making it less likely a compound will be required * Start refactor addEntryToUTXOSet * Add GetUTXOsByBalances command to rpc * Store list of addresses, updated with the collectAddresses methods (replacing collectUTXOs methods) * Fix wrong commands in GetBalanceByAddress * Rename: refreshExistingUTXOs -> refreshUTXOs * Display progress of IBD process in Kaspad logs (#1938) * Report progress percentage when downloading headers in IBD. * Extract reporting logic to a separate type. * Report progress for IBD missing block bodies. * Split ApplyPruningPointProof to multiple small database transactions (#1937) * Split ApplyPruningPointProof to multiple small database transactions. * Increase the timeout duration in TestIBDWithPruning. * Increase the timeout duration in simple-sync. * Explain that if ApplyPruningPointProof fails, the database must be discarded. * Increase headers proof timeout and add more progress logs (#1939) * Increase timeout for pruning proof and add some logs * Show resolving virtual progress as whole percents * Add AllowConnectionToDifferentVersions flag to kaspactl (#1940) * Remove duplicate median time calculation on tx validation (#1943) Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com> * Drop support for p2p v3 (#1942) * Drop support for p2p v3 * Remove redundant aliases * Remove redundant condition * In profile dump file - use a time format supporting all file systems (#1945) * Use a time format without ":" to support all file systems * go fmt * Bump version to v0.11.12 (#1941) Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com> Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com> Co-authored-by: Ori Newman <orinewman1@gmail.com> Co-authored-by: Michael Sutton <mikisiton2@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes #1920
ApplyPruningPointProof is always called on the Staging Consensus during IBD, so there's no risk of data inconsistency.
This PR splits the writes from this function to the database to multiple small transactions, so that the garbage collector has a chance to clean up while data is being written
Note that it's still a big, quick write, so RAM consumption isn't reduced to zero. On my VM I've watched it consume up to 1gb additional memory throughout the process