Make moving the pruning point faster #1660
Conversation
…r pruning point utxoset
…the old pruning utxo set
elichai
force-pushed
the
optimize-pruning
branch
from
d8d83d7
to
5ec128b
Compare
Codecov Report
@@ Coverage Diff @@
## v0.10.0-dev #1660 +/- ##
===============================================
- Coverage 59.54% 59.47% -0.08%
===============================================
Files 548 548
Lines 21695 21789 +94
===============================================
+ Hits 12919 12958 +39
- Misses 6726 6769 +43
- Partials 2050 2062 +12
Continue to review full report at Codecov.
|
| @@ -9,6 +9,7 @@ type pruningStagingShard struct { | |||
| store *pruningStore | |||
|
|
|||
| newPruningPoint *externalapi.DomainHash | |||
| oldPruningPoint *externalapi.DomainHash | |||
There was a problem hiding this comment.
Please rename newPruningPoint to currentPruningPoint and oldPruningPoint to previousPruningPoint here and everywhere else.
| // Insert all the new UTXOs into the database | ||
| for ok := utxoSetIterator.First(); ok; ok = utxoSetIterator.Next() { | ||
| outpoint, entry, err := utxoSetIterator.Get() | ||
| // Delete all the old UTXOs from the database |
There was a problem hiding this comment.
Please remove this comment. It's a duplicate of the one in line 90.
| toAddIterator := diff.ToAdd().Iterator() | ||
| defer toAddIterator.Close() | ||
| for ok := toAddIterator.First(); ok; ok = toAddIterator.Next() { | ||
| toRemoveOutpoint, entry, err := toAddIterator.Get() |
There was a problem hiding this comment.
Please rename toRemoveOutpoint to toAddOutpoint.
| @@ -535,28 +615,43 @@ func (pm *pruningManager) UpdatePruningPointUTXOSetIfRequired() error { | |||
| return nil | |||
| } | |||
|
|
|||
| func (pm *pruningManager) updatePruningPointUTXOSet(stagingArea *model.StagingArea) error { | |||
| func (pm *pruningManager) updatePruningPointUTXOSet() error { | |||
There was a problem hiding this comment.
It appears that now you're also deleting data within this function. Please update its name accordingly.
There was a problem hiding this comment.
This function always removed and added data (removed the old pruning point UTXO set and added the new pruning UTXO set)
There was a problem hiding this comment.
I meant deleting block (and other) data. As in, the actual pruning.
| err := pm.validateUTXOSetFitsCommitment(stagingArea, pruningPointHash) | ||
| if err != nil { | ||
| oldPruningPoint, err := pm.pruningStore.PruningPoint(pm.databaseContext, stagingArea) | ||
| if err != nil && !errors.Is(err, database.ErrNotFound) { |
There was a problem hiding this comment.
I'd prefer to handle the genesis block explicitly instead of checking whether the pruning point happens to exist in the database.
| return nil, err | ||
| } | ||
|
|
||
| // do the quick restore. |
There was a problem hiding this comment.
Did you mean to delete this?
| @@ -468,6 +466,88 @@ func (pm *pruningManager) validateUTXOSetFitsCommitment(stagingArea *model.Stagi | |||
| return nil | |||
| } | |||
|
|
|||
| func (pm *pruningManager) calculateDiffBetweenOldAndNewPruningPoints(stagingArea *model.StagingArea, newPruningHash *externalapi.DomainHash) (externalapi.UTXODiff, error) { | |||
There was a problem hiding this comment.
Add a comment explaining how this function works.
| oldPruningCurrentDiffChildBlueWork := oldPruningGhostDAG.BlueWork() | ||
|
|
||
| var oldDiffs []externalapi.UTXODiff | ||
| var newDiffs []externalapi.UTXODiff |
There was a problem hiding this comment.
Please rename the above to previousPruningPointUTXODiffs and currentPruningPointUTXODiffs.
| } | ||
| } | ||
| oldDiff := utxo.NewMutableUTXODiff() | ||
| for i := len(oldDiffs) - 1; i >= 0; i-- { |
There was a problem hiding this comment.
Please explain in a comment why these diffs need to be applied in reverse order.
elichai
force-pushed
the
optimize-pruning
branch
from
955b6d6
to
6c9c77e
Compare
Related to #1629, this makes moving the pruning point much faster (2-4 times faster)
I kept the commitment validation for now, but if everything works fine in ~2 weeks, then I think we can remove it to improve performance.
This should also help us do this in a separate thread, as it no longer has anything to do with the virtual.