Pruning #101
Pruning #101
Conversation
b00ris
commented
Sep 30, 2019
b00ris
commented
- added async pruning
cmd/utils/flags.go
Outdated
| } | ||
| GCModeTickTimeout = cli.DurationFlag{ | ||
| Name: "gcmode.tick", | ||
| Usage: `Block to prune per tick"`, |
There was a problem hiding this comment.
Usage value needs to be changed
core/pruner.go
Outdated
|
|
||
| func batchDelete(db *bolt.DB, keys *keysToRemove) error { | ||
| log.Debug("Removed: ", "accounts", len(keys.Account), "storage", len(keys.Storage), "suffix", len(keys.Suffix)) | ||
| return db.Update(func(tx *bolt.Tx) error { |
There was a problem hiding this comment.
If this may be happening concurrently with other updates, we might consider using db.Batch() in both places (then boltdb will be able to merge concurrent updates sometimes). But the requirement is that all updates need to be idempotent (have the same effect if repeated multiple times)
core/pruner.go
Outdated
| "time" | ||
| ) | ||
|
|
||
| const NumOfPrunedBlocks = 10 |
core/pruner.go
Outdated
| if !ok { | ||
| return errors.New("it's not ethdb.BoltDatabase") | ||
| } | ||
| if p.config.BlocksToPrune == 0 || p.config.PruneTimeout.Seconds() < 1 { |
There was a problem hiding this comment.
Sanitise this at configuration stage
core/pruner.go
Outdated
|
|
||
| db ethdb.Database | ||
| chain BlockChainer | ||
| LastPrunedBlockNum *big.Int |
There was a problem hiding this comment.
Can use uint64 and atomic, to get rid of mutex
There was a problem hiding this comment.
As we discussed, this does not even need to be protected, it is only used in one goroutine
core/pruner.go
Outdated
| continue | ||
| } | ||
| p.RLock() | ||
| numOfBlocks := calculateNumOfPrunedBlocks(cb.Number().Uint64(), p.LastPrunedBlockNum.Uint64(), p.config.BlocksBeforePruning, p.config.BlocksToPrune) |
There was a problem hiding this comment.
That function can compute from and to
core/pruner.go
Outdated
|
|
||
| changedKeys := dbutils.Suffix(v) | ||
|
|
||
| err := changedKeys.Walk(func(addrHash []byte) error { |
There was a problem hiding this comment.
addrHash here could mean not just address hash, but also concatenation of address hash and the key hash, right? Consider renaming it for clarity
core/pruner.go
Outdated
| err := changedKeys.Walk(func(addrHash []byte) error { | ||
| compKey, _ := dbutils.CompositeKeySuffix(addrHash, timestamp) | ||
| ck := make([]byte, len(compKey)) | ||
| copy(ck, compKey) |
There was a problem hiding this comment.
If the function CompositeKeySuffix already allocates a new slice for the key, this is not needed, wasteful extra allocation. Please remove if so
|
|
||
| func Prune(db *ethdb.BoltDatabase, blockNumFrom uint64, blockNumTo uint64) error { | ||
| keysToRemove := newKeysToRemove() | ||
| err := db.Walk(dbutils.SuffixBucket, []byte{}, 0, func(key, v []byte) (b bool, e error) { |
There was a problem hiding this comment.
Please add the limit to how large the keysToRemove can become. For example, no more than 100k items. Otherwise, bolt transaction is going to be too large, and the whole program can crash with Out Of Memory
…ployer [Canyon Hard Fork] Deploy create2deployer Precompile
