[R4R]offline block prune #543

Mercybudda · 2021-11-14T09:54:44Z

Description

This is for the off-line block prune, in bsc storage, there's db wrapper of ancientDb(freezer) and kvDb(levelDB), the space was mainly occupied by ancientDb, the purpose of this is to prune and remain the specific number of old blocks in ancientDb by specifying the amount of blocks you'd like to remain after pruning in the CLI flag. Then the pruner will prune and only remain this specified amount backward, this first block is not necessarily the genesis block because we may prune many rounds. Actually we can prune and remain any amount of blocks as long as the amount specified did exceed the number of blocks inside ancientDb
###Example:
Assume all blocks in ancientDb are [0,1,... 1000], if pruning amount of 500 blocks, then [0,...499] will be pruned, the data in ancientDb will be [500, 1000]. The next round for pruning will start with BlockNumber 500.

Brief Coding workflow is as follows

Create a new ancientDb without a kvdb wrapper, it will create a new folder ./node/geth/chaindata/ancient_backup for initializing new ancientDb.
Back-up all the blocks at and after oldOffSet + frozen - BlockPruneAmountLeft from old ancientDb into new ancientBackup. In this process, pruner will read one by one for block, receipts, difficulty from old ancientDb and write to new backup ancientDb. To be noted that there’ll be a new offset of new ancientDb, this offset will be recorded into kvdb.
Delete directly for the old folder./node/geth /chaindata/ancient/, i.e. the old ancientDb.
Rename the new folder of ./node/geth/chaindata/ancient_backup into ./node/geth/chaindata/ancient, then assemble with the same kvdb of the original DB wrapper in which old ancientDb resides.
Finally the ancientDb in DB wrapper is replaced successfully, all the block data we want to reserve were in this new ancientDb.

Self Test

Unit test: Initialized a blockchain of 500 blocks, testing the pruning of 328 amount of blocks remaining. passed
Locally geth test, start a full node: ./geth --config ./config.toml --datadir ./node --cache 12000 --rpc.allow-unprotected-txs --txlookuplimit 0 --syncmode full --usb, after a while until there's enough old block for pruning in ancientDb.
1st round of prune test: Start the prune process: ./geth snapshot prune-block --datadir ./node --datadir.ancient ./chaindata/ancient --block-amount-reserved 10000. This will prune and reserve 10000 blocks.
Passed and get a new ancientDb with amount of 10000 blocks
2nd round of prune test, start the geth again, repeat the step 2 & 3.
Introduced the flag inspect-reserved-oldest-blocks, can be used for checking information of ancientStore like offset and reserved items amount after pruning done.
./geth db inspect-reserved-oldest-blocks --datadir ./node
+--------------------------------+--------------------------------+--------+
| DATABASE | CATEGORY | ITEMS |
+--------------------------------+--------------------------------+--------+
| Offset/StartBlockNumber | Offset/StartBlockNumber after | 302672 |
| | BlockPrune | |
| Amount of remained items in | remaining items after | 10000 |
| AncientStore | BlockPrune | |
+--------------------------------+--------------------------------+--------+
| ANCIENTSTORE INFORMATION AFTER |
| OFFLINE BLOCKPRUNE |
+--------------------------------+--------------------------------+--------+

Running command

e.g.
./geth snapshot prune-block --datadir ./node --datadir.ancient ./chaindata/ancient --block-amount-reserved 9000 --triesInMemory 32 --check-snapshot-with-mpt
--block-amount-reserved: Amount of blocks want to reserve.
--triesInMemory(optional): The layer of tries trees that keep in memory.
--check-snapshot-with-mpt(optinal): Indicate if enable/disable checking for the match between MPT and snapshot.

Check the info of ancientDB
./geth db inspect-reserved-oldest-blocks --datadir ./node

Result

The offset of ancientDb changed as expected.
Geth client restarted successfully.
new block data freeze() moving from kvDb to ancientDb as expected.

Preflight checks

build passed (make build)
tests passed (make test)
manual transaction test passed

cmd/geth/snapshot.go

unclezoro · 2021-12-20T04:04:30Z

cmd/geth/snapshot.go

+				},
+				Description: `
+Offline prune for block data.
+For AncientFlag, please specify the absolute path of node/geth/chaindata.


Why it have to be absolute path?

unclezoro · 2021-12-20T04:06:51Z

cmd/geth/snapshot.go

+	return nil
+}
+
+func getAncientPath(ctx *cli.Context) string {


let us follow the same code style. Please remove this func, but use ctx.GlobalString(utils.AncientFlag.Name) directly.

unclezoro · 2021-12-20T04:07:59Z

cmd/geth/snapshot.go

+	defer stack.Close()
+
+	chaindb := utils.MakeChainDatabaseForBlockPrune(ctx, stack, false)
+	chaindb.Close()


Are you meaning defer chaindb.Close()?

No, this is for closing it on purpose, because in the later code, there'll be many times of open/close the same db, crash will occur if did not close it.

unclezoro · 2021-12-20T04:08:57Z

cmd/geth/snapshot.go

+		oldAncientPath = path + "/ancient"
+		newAncientPath = path + "/ancient_back"
+	} else {
+		utils.Fatalf("Prune failed, did not specify the AncientPath %v")


Better have a default ancient path.

You have %v here , but no arg.

unclezoro · 2021-12-20T04:09:32Z

cmd/geth/snapshot.go

+		return errors.New("Prune failed, did not specify the AncientPath")
+	}
+
+	//TODO for filelock


Will we have file lock?

unclezoro · 2021-12-20T04:11:09Z

cmd/geth/snapshot.go

+	// if err != nil {
+	// 	utils.Fatalf("Failed to read genesis file: %v", err)
+	// }
+	for _, name := range []string{"chaindata"} {


why range a slice that only have one const item?

unclezoro · 2021-12-20T04:12:27Z

cmd/geth/snapshot.go

+		root := stack.ResolvePath(name)
+		switch {
+		case oldAncientPath == "":
+			oldAncientPath = filepath.Join(root, "ancient")


It is not possible to happen, since you forbidden it to be empty in previous logic.

unclezoro · 2021-12-20T04:14:57Z

cmd/geth/snapshot.go

+		}
+		pruner, err := pruner.NewBlockPruner(chaindb, stack, oldAncientPath)
+		if err != nil {
+			utils.Fatalf("Failed to create block pruner", err)


Get err arg here, but no %v

unclezoro · 2021-12-20T04:16:26Z

cmd/geth/snapshot.go

+	log.Info("geth block offline pruning backup successfully")
+
+	//After backing up successfully, rename the new ancientdb name to the original one, and delete the old ancientdb
+	if err := pruner.BlockPrune(oldAncientPath, newAncientPath); err != nil {


It is quite weried you already pass oldAncientPath when NewBlockPruner, but pass it again when doing BlockPruneBackUp, it is duplicated.

unclezoro · 2021-12-20T05:35:50Z

cmd/utils/flags.go

+		err     error
+		chainDb ethdb.Database
+	)
+	if ctx.GlobalString(SyncModeFlag.Name) == "light" {


if sync mode is light, there is no need to do block prune

unclezoro · 2021-12-20T05:42:43Z

node/node.go

+// OpenDatabaseWithFreezerForPruneBlock opens an existing database with the given name (or
+// creates one if no previous can be found) from within the node's data directory,
+// also attaching a chain freezer to it. If the node is an ephemeral one, a memory database is returned.
+func (n *Node) OpenDatabaseWithFreezerForPruneBlock(name string, cache, handles int, freezer, namespace string, readonly bool) (ethdb.Database, error) {


Please try not to invent new function, too many duplicated code.

These new functions were the function without freeze go-routine running, to avoid the case that freeze go-routine did not stop while db closing.

unclezoro · 2021-12-20T05:43:32Z

node/node.go

+// creates one if no previous can be found) from within the node's data directory,
+// also attaching a chain freezer to it. If the node is an ephemeral one, a
+// memory database is returned.
+func (n *Node) OpenDatabaseWithFreezerBackup(offset uint64, name string, cache, handles int, freezer, namespace string, readonly bool) (ethdb.Database, error) {


unclezoro · 2021-12-20T05:52:24Z

core/state/pruner/pruner.go

+		return errors.New("can't access the freezer or it's empty, abort")
+	}
+	// Get the actual start block number.
+	startBlockNumber := frozen - 128 + oldOffSet


128 is a magic number here, please use constant value.

* offline block prune * update * update * update and add unit test * addressed comments from walt * Addressed comments from walt and Igor * ensure MPT and snapshot matched * add one more parameter to indicate blockprune * update the logic of creating freezerDb * update flag command description * expose the function for db inspect the offset/startBlockNumber * add flags to inspect prune info * rename flag of reserved-recent-blocks to block-amount-reserved * addressed comments from walt * handle the case of command interruption * refined goimports * addressed comments from walt * change the logic as restarting prune after interruption * addressed comments * reclaimed freezer logic * introduce flag to enable/disable check between MPT and snapshot * update the logic of frozen field in freezerDB * update the code in all places related to freezer change * addressed comments from dylan * update the logic for backup block difficulty * addressed comments from dylan

Signed-off-by: Delweng <delweng@gmail.com>

Signed-off-by: jsvisa <delweng@gmail.com>

Signed-off-by: Delweng <delweng@gmail.com>

* core/state: typo Signed-off-by: Delweng <delweng@gmail.com> * core/rawdb: backport from bnb-chain/bsc#543 Signed-off-by: Delweng <delweng@gmail.com> * eth,ethdb,node,core/state: backport from bnb-chain/bsc#543 Signed-off-by: Delweng <delweng@gmail.com> * eth,core: backport from bnb-chain/bsc#543 Signed-off-by: Delweng <delweng@gmail.com> * cmd: open db with freeze disabled Signed-off-by: Delweng <delweng@gmail.com> * cli: snapshot prune-block Signed-off-by: Delweng <delweng@gmail.com> * fix typo Signed-off-by: Delweng <delweng@gmail.com> * cli/snapshot: fix the issue of dup open db error Signed-off-by: Delweng <delweng@gmail.com> * cli/snapshot: resolve datadir and ancient before backup Signed-off-by: Delweng <delweng@gmail.com> * core: more prune-block log Signed-off-by: Delweng <delweng@gmail.com> * core: truncatetail missing f.offset Signed-off-by: Delweng <delweng@gmail.com> * core/rawdb: indextx adjust offset of pruned block Signed-off-by: Delweng <delweng@gmail.com> * core/rawdb: freezer batch should implement the offset commit, ref bnb-chain/bsc#1005 Signed-off-by: Delweng <delweng@gmail.com> * core: check of ancientdb, backport bnb-chain/bsc#817 Signed-off-by: Delweng <delweng@gmail.com> * core/state: read raw borReceipt to backup Signed-off-by: Delweng <delweng@gmail.com> * core/rawdb: bor receipt maybe in []Receipt or Receipt RLP format Signed-off-by: Delweng <delweng@gmail.com> * core/state: typo and error msg Signed-off-by: Delweng <delweng@gmail.com> * core/rawdb: offSet -> offset Signed-off-by: Delweng <delweng@gmail.com> * cli/snapshot: comment Signed-off-by: Delweng <delweng@gmail.com> * cli/snapshot: add prune-block doc Signed-off-by: Delweng <delweng@gmail.com> * docs: add prune-block document Signed-off-by: Delweng <delweng@gmail.com> * core/rawdb: print wrong bor-receipt length Signed-off-by: Delweng <delweng@gmail.com> * internal/cli: add snapshot prune block tests (referenced from bsc's PR) * improve errors * cmd, core, eth, internal: fix lint * internal/cli: refactor snapshot prune block test * fix linters in tests * internal/cli: add inspect-ancient-db command, update docs * pruner: use a generic function for simplification * internal/cli: fixes for inspect-db command * internal/cli: improve pruning tests * core/rawdb: update end block calculation logic in inspect command * core/rawdb: improve checks db initialisation * core/rawdb: remove offset check * update mocks for span, ethdb and add command in makefile * docs/cli: update docs with inspect command * go mod tidy * refactor and resolve conflicts * resolve more conflicts * refactor * explicitly read node for hash scheme * add check for hash scheme, fix tests * fix typo * update docs and add warning * raise error if pbss is enabled * revert read raw bor receipt change * consensus/bor: handle nil header case in get root hash * address comments * core/rawdb: check chain continuity by matching parent hash * core/rawdb: account for pruned ancient blocks * go mod tidy * fix tests * fix tests --------- Signed-off-by: Delweng <delweng@gmail.com> Co-authored-by: Delweng <delweng@gmail.com>

Mercybudda force-pushed the blockprune branch 2 times, most recently from 306e6c7 to 2b0d305 Compare December 9, 2021 17:33

Mercybudda requested review from unclezoro and RenRick December 10, 2021 02:49

Mercybudda force-pushed the blockprune branch 2 times, most recently from e33105f to 4046f25 Compare December 10, 2021 06:44

unclezoro reviewed Dec 10, 2021

View reviewed changes

cmd/geth/snapshot.go Show resolved Hide resolved

Mercybudda force-pushed the blockprune branch 3 times, most recently from 4fe76ff to cad935c Compare December 17, 2021 08:00

Mercybudda requested review from j75689, yutianwu and KeefeL December 17, 2021 08:05

Mercybudda force-pushed the blockprune branch from cad935c to fa8de66 Compare December 19, 2021 17:36

unclezoro changed the base branch from master to develop December 20, 2021 02:21

unclezoro changed the base branch from develop to master December 20, 2021 04:02

unclezoro reviewed Dec 20, 2021

View reviewed changes

cmd/geth/snapshot.go Outdated Show resolved Hide resolved

unclezoro reviewed Dec 20, 2021

View reviewed changes

addressed comments from dylan

37f2e89

j75689 approved these changes Jan 19, 2022

View reviewed changes

unclezoro merged commit 476d520 into develop Jan 19, 2022

This was referenced Jan 26, 2022

[R4R]prepare for relase v1.1.8 #741

Merged

[R4R] Release v1.1.8 #743

Merged

Mercybudda deleted the blockprune branch February 8, 2022 03:04

voron mentioned this pull request Feb 17, 2022

[bsc] add prune-block init dysnix/charts#192

Merged

This was referenced Feb 3, 2023

Feature request: ability to prune the old ancient blockchain data ethereum/go-ethereum#26596

Closed

Feature request: ability to prune the old ancient blockchain data maticnetwork/bor#718

Closed

jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023

core/rawdb: backport from bnb-chain/bsc#543

842faa8

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023

eth,ethdb,node,core/state: backport from bnb-chain/bsc#543

af405bf

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023

eth,core: backport from bnb-chain/bsc#543

9e74503

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023

core/rawdb: backport from bnb-chain/bsc#543

b450f81

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023

eth,ethdb,node,core/state: backport from bnb-chain/bsc#543

b0602fc

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023

eth,core: backport from bnb-chain/bsc#543

d06b621

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa mentioned this pull request Feb 21, 2023

feature: ancient blockchain pruner maticnetwork/bor#751

Closed

16 tasks

jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 1, 2023

core/rawdb: backport from bnb-chain/bsc#543

10ee8ba

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 1, 2023

eth,ethdb,node,core/state: backport from bnb-chain/bsc#543

927ef2b

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 1, 2023

eth,core: backport from bnb-chain/bsc#543

45b56cf

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 17, 2023

core/rawdb: backport from bnb-chain/bsc#543

a4b592c

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 17, 2023

eth,ethdb,node,core/state: backport from bnb-chain/bsc#543

8c35ece

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 17, 2023

eth,core: backport from bnb-chain/bsc#543

2b8ca73

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Jun 28, 2023

core/rawdb: backport from bnb-chain/bsc#543

9d59198

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Jun 28, 2023

eth,ethdb,node,core/state: backport from bnb-chain/bsc#543

f884677

Signed-off-by: Delweng <delweng@gmail.com>

jsvisa added a commit to jsvisa/bor that referenced this pull request Jun 28, 2023

eth,core: backport from bnb-chain/bsc#543

3c6f856

Signed-off-by: Delweng <delweng@gmail.com>

manav2401 pushed a commit to maticnetwork/bor that referenced this pull request Apr 9, 2024

all: backport from bnb-chain/bsc#543

8224aae

Signed-off-by: jsvisa <delweng@gmail.com>

manav2401 pushed a commit to maticnetwork/bor that referenced this pull request Apr 9, 2024

eth,ethdb,node,core/state: backport from bnb-chain/bsc#543

624f051

Signed-off-by: Delweng <delweng@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[R4R]offline block prune #543

[R4R]offline block prune #543

Mercybudda commented Nov 14, 2021 •

edited

unclezoro Dec 20, 2021

unclezoro Dec 20, 2021

unclezoro Dec 20, 2021

Mercybudda Dec 20, 2021

unclezoro Dec 20, 2021

unclezoro Dec 20, 2021

unclezoro Dec 20, 2021

unclezoro Dec 20, 2021

unclezoro Dec 20, 2021

unclezoro Dec 20, 2021

unclezoro Dec 20, 2021

unclezoro Dec 20, 2021

unclezoro Dec 20, 2021

Mercybudda Dec 20, 2021

unclezoro Dec 20, 2021

unclezoro Dec 20, 2021

[R4R]offline block prune #543

[R4R]offline block prune #543

Conversation

Mercybudda commented Nov 14, 2021 • edited

Description

Brief Coding workflow is as follows

Self Test

Running command

Result

Preflight checks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mercybudda commented Nov 14, 2021 •

edited