refactor!: abstractions for snapshot and pruning; snapshot intervals eventually pruned; unit tests #11496

p0mvn · 2022-03-29T20:40:17Z

Description

We've run into issues where attempting to prune a height under snapshot started being too frequent. To temporarily mitigate, we had to require node operators to have a large pruning-keep-recent.

This PR fixes this problem by avoiding pruning snapshot heights until after a snapshot is complete.

In addition, an abstraction for pruning (pruning/manager.go) was added. Also, unit tested rigorously from config to base app.

Marked as API breaking because changed the following:

snapshot options
pruning options
baseapp
type Committer interface
type Snapshotter interface

Closes: #XXXX

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

confirmed the correct type prefix in the PR title
confirmed ! in the type prefix if API or client breaking change
confirmed all author checklist items have been addressed
reviewed state machine logic
reviewed API design and naming
reviewed documentation is accurate
reviewed tests and test coverage
manually tested (if applicable)

codecov · 2022-03-29T21:21:57Z

Codecov Report

Merging #11496 (e53691a) into master (d4dd444) will increase coverage by 0.23%.
The diff coverage is 73.57%.

@@            Coverage Diff             @@
##           master   #11496      +/-   ##
==========================================
+ Coverage   65.91%   66.14%   +0.23%     
==========================================
  Files         667      678      +11     
  Lines       70466    72010    +1544     
==========================================
+ Hits        46448    47632    +1184     
- Misses      21323    21598     +275     
- Partials     2695     2780      +85

Impacted Files	Coverage Δ
server/config/toml.go	`66.66% <ø> (ø)`
server/mock/store.go	`19.38% <0.00%> (-0.83%)`	⬇️
server/rollback.go	`0.00% <0.00%> (ø)`
simapp/simd/cmd/root.go	`71.48% <0.00%> (-0.93%)`	⬇️
store/iavl/store.go	`67.31% <0.00%> (ø)`
store/mem/store.go	`82.35% <0.00%> (-5.15%)`	⬇️
store/rootmulti/dbadapter.go	`0.00% <0.00%> (ø)`
store/types/store.go	`70.73% <ø> (ø)`
store/v2alpha1/mem/store.go	`84.61% <0.00%> (ø)`
store/v2alpha1/transient/store.go	`86.66% <0.00%> (ø)`
... and 31 more

snapshots/manager.go

alexanderbez · 2022-03-31T17:55:25Z

@p0mvn is this ready for review?

p0mvn · 2022-03-31T18:10:04Z

@alexanderbez not yet, need to fix one more test. I will ping you once ready

p0mvn · 2022-03-31T18:22:08Z

The test I referred to above is broken on master so this is ready. @alexanderbez

I have some questions about v2alpha1. I did not refactor it. I think it might be best to do it in a separate PR to limit the scope. Would appreciate reviewer feedback on this

server/pruning.go

server/pruning_test.go

server/start.go

alexanderbez · 2022-04-01T13:36:14Z

pruning/README.md

+
+The strategies are configured in `app.toml`:
+pruning = "< strategy >" # where the options are:
+- `default`: only the last 100,000 states(approximately 1 week worth of state) are kept; pruning at 100 block intervals


We changed the default value some time ago. Did you change it back to 100k in this PR?

We have 100k in Osmosis. I missed this difference when cherry-picking. Changed back to the updated default value of 362880.

What was the reason for this change?

alexanderbez

This is a pretty large PR to review in detail TBH. But since it was merged in Osmosis, I see no reason it can't be merged here.

tac0turtle · 2022-04-06T10:02:19Z

.github/workflows/lint.yml

@@ -1,31 +1,17 @@
 name: Lint
-# Lint runs golangci-lint over the entire cosmos-sdk repository


can we revert changing this file, its setup to not run if no go code is down to not clog and slow down ci for docs prs

This was an accidental change, removed now. Thanks for catching that

odeke-em

Thank you for this change @p0mvn. I've added a review, please take a look.

odeke-em · 2022-04-06T20:15:05Z

baseapp/baseapp.go

+	if !ok {
+		return errors.New("rootmulti store is required")
+	}
+	if err := rms.GetPruning().Validate(); err != nil {


We can simply do: return rms.GetPruning().Validate()
and that will remove the need for the 3+ lines below.

odeke-em · 2022-04-06T23:40:29Z

pruning/manager.go

+func (m *Manager) flushPruningSnapshotHeights(batch dbm.Batch) {
+	m.mx.Lock()
+	defer m.mx.Unlock()
+	bz := make([]byte, 0)


We can preallocate this buffer by doing:
bz := make([]byte, 0, m.pruneSnapshotHeights.Len()*8)

pruning/manager.go

odeke-em · 2022-04-06T23:43:14Z

pruning/manager.go

+}
+
+func (m *Manager) flushPruningHeights(batch dbm.Batch) {
+	bz := make([]byte, 0)


bz := make([]byte, 0, len(m.pruneHeights)*8)

odeke-em · 2022-04-06T23:45:48Z

pruning/manager.go

+		offset += 8
+	}
+
+	if pruneSnapshotHeights.Len() > 0 {


This condition will never fail given that offset which starts off as 0 is ALWAYS less than len(bz) which you just tested right above and have as the invariant for the loop. You will always add at least 1 element to the list.

snapshots/manager.go

odeke-em · 2022-04-07T00:26:14Z

store/rootmulti/store.go

+	for key, store := range rs.stores {
+		// If the store is wrapped with an inter-block cache, we must first unwrap
+		// it to get the underlying IAVL store.
+		if store.GetStoreType() == types.StoreTypeIAVL {


You can reverse conditional this and avoid the nesting so

if store... != types.StoreTypeIAVL. { continue } ...

odeke-em · 2022-04-07T00:26:53Z

store/rootmulti/store.go

+
+			store = rs.GetCommitKVStore(key)
+
+			if err := store.(*iavl.Store).DeleteVersions(pruningHeights...); err != nil {


Ditto here if you'd like above reverse conditional

odeke-em · 2022-04-07T00:27:34Z

store/rootmulti/store.go

-// GetStoreByName performs a lookup of a StoreKey given a store name typically
+func (rs *Store) handlePruning(version int64) error {
+	rs.pruningManager.HandleHeight(version - 1) // we should never prune the current version.
+	if rs.pruningManager.ShouldPruneAtHeight(version) {


Please reverse conditional this to return if !rs...

odeke-em · 2022-04-07T00:28:28Z

store/rootmulti/store.go

@@ -26,11 +21,18 @@ import (
 	"github.com/cosmos/cosmos-sdk/store/transient"
 	"github.com/cosmos/cosmos-sdk/store/types"
 	sdkerrors "github.com/cosmos/cosmos-sdk/types/errors"
+
+	iavltree "github.com/cosmos/iavl"


Please revert this block imports as they were idiomatic before

There is a "github.com/cosmos/cosmos-sdk/store/iavl" on top. So we need a rename to avoid the clash.

alexanderbez · 2022-04-07T12:35:01Z

@odeke-em provided excellent feedback here. TY!

p0mvn · 2022-04-08T21:29:54Z

@odeke-em Thank you for your review. All comments are now addressed. Please take a look when you have time

odeke-em

Thank you for addressing my feedback @p0mvn! Nice work and LGTM!

odeke-em · 2022-04-08T21:47:35Z

server/start.go

@@ -25,13 +25,14 @@ import (
 	"github.com/cosmos/cosmos-sdk/client"
 	"github.com/cosmos/cosmos-sdk/client/flags"
 	"github.com/cosmos/cosmos-sdk/codec"
+


No need for this newline though as this is the same block of imports :-)

snapshots/manager.go

odeke-em · 2022-04-08T21:52:24Z

store/rootmulti/store.go

@@ -559,7 +544,47 @@ func (rs *Store) GetKVStore(key types.StoreKey) types.KVStore {
 	return store
 }

-// GetStoreByName performs a lookup of a StoreKey given a store name typically
+func (rs *Store) handlePruning(version int64) error {
+	rs.pruningManager.HandleHeight(version - 1) // we should never prune the current version.


Could you please check that this "version" is always greater than 1?

version can be 1. However, that is not a problem because we always require keep-recent > 2 so a 0 height never gets pruned. Just in case this ever gets changed, I added a guard against previousHeight == 0 in HandleHeight

What happens if version becomes negative? Checking for == 0 only guards against that single value

That is not possible. The only valid values are 0 and greater

It's always a good idea to practice defensive programming. As version is of type int64 it's possible this method is called with a value of 0 or -1 or -123456. If HandleHeight has specific constraints on the values it accepts, then this method should enforce those constraints before calling the method.

Added a check to cover all possible inputs, table tests TestHandleHeight_Inputs to test all possible behaviors, and updated godoc

odeke-em · 2022-04-08T21:53:35Z

store/rootmulti/store.go

+	err := rs.pruneStores()
+	if err != nil {


Minor nit, you can make a compound one-liner

if err := rs.pruneStores(); err != nil { panic(err) }

p0mvn · 2022-04-20T22:45:35Z

I tested all these changes on Osmosis v7.x by cherry-picking all relevant commits onto the desired branch. Works exactly as expected.

The summary of the tests can be seen here: osmosis-labs#184 (comment)

@alexanderbez Please let me know how we should proceed. I'm ready to merge this if you agree that the tests are acceptable

alexanderbez

LGTM. We need to undo the changelog changes. Then we can proceed with merging.

Also, tbh, I haven't fully reviewed this PR in complete detail -- it's just way too large. So I'm mainly giving an ACK as it's been seen to work reliably for Osmosis.

snapshots/manager.go

…nager

CHANGELOG.md

alexanderbez · 2022-04-21T18:42:37Z

Will merge once CI passes 👍

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

cosmos/cosmos-sdk#11496

…eventually pruned; unit tests (cosmos#11496)

github-actions bot added C:Store C:x/upgrade labels Mar 29, 2022

This was referenced Mar 29, 2022

[Epic] Database layer performance and stability osmosis-labs/osmosis#1016

Closed

Upstream iavl and sdk database changes osmosis-labs/osmosis#1175

Closed

p0mvn changed the title ~~refactor: abstractions for snapshot and pruning; snapshot intervals eventually pruned; unit tests~~ refactor!: abstractions for snapshot and pruning; snapshot intervals eventually pruned; unit tests Mar 30, 2022

p0mvn commented Mar 30, 2022

View reviewed changes

snapshots/manager.go Show resolved Hide resolved

p0mvn marked this pull request as ready for review March 31, 2022 18:21

p0mvn requested review from aaronc and alexanderbez as code owners March 31, 2022 18:21

alexanderbez reviewed Apr 1, 2022

View reviewed changes

tac0turtle requested review from robert-zaremba and amaury1093 April 4, 2022 09:03

alexanderbez approved these changes Apr 4, 2022

View reviewed changes

amaury1093 mentioned this pull request Apr 4, 2022

v0.46.0 Release #11096

Closed

56 tasks

p0mvn force-pushed the roman/upstream/snapshot-pruning-refactor branch from 74d245d to b57d056 Compare April 6, 2022 00:26

github-actions bot added the T: CI label Apr 6, 2022

tac0turtle reviewed Apr 6, 2022

View reviewed changes

p0mvn force-pushed the roman/upstream/snapshot-pruning-refactor branch from 519c590 to b57d056 Compare April 6, 2022 17:13

github-actions bot removed the T: CI label Apr 6, 2022

odeke-em requested changes Apr 7, 2022

View reviewed changes

p0mvn force-pushed the roman/upstream/snapshot-pruning-refactor branch from b57d056 to 0b15651 Compare April 8, 2022 18:32

p0mvn mentioned this pull request Apr 8, 2022

refactor: address snapshot/pruning refactor comments upstream osmosis-labs/cosmos-sdk#184

Merged

odeke-em approved these changes Apr 8, 2022

View reviewed changes

p0mvn force-pushed the roman/upstream/snapshot-pruning-refactor branch from 062847f to d43a785 Compare April 8, 2022 22:16

p0mvn added 7 commits April 16, 2022 14:55

update SetSnapshotInterval comment

f3b5485

remove snapshot store type

2023fa3

remove pruning and snapshot wrappers from store types

823971f

fix HandleHeight comment in pruning manager

73b75ee

Merge branch 'master' into roman/upstream/snapshot-pruning-refactor

986f6ff

Merge branch 'master' into roman/upstream/snapshot-pruning-refactor

19669c4

snapshot happens in a separate goroutine

bbe9d5e

alexanderbez reviewed Apr 21, 2022

View reviewed changes

snapshots/manager.go Outdated Show resolved Hide resolved

Merge branch 'master' into roman/upstream/snapshot-pruning-refactor

073b664

p0mvn force-pushed the roman/upstream/snapshot-pruning-refactor branch from a65917f to 073b664 Compare April 21, 2022 14:10

p0mvn added 4 commits April 21, 2022 13:43

set snapshot interval on multistore in baseapp instead of snapshot ma…

2dc26e5

…nager

Merge branch 'master' into roman/upstream/snapshot-pruning-refactor

d6a3569

fix snapshot manager unit tests

8eee2f7

Merge branch 'master' into roman/upstream/snapshot-pruning-refactor

5d85f99

alexanderbez approved these changes Apr 21, 2022

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

alexanderbez and others added 2 commits April 21, 2022 14:52

Merge branch 'master' into roman/upstream/snapshot-pruning-refactor

eb5a3b3

Update CHANGELOG.md

e53691a

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

alexanderbez merged commit 42f8d45 into cosmos:master Apr 21, 2022

alexanderbez deleted the roman/upstream/snapshot-pruning-refactor branch April 21, 2022 19:30

p0mvn mentioned this pull request Apr 24, 2022

Add modules for deterministic total ordering derivation from partial ord osmosis-labs/osmosis#1335

Merged

p0mvn mentioned this pull request May 3, 2022

feat: Retroactive prune very old blocks from store #11064

Closed

19 tasks

mkXultra added a commit to UnUniFi/chain that referenced this pull request Jul 11, 2022

change in use of snapshot

83e8606

cosmos/cosmos-sdk#11496

mkXultra added a commit to UnUniFi/chain that referenced this pull request Jul 14, 2022

change in use of snapshot

18b1545

cosmos/cosmos-sdk#11496

tkxkd0159 mentioned this pull request Oct 25, 2022

Snapshot() Function Does Not Return An Error When It Fails Finschia/finschia-sdk#749

Closed

4 tasks

mhofman mentioned this pull request Feb 24, 2023

feat: agoric-sdk state-sync support agoric-labs/cosmos-sdk#297

Merged

19 tasks

larry0x pushed a commit to larry0x/cosmos-sdk that referenced this pull request May 22, 2023

refactor!: abstractions for snapshot and pruning; snapshot intervals …

8e4fe5e

…eventually pruned; unit tests (cosmos#11496)

alejandrosuarez mentioned this pull request Aug 3, 2023

[Snyk] Upgrade tailwindcss from 3.3.2 to 3.3.3 alejandrosuarez/cosmos-sdk#11

Closed

RollkitBot mentioned this pull request Aug 24, 2023

[Snyk] Upgrade tailwindcss from 3.2.2 to 3.3.3 rollkit/cosmos-sdk-old#44

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor!: abstractions for snapshot and pruning; snapshot intervals eventually pruned; unit tests #11496

refactor!: abstractions for snapshot and pruning; snapshot intervals eventually pruned; unit tests #11496

p0mvn commented Mar 29, 2022 •

edited

Loading

codecov bot commented Mar 29, 2022 •

edited

Loading

alexanderbez commented Mar 31, 2022

p0mvn commented Mar 31, 2022

p0mvn commented Mar 31, 2022

alexanderbez Apr 1, 2022

p0mvn Apr 1, 2022

alexanderbez left a comment

tac0turtle Apr 6, 2022

p0mvn Apr 6, 2022

odeke-em left a comment

odeke-em Apr 6, 2022

odeke-em Apr 6, 2022

odeke-em Apr 6, 2022

odeke-em Apr 6, 2022

odeke-em Apr 7, 2022

odeke-em Apr 7, 2022

odeke-em Apr 7, 2022

odeke-em Apr 7, 2022

p0mvn Apr 8, 2022

alexanderbez commented Apr 7, 2022

p0mvn commented Apr 8, 2022

odeke-em left a comment

odeke-em Apr 8, 2022

odeke-em Apr 8, 2022

p0mvn Apr 8, 2022

odeke-em Apr 8, 2022

p0mvn Apr 8, 2022

peterbourgon Apr 10, 2022

p0mvn Apr 11, 2022

odeke-em Apr 8, 2022

p0mvn commented Apr 20, 2022

alexanderbez left a comment

alexanderbez commented Apr 21, 2022

		@@ -1,31 +1,17 @@
		name: Lint
		# Lint runs golangci-lint over the entire cosmos-sdk repository


		store = rs.GetCommitKVStore(key)

		if err := store.(*iavl.Store).DeleteVersions(pruningHeights...); err != nil {

refactor!: abstractions for snapshot and pruning; snapshot intervals eventually pruned; unit tests #11496

refactor!: abstractions for snapshot and pruning; snapshot intervals eventually pruned; unit tests #11496

Conversation

p0mvn commented Mar 29, 2022 • edited Loading

Description

Author Checklist

Reviewers Checklist

codecov bot commented Mar 29, 2022 • edited Loading

Codecov Report

alexanderbez commented Mar 31, 2022

p0mvn commented Mar 31, 2022

p0mvn commented Mar 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexanderbez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

odeke-em left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexanderbez commented Apr 7, 2022

p0mvn commented Apr 8, 2022

odeke-em left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p0mvn commented Apr 20, 2022

alexanderbez left a comment

Choose a reason for hiding this comment

alexanderbez commented Apr 21, 2022

p0mvn commented Mar 29, 2022 •

edited

Loading

codecov bot commented Mar 29, 2022 •

edited

Loading