Enable local statesync/snapshot restore #13521

chillyvee · 2022-10-12T15:13:46Z

Seeking guidance for commit:
82f708f (rebased on main for ease of viewing, not tested, but good for comment)

Branch which works with Juno Uni Testnet https://github.com/chillyvee/cosmos-sdk/tree/cv_local_statesync_juno_uni

This PR enables cosmos-sdk to initiate local snapshot restore with tendermint finishing up the stateStore and blockStore data restoration.

This development is based on juno's uni testnet (cosmos-sdk v0.45.6, tendermint v0.34.20) and pairs with tendermint PR enhancement tendermint/tendermint#9541

This PR enables a validator quickly restore without needing to search for remote snapshot servers. In basis, if a validator already takes periodic snapshots, it is wasteful in time and bandwidth to search over P2P in hopes of finding a snapshot.

However, an RPC server is still required to fetch the last, current, and next blocks to fully restore stateStore and blockStore.

statesync/stateprovider.go

146:    lastLightBlock, err := s.lc.VerifyLightBlockAtHeight(ctx, int64(height), time.Now())
150:    currentLightBlock, err := s.lc.VerifyLightBlockAtHeight(ctx, int64(height+1), time.Now())
154:    nextLightBlock, err := s.lc.VerifyLightBlockAtHeight(ctx, int64(height+2), time.Now())

The use case is to quickly reduce disk usage which normally grows over time to an arguably large size without the need to slowly prune IAVL trees.

Before further development, we'd like to ask:

Do you prefer such work to be done on the main branch and then backport?

If testing is required, how much additional verification is preferred? For example, juno/uni was chosen because this proves that cosmwasm stores are properly restored. However it would not be possible to verify with just cosmos-sdk tests.

Please take a look at the structure of code and indicate whether any rules regarding ABCI or API/Responsibilities have been broken.

The current code design is meant for ease of review/comment before adding multiple additional layers of abstraction to put each bit of code in the right place.

What is an efficient way to check if local stores already have existing height data prior to restoring the snapshot?

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

confirmed the correct type prefix in the PR title
confirmed ! in the type prefix if API or client breaking change
confirmed all author checklist items have been addressed
reviewed state machine logic
reviewed API design and naming
reviewed documentation is accurate
reviewed tests and test coverage
manually tested (if applicable)

tac0turtle · 2022-10-12T15:42:09Z

hey looks like there was an issue with the branch you have locally. any way you could rebase to main?

alexanderbez · 2022-10-23T18:26:04Z

Ok, with that frame, I will re-review. I also would encourage we name this "feature" (I think it's a feature?) something like "Lazy State Sync". WDYT?

server/start.go

alexanderbez · 2022-10-23T18:35:01Z

server/start.go

@@ -244,6 +247,53 @@ func startStandAlone(ctx *Context, appCreator types.AppCreator) error {
 	return WaitForQuitSignals()
 }

+func restoreLocalSnapshot(ctx *Context, app types.Application, ssRestoreHeight uint64) error {
+	ctx.Logger.Info(fmt.Sprintf("Local Statesync Restore Snapshot With Height: %d", ssRestoreHeight))
+	ctx.Config.StateSync.RestoreHeight = ssRestoreHeight // Should be a one-shot command


Why do we need to update the config?

Will remove (maybe can't, see next comment)

FYI: Used here in the tendermint layer to detect a statesync, but there might be a way to remove it.

https://github.com/tendermint/tendermint/pull/9541/files#diff-4eaef000463512634f080db6f15d6aeb1b856e995a148bbeea5ee94758210fe1R281

Discussing with the tendermint maintainers if there's a more preferred way to send a signal that cosmos-sdk performed a "Lazy State Sync".

They are asking to consider a slightly different code path which might require a variable in a different place (or no variable at all).

tendermint/tendermint#9541 (comment)

I'm having some trouble implementing the idea they provided due to some objects not being in scope to call so I'll post back here as soon as I get some assistance.

server/start.go

chillyvee · 2022-10-23T22:10:12Z

Thank you for the review. Will clean up the code as recommended in a day or two.

chillyvee · 2022-10-23T22:12:18Z

Ok, with that frame, I will re-review. I also would encourage we name this "feature" (I think it's a feature?) something like "Lazy State Sync". WDYT?

"Lazy State Sync" works.

Earlier thought of "Local State Sync", but perhaps the word "Local" is a bit overused.

chillyvee · 2022-10-25T21:02:29Z

Regarding the usage of ctx.Logger.Error, do you have any recommendations on how to return an error after Logging it to console?

Latest changes based on your feedback are added to this branch.

alexanderbez · 2022-10-26T16:27:12Z

Regarding the usage of ctx.Logger.Error, do you have any recommendations on how to return an error after Logging it to console?

IIRC, error should not cause the process to halt. You can return the error after you log it.

chillyvee · 2022-10-26T19:21:14Z

Appreciate all the reviews and recommendations.

Please hold off merging until we finalize the tendermint side of this effort. :)

github-actions · 2022-12-03T00:04:27Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

chillyvee · 2022-12-03T00:46:22Z

Hey who's calling us stale?!

github-actions · 2023-01-13T00:05:14Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

alexanderbez · 2023-01-16T22:43:26Z

@chillyvee this feature is really awesome and I personally would love to see this land. I know this is contingent upon TM work being delivered first. Any progress on that?

Or is there anyway we can define or merge this w/o waiting on TM s.t. when TM finally does include the necessary changes, this will just automatically work?

Also, I'm more than happy to push this over the finish line myself too if you're too busy. Either way, just lmk 👍

chillyvee · 2023-01-17T05:34:30Z

Any progress on that?

Or is there anyway we can define or merge this w/o waiting on TM s.t. when TM finally does include the necessary changes, this will just automatically work?

On TM - Do you know if we're switching the official tendermint repo? If so we may need to issue a new PR.

There was a bit of talk about how to structure the code in TM (how to take the implementation, restructure where various lines of code go). In the process, a lot of intermediate changes were requested, which raised questions, but not all were answered because the maintainer there had other priorities.

Also TM wants to merge against main/head and backport, so it will take a few more iterations before we can reference from cosmos-sdk.

Do you think it's possible to work together on both sides of cosmos-sdk/tendermint repo?

chillyvee · 2023-01-17T05:36:18Z

Or is there anyway we can define or merge this w/o waiting

Will check again :) There are a few more follow in features that might be worth adding in, but trying to keep code scope small so we can merge "quickly" without additional delays

yihuang · 2023-01-17T06:26:31Z

it could be useful even without the tm part, for example transform an archive node to a non-archive one fast, by restore to a previous snapshot and replay the blocks from tm.

alexanderbez · 2023-01-17T16:42:20Z

So the TM repo is moved to Informal's org and will be renamed in the coming days/weeks. Work will continue there, so yes, be sure to make sure all relevant issues and PRs are there now @chillyvee

tac0turtle · 2023-02-19T00:52:05Z

hey, really love this work, I will close it until we have the change in comet. We dont have a timeline for that, so dont want to leave this pr open for a unknown amount of time. Once we have a timeline i will contact you. Sorry for the situation. The comet team is working hard so i can see this happening soon.

chillyvee requested a review from a team as a code owner October 12, 2022 15:13

This was referenced Oct 12, 2022

Enable local snapshot restore tendermint/tendermint#9541

Open

feat: Retroactive prune very old blocks from store #11064

Closed

chillyvee force-pushed the cv_local_statesync branch from 8b44455 to 82f708f Compare October 12, 2022 16:48

github-actions bot removed C:x/authz C:x/capability labels Oct 12, 2022

alexanderbez suggested changes Oct 23, 2022

View reviewed changes

CV Apply feedback part 1

fed5b22

chillyvee force-pushed the cv_local_statesync branch from 85a2277 to 49866ff Compare October 25, 2022 21:01

Use Logger.Error

d104aa7

chillyvee force-pushed the cv_local_statesync branch from 49866ff to d104aa7 Compare October 25, 2022 21:05

alexanderbez approved these changes Oct 26, 2022

View reviewed changes

alexanderbez added the S:blocked Status: Blocked label Oct 27, 2022

github-actions bot added the stale label Dec 3, 2022

tac0turtle removed the stale label Dec 3, 2022

github-actions bot added the stale label Jan 13, 2023

github-actions bot removed the stale label Jan 17, 2023

aaronc removed their assignment Jan 25, 2023

yihuang mentioned this pull request Feb 8, 2023

EPIC: Optimizing transaction processing #14929

Closed

tac0turtle closed this Feb 19, 2023

yihuang mentioned this pull request May 8, 2023

feat: save restored snapshot locally #16060

Merged

19 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable local statesync/snapshot restore #13521

Enable local statesync/snapshot restore #13521

chillyvee commented Oct 12, 2022 •

edited

tac0turtle commented Oct 12, 2022

alexanderbez commented Oct 23, 2022

alexanderbez Oct 23, 2022

chillyvee Oct 23, 2022 •

edited

chillyvee Oct 25, 2022

chillyvee Oct 25, 2022 •

edited

chillyvee commented Oct 23, 2022

chillyvee commented Oct 23, 2022

chillyvee commented Oct 25, 2022 •

edited

alexanderbez commented Oct 26, 2022

chillyvee commented Oct 26, 2022

github-actions bot commented Dec 3, 2022

chillyvee commented Dec 3, 2022

github-actions bot commented Jan 13, 2023

alexanderbez commented Jan 16, 2023 •

edited

chillyvee commented Jan 17, 2023

chillyvee commented Jan 17, 2023

yihuang commented Jan 17, 2023 •

edited

alexanderbez commented Jan 17, 2023

tac0turtle commented Feb 19, 2023

Enable local statesync/snapshot restore #13521

Enable local statesync/snapshot restore #13521

Conversation

chillyvee commented Oct 12, 2022 • edited

Author Checklist

Reviewers Checklist

tac0turtle commented Oct 12, 2022

alexanderbez commented Oct 23, 2022

alexanderbez Oct 23, 2022

Choose a reason for hiding this comment

chillyvee Oct 23, 2022 • edited

Choose a reason for hiding this comment

chillyvee Oct 25, 2022

Choose a reason for hiding this comment

chillyvee Oct 25, 2022 • edited

Choose a reason for hiding this comment

chillyvee commented Oct 23, 2022

chillyvee commented Oct 23, 2022

chillyvee commented Oct 25, 2022 • edited

alexanderbez commented Oct 26, 2022

chillyvee commented Oct 26, 2022

github-actions bot commented Dec 3, 2022

chillyvee commented Dec 3, 2022

github-actions bot commented Jan 13, 2023

alexanderbez commented Jan 16, 2023 • edited

chillyvee commented Jan 17, 2023

chillyvee commented Jan 17, 2023

yihuang commented Jan 17, 2023 • edited

alexanderbez commented Jan 17, 2023

tac0turtle commented Feb 19, 2023

chillyvee commented Oct 12, 2022 •

edited

chillyvee Oct 23, 2022 •

edited

chillyvee Oct 25, 2022 •

edited

chillyvee commented Oct 25, 2022 •

edited

alexanderbez commented Jan 16, 2023 •

edited

yihuang commented Jan 17, 2023 •

edited