State sync from local snapshot: Understand problem, explore solutions

[sergio-mena]

Tasks:

• Analyze and understand the problem ADR083 is trying to address
• Gauge users' interest in solving this problem (how painful it is not to solve it?)
• Explore all possible solutions, among them ADR083 (ADR 083: Supporting out of band state sync tendermint/tendermint#9651)
• Adapt ADR083 according to the solution decided, finalize and merge it

DoD:

• We have made sure that users want the problem solved
• We have decided on a solution, and implemented it
• The ADR is merged

Original issue: tendermint/tendermint#9947

[cason]

So, some feedback on the state here:

Analyze and understand the problem ADR083 is trying to address

We understand.

Gauge users' interest in solving this problem (how painful it is not to solve it?)

They definitely want. It is very painful not to, they are ready to solve the problem by them own.

Adapt ADR083 according to the solution decided, finalize and merge it

My question here is another. Should we use old ADR 083, new ADR 103 (it will be renumbered, but anyway) to propose the new solution discussed on May 4th meeting? Or should we start a new ADR X with that solution?

A possible solution would be:

• Keep ADR 103 as it is, with the original solution. Cite ADR X as an alternative. Deprecate ADR 103 in favor of ADR X.
• Create a new ADR X proposing the new solution. Point to ADR 103 as an alternative.

[cason]

Keep ADR 103 as it is, with the original solution. Cite ADR X as an alternative. Deprecate ADR 103 in favor of ADR X.

What I mean here is to keep ADR 103 proposing a solution that is not a local sync, as it involves no snapshot. Make it clear on its wording. Point out some disadvantages of this solution.

[sergio-mena]

The way I see it (with the context I have):

• Set ADR 103 as rejected
• Write ADR X as proposed
• Add references to each other in a visible place

[cason]

So, we have to consider here another option that was implemented (is already merged) by the SDK. It consists of two parts:

1. feat: add local snapshots management commands cosmos/cosmos-sdk#16067

This PR creates a snapshot manager binary simd to manage snapshots of a SDK-based application. From my understanding, the SDK has already a module to manage application snapshots, so what this program does is to offer a CLI interface to interact with the snapshots manager.

In the context of this issue, 3 commands are relevant:

• dump: exports a snapshot produced by the application to a tarball (a tar.gz file). The snapshot is identified by the height it was taken.
• load: loads a snapshot from a tarball, adding it to the snapshots store. Together with dump, this enables out-of-band snapshot transfer.
• restore: apply a snapshot present in the store to the application. Since the application is stored in a leveldb-style DB, this is possible even with a running application, it is enough to lock the store.

These three commands, plus the snapshot manager CLI, enables transferring the application state between two nodes in a pretty safe way. The problem at this point is that the restored snapshot might produce an application state that is not compatible with the CometBFT state (putting it simple, the heights don't match).

2. feat!: bootstrap comet cmd for local state sync  cosmos/cosmos-sdk#16061

This second PR bootstraps CometBFT's state (namely, state store and block store) at a given height, relying on a light client. The light client configuration is set on the CometBFT's configuration file. The light client does its magic and, given a height of the blockchain, produces a state and a commit object. The state store is created/initialized from the state object. The block store stores the commit. With this information set to height H, and provided that the application state reflects the execution of block H, consensus can start from height H+1.

The code added by this PR is identical to the code used by State Sync to bootstrap a node. In the case of State Sync, the light client is configured with the height of the candidate snapshot retrieved from the p2p network. Assuming that the snapshot is valid and can be fully downloaded and installed, CometBFT is bootstrap at the snapshot's height, and starts its operation (either via block sync, or directly via consensus) from the next height.

Regarding the SDK solution, specifically, the bootstrap-state command has an optional height parameter. If set, the light client is used to produce the relevant state for that height. When not set, this should be the most common case, height is set to the application's latest commit height. Notice that in this case, after running the above described procedure, the state of CometBFT (state and block store) will be consistent with the application state.

This solution, putting in simple works, setups a state for all components (state store, block store, and application) that enables the famous Handshake procedure to execute without errors. This procedure in this case will essentially submit an Info ABCI call to the application in order to retrieve its appHash and compare it with the information stored in the state store, which should match by construction.