Store v2 Migration

[alexanderbez]

Context

When store v2 nears completion, we'll need to devise a strategy for existing applications to migrate from store v1 to store v2. Specifically, the state that exists in store v1 is not compatible with store v2 as the latter uses a btree (SQLite) to index and store data in SC. In addition, we'll need to populate SS with data as the node syncs/migrates.

This issue is to track various proposals and approaches to migration and come to consensus on one. One such proposal is to utilize state sync and have a node populate SS and SC backends. However, this has potential characteristics of degraded performance.

Another idea is to sync latest state and then retroactively sync historical state in the background.

Tasks

• Implement the process to migrate the whole state at the given height to the new store/v2 backends #19326
• Implement the catch-up process to catch up the processed blocks while the state syncing #19437

---

cc @kocubinski @tac0turtle @cool-develope @yzang2019

Tasks

Beta Give feedback

1. Add an option to avoid the commitment migration when users want to use the iavl/v0 or v1 as a commitment layer without migration #20179
   C:Store +1
   
2. Implement Checkpointing per Module for Resumable Migration when restart #19438
   C:Store +1
3. Replace the commitSnapshotter of the snapshot manager when the migration is done #19705
   C:Store +1
   
4. Avoid to write & query against SS while migration #20203
   C:Store +1
   

[tac0turtle]

The lazy way i have in mind is to start storev2 with iavlv0/1 and iavlv2. In this approach there are two ways to rebuild current state.

A) Iavlv0/1 is used as the tree for the node while SS and SC is populated in parallel. Once the state is moved over, if there is a way to know this in the code base we would close the old database and continue with storev2 ss and sc.
B) we write to iavlv0/1 and iavlv2 until current state is migrated. It will take longer as a downside but may be more performant if we cant have multiple readers on the leveldb (the default db of store v1)

There could be more ways but i see these as two possible ways for doing it in a lazy fashion.

Out of band migration for archive nodes is also possible but less complex as we can tell node operators with larger databases to run a side process to do the migration then restart the node once the migration has completed. This could take a very long time for archive nodes

[yzang2019]

Assuming the new binary is backward compatible, we could just seamlessly migrate to the new binary first without enabling storeV2, so the new binary will still run iavl v0/v1 backend, any node later can just turn on the configuration to switch to storeV2 and do a state sync to finish the switch to storeV2 at any time, since state snapshot are completely the same and compatible.

[tac0turtle]

this is fine for validators, it is better ux to automate all of it. Leaving it to node operators will make it so that some dont migrate and will be forced to do it once we deprecate the migration step. In order to avoid maintaining v1/v0 integration for a long time we should not make it optional

[cool-develope]

Importing the snapshot is slower than directly sync the latest state, I am a little worried if it may lead to delays due to load the snapshot after binary switching.
Personally, I like the way of operating v0/v1 and v2 together, background migration on v2 while working v0/v1 as the store.

[alexanderbez]

So after speaking with @cool-develope we've proposing the following:

• Upon initial start, the node stores the current (i.e. latest) height as a checkpoint, C
• All node operators run their nodes with store v2 enabled, using IAVL v0/v1 as both SS and SC (we might need to write a wrapper for this). This allows for all queries (from the state machine and from clients) <= C to be served.
• As the node processes blocks, it will start from C and populate the real SS backend (e.g. RocksDB) and SC (IAVL v2) in the background, working it's way back to the earliest height the particular node has.
• Simultaneously, as new blocks are processed they will be fed into the real SS and SC backends.
• Once the background sync process is complete, the v0/v1 wrapper can be safely removed.

In summary, store v2 will use v0/v1 and v2 in conjunction, with some checkpoint C. All queries <= C, will be served by v0/1, while all queries > C will be served by the real SS and SC backends. This means store v2 will need to be aware of this checkpoint C logic (until everything is synced).

[cool-develope]

we can't process new blocks against SC until state sync, since the tree is not constructed yet.
Maybe, if the state sync is done at C+V, then we will fetch the changesets for C+1 to C+V from SS and apply them to SC, so that we can sync the current height (C+V), after that we can remove v0/v1 completely