agd does not support joining with state sync

[dckc]

Describe the bug

While there is a practice of sharing informal snapshots, the only in-protocol way to join an Agoric chain, currently, is to replay all transactions from genesis; this may take days or weeks. Contrast this with the norm in the Cosmos community:

With block sync a node is downloading all of the data of an application from genesis and verifying it. With state sync your node will download data related to the head or near the head of the chain and verify the data. This leads to drastically shorter times for joining a network.
-- State Sync | Tendermint Core

Other blockchain systems have similar features. In Bitcoin and Ethereum, software releases include a hash of a known-good state; this way, new nodes can download a state that is not more than a few months old and start verifying from there.

Design Notes

• consensus on swingset kernel DB state: currently, the swingset DB state is not part of consensus; only the sequence of messages. Mnemonic: "you can think whatever you want, as long as you say the same thing that everyone else says". Fast sync most likely requires including (most of) the KB state in consensus Merkle tree proofs.
• consensus on xsnap snapshots: currently, XS snapshots are not part of consensus; we don't require that all validators deterministically get exactly the same bytes in their snapshots. (In particular, xsnap + SES boot not deterministic #2776 is an observed case of non-determinism in snapshots). Fast sync most likely requires that we include snapshots in consensus.
• cosmos-sdk hooks to publish swingset state: baking our own system is undesirable; we just need some hooks to be able to leverage the Cosmos/Tendermint mechanisms for shipping states from select RPC nodes to the joining node (Add hooks to allow app modules to add things to state-sync cosmos/cosmos-sdk#7340 (comment))

cc @michaelfig @erights

[rowgraus]

Conservatively, I am putting it on phase 1. Feel free to postpone it as you see fit.

Tend to agree unless there are good arguments for postponing this

[dckc]

@dtribble suggests that as long as catching up is 3x to 5x faster than the running chain, we can postpone this to a later milestone.

possible optimization:

• replay from snapshot in parallel

(I think we only replay on-line vats, of which there is a bounded number, so that optimization doesn't seem worthwhile)

[Tartuffo]

First need to measure how long it currently takes based on estimated number of blocks, and or calc the blocks / time of recovery. Create sub-ticket for this initial measurement.

[dckc]

some mainnet0 data shows new nodes should eventually catch up, but it takes a long time.
#4106 (comment)

The validator community seems to prefer informal snapshot sharing.
Agoric/testnet-notes#42

[mhofman]

A few quick thoughts:

• Without state sync, catching up a new validator relies on the same execution paths as a live execution. Whatever work we would do to optimize the time it takes to catch up from scratch, especially at the SwingSet / JS level, would be optimization we'd do for normal execution.
• That means if we ever get to a fairly high utilization* (regardless of parallelization), it won't be possible to catch up by mere replay in a meaningful amount of time (unless you throw a lot more compute power at the catchup node, to reduce utilization)
• During the catch up time, the chain continues to make progress, which has to be caught up to as well. IOU a formula

*: My understanding is that the meaningful number to measure is the amount of time spent in Swingset, compared to time elapsed since genesis. That gives Swingset utilization. If you consider the cosmos processing to be negligible comparatively, you can then calculate time it'd take to rebuild all the JS state through catch up. It also gives a lower bound.

[erights]

Can we replay each vat separately and in parallel?

[michaelfig]

Can we replay each vat separately and in parallel?

We need "state sync" to jump to a snapshot of the kernel data close to the current block or else we can only replay then verify a single block at a time since genesis. That's really slow, even if we do more in parallel.

[dckc]

@warner further to the discussion we just had about trade-offs between performance and integrity of snapshots, as I mentioned, our validator community is doing some informal snapshot sharing currently: Agoric/testnet-notes#42.

I looked around and found that it seems to take about 3.5min of downtime to do a daily mainnet0 snapshot.
Some follow-up step to make the snapshot available seems to take significantly longer; I'm not sure what going on in there...

---------------------------

|2022-03-14_01:00:01| LAST_BLOCK_HEIGHT 4103449
|2022-03-14_01:00:01| Stopping agoric.service
0
|2022-03-14_01:00:01| Creating new snapshot
|2022-03-14_01:03:33| Starting agoric.service
0
|2022-03-14_01:03:33| Moving new snapshot to /home/snapshots/data/agoric
155G	/home/snapshots/snaps/agoric_2022-03-14.tar
|2022-03-14_02:19:47| Done
---------------------------

|2022-03-15_01:00:01| LAST_BLOCK_HEIGHT 4116868
|2022-03-15_01:00:01| Stopping agoric.service
0
|2022-03-15_01:00:01| Creating new snapshot
|2022-03-15_01:03:27| Starting agoric.service
0
|2022-03-15_01:03:27| Moving new snapshot to /home/snapshots/data/agoric
156G	/home/snapshots/snaps/agoric_2022-03-15.tar
|2022-03-15_03:06:14| Done
---------------------------

-- https://snapshots.stake2.me/agoric/agoric_log.txt

[dckc]

one validator notes:

the post upgrade mainnet snapshot is already nearly 2GB in size.

[warner]

One data point: I watched a node crash today, it missed about 200s before getting restarted. The restart took 2m10s to replay vat transcripts enough to begin processing blocks again, then took another 33s to replay the 95-ish (empty) missed blocks, after which is was caught up and following properly again.

The vat-transcript replay time is roughly bounded by the frequency of our heap snapshots: we take a heap snapshot every 2000 deliveries, so no single vat should ever need to replay more than 2000 deliveries at reboot time, so reboot time will be random but roughly constant (depends on deliveryNum % 2000 summed across all vats).

Note that this doesn't tell us anything about how long it takes to start up a whole new validator from scratch.

[mhofman]

After discussing state-sync the other day, @arirubinstein mentioned validators leverage state sync to work around a cosmos DB pruning issue: they start a new node state syncing from their existing node to prune their DB.

In case for some reason we can't figure out state sync by the time the DBs grows out too large, we should check if the following rough hack may work:

• shut down node at height of a block N that won't be pruned from cosmos DB.
• Make copy of swingset state dir
• Restart node
• Start new node from swingset state copy, and using cosmos state-sync at same block height N to re-populate the IAVL tree.

For consistency protection, Swingset saves the block height it last committed, and checks that the next block it sees is either the next block N + 1, or the same block N (in which case it doesn't execute anything but simply replays calls it previously made back to the go / cosmos side).

[dckc]

... as long as catching up is 3x to 5x faster than the running chain ...

a recent data point: 26hrs to catch up on 26 chain days. So 24x.