Side load of fully-signed snapshot

[noonio]

Todo:

• Refine this description and potential solutions
• Define a scenario that we can use to test solutions
  • [] Need to find out how to construct these "diverging views" and how to resolve (pumba sets? Maybe, if any still fail after raft!)
• Understand why this is a better solution than Clear pending transactions API command #1284 (i.e. doesn't fall into the same trap)

Description

Processing transactions in a Hydra head requires each node to agree on transactions. The protocol will validate transactions (on NewTx command) against it's local view of the ledger state, using the passed --ledger-protocol-parameters. As transactions can be valid or invalid based on configuration (or to some extent exact build versions of hydra-node), it is possible that one node accepts a transaction, while the peer nodes do not.

Currently, this means that the node which accepted the transaction now has a different local state than the other nodes and might try to spend outputs that other nodes don't see available. For example, when using hydraw, the node would be using outputs introduced by the previous pixel paint transaction, but other nodes will deem any new transaction invalid with a BadInputs error.

Within this feature, we want to improve the UX of hydra-node in presence of such misalignments.

Note: We should only adopt snapshots that are enforceable on L1.

Suggested solution

• Allow adoption of a new snapshot This snapshot has to be:

  • the same for everyone, and to have a snapshot number and version strictly bigger than previous.
  • enforceable and valid against the current state of the protocol on L1:
    • must be signed by everyone (somehow).

• Allow introspection of the current snapshot in a particular node

  • We want to be able to notice if the head has become stuck. The why might be tricky but the whom would be sufficient for the time being.
  • We want to be able to observed who is missing to sign the current snapshot in flight (which is preventing from getting it confirmed)
  • Having flow metrics would not help in this scenario, given we can face a use-case where a single tx its being rejected by one of the peers and this does not depend on volume.

• Work out what constraints are required to accept a new snapshot

What

In this setting, every peer has to manually cooperate by posting a command to reset their local state to a previous snapshot confirmed.

Scenarios

1. A configuration discrepancy (like: --ledger-protocol-parameters) arises after Head is open, examples maxTxSize/maxTxExecutionUnits could be good for for the first NewTx but too much for the second, making a peer come in disagreement.

   • this also requires the miss-configured peer to reset its node and fix their config to be aligned with the rest of the party.

2. Having a peer going offline for too long and missing to catchup or resending AckSn.

Additional context

Compared to #1284, the solution does not depend on how long a peer becomes offline.
Here when a peer becomes back online, the networking layer will make sure to catch up the reconnecting peer, but if someone clear its pending txs while doing so, will create a worse scenario, where parties end up in different confirmed snapshots.

[noonio]

As discussed, there are two quirks here:

1. The networking side is resolved, basically, by the Replace network with etcd #1854 branch. So we won't try and replicate that here (even though we could maybe attempt it on top of Investigate resiliance of hydra when one node goes offline and a tx is submitted #1792 ).
2. The protocol parameter changes were not quite found to be as impactful because of details of how we write the persistent when detecting that situation ( cc @ch1bo )

That said, I think there are still two ways/reasons to proceed here:

1. This feature is a nice "escape hatch" if there does ever happen to be a stuck head. We can't know all the ways this can happen, but it may be useful to have this as a debugging/hot-fixing tool for when we encounter it.
2. It should still be possible to write a BehaviourSpec test that simply rejects Tx's (i.e. mocking out invalid protocol parameters) and then still allows a snapshot to be loaded.

Regarding this final point, @ffakenz do you want to take a shot at writing up the gist of a test that would do that? And we can at least see if we can get to a failure that this feature could resolve?

[GeorgeFlerovsky]

@ch1bo @noonio
I thought about this problem recently while writing the Hydrozoa spec. My solution is to no longer have the non-leader peers maintain unconfirmed local ledger states.

Specifically:

• When a transaction is broadcast via ReqTx, all non-leader peers simply cache it in the mempool without verifying. Only the next block's leader verifies it, updating the leader's unconfirmed ledger state.

• When the current block obtains all signatures, the next block is broadcast by its leader as usual, but the block's contents are slightly different. The block affirms one list of transactions and rejects another list of transactions.

• The non-leader peers only start validating transactions when they receive the next block from the leader. Verification proceeds as follows:

  1. Initialize a temporary ledger state, setting it equal to the confirmed ledger state.
  2. Apply each affirmed tx to the temporary ledger state.
  3. Check that each rejected tx is invalid relative the last ledger state reached in (2).

• If block verification succeeds, the peer updates its local confirmed ledger state, removes the block's affirmed and rejected txs from mempool, and broadcasts the block signature.

The advantages of this approach are that peers' local ledger states never diverge and peers' mempools are regularly flushed of stale/invalid txs.