Provide some tests demonstrating the resilience of a Hydra Head cluster

[abailly-iohk]

Why

We have been working on improving the resilience of a Hydra Head cluster so that transient failures, whether from the network connection, a node crashing and recovering, or temporary partitions, do not lead to the Head being stuck and forces an expensive closing and reopening of the Head:

• Introduce Reliability network layer #1074
• Network resilience to disconnects #188
• Reliable persistence #1101
• Network resilient to node's crash-recovery failure mode #1079

We know there are still corner cases which are not covered, but they seem to be very unlikely to occur and as these kind of issues do not breach the intrinsic safety of a Head, we think there's not much value in closing those gaps.

However:

• We may be wrong in evaluating the frequency of those issues: They may occur more easily,
• We don't have any concrete evaluation of how resilient is a Hydra cluster today with all those changes

What

Implement some "Chaos monkey" tests, possibly manually or semi-automated, that demonstrates a Hydra Head can survive (or not) transient random crashes, network partitions, connection drops, etc.

How

• We could perhaps reuse jepsen although it's a bit complicated to setup
• We did some manual tests using iptables to drop connections and manually killing nodes, but we would like a more systematic exploration to improve coverage
• The hydra-cluster benchmarks would be a good basic, eg. we don't care about the L1 connectivity and we could even use a mock L1.