cluster: Tests around cluster isolation

[philip-stoev]

Test that if one cluster is disrupted in any way, the other
cluster can continue processing queries as usual.

Motivation

• This PR adds a known-desirable feature.

#12220

Tips for reviewer

@benesch I made some small changes to c.testdrive() so that I can pass args to the testdrive container in order to control the random seed.

[benesch]

The mzcompose bits all LGTM, but @frankmcsherry or someone else from the compute team might have thoughts on whether mz_sleep or mz_panic are actually going to exercise the right things, because I don't think there's any guarantee that they're executed on the active cluster (vs during optimization in the coordinator).

[frankmcsherry]

I'm afraid I don't have thoughts other than that I don't know that mz_panic() can be certain to do anything. Committing it to a timestamp seems to create serializability issues if we hope to do any work at later timestamps. Generally "side-effecting" methods are hard to pin down, especially if you want to be certain they do a thing (optimizing out execution of mz_panic would be both a sane thing to do at some point, and also might allow tests to pass that you would want to fail).

[philip-stoev]

I am removing the use of mz_panic() to avoid further controversy. Using mz_sleep however will exercise a legitimate scenario where the TCP socket is open between materialized and computed but computed is otherwise indisposed to make progress on any other dataflow.

As written, the mz_sleep will surely be run on computed as it has an non-constant argument.