More interesting examples of testing distribution system building blocks

[chenliu0831]

Description

It would be great to provide a few more complex example to showcase more Turmoil capabilities. The example should be succinct with lightweight dependencies but need functional testing for:

1. Fault tolerance
2. Scalability to many nodes

Proposal

Food for thought - https://martinfowler.com/articles/patterns-of-distributed-systems/ with a few candidates:

1. Heartbeat: seems straightforward, can be made more complex with Gossip?
2. Leader - Follower: some reference implementation here for raft, seems to touch heartbeat, quorum as well - too big?
3. 2PC
   Others?? Happy to contribute

[chenliu0831]

I find a good starting point for 2PC in Rust from UT-Austin CS Lab (https://www.cs.utexas.edu/~rossbach/cs378-f21/lab/two-phase-commit-cs378-process.html). Extra Credit involves handling some fault tolerance cases:

• Implement a recovery protocol. Allow a failed participant to rejoin.
• Support coordinator failure.

[mcches]

@chenliu0831 this has been open for awhile. If you aren't planning to implement any examples I'd like to close the issue.

[mcches]

Closing due to inactivity.