Repository navigation
Design Choices
The readme explains that the happy path galloping state is optimal. Therefore, this implementation makes the following design decision:
Do not optimise the happy path for lost messages. Have nodes request retransmission.
In dealing with many real-world outages lead me to the “let it crash” philosophy:
Do not attempt to use any error recovery logic to deal with IO problems.
The safest approach is to follow and have something monitor and kill processes that get “stuck” due to infrastructure outages.
If this implementation sees something unexpected outside of a steady state, the TrexNode will first
clear any leadership state and return to being a follower. That may lead to the leader's recovery protocol
being rerun to ensure correctness due to lost messages.
If this implementation hits an IOException or anything else in the critical code, it marks itself
as “crashed” and refuses to do anything else. You need to kill the process to reinitialise everything from the
durable state of the Journal.
My final observation is that it is hard-won over a quarter of a century of working on distributed systems. Bugs always get through. The moment we know a bug exists, we usually find it within an hour and patch it. Yet recovery in production and rolling out the patch can take multiple days. We should aim to write brutally simple code with less surface for bugs to hide. We should do brute-force style testing on mission-critical code. This leads to this design choice:
Keep it simple and attempt a brute-force search for bugs.
Only the author's mistakes in the specification and implementation of the invariants and their tests will remain. The benefit of open source is that, with enough eyes, any errors can be found and fixed.