Spurious NOLEADER in healthy clusters

[aphyr]

I'm not entirely sure what's up with this case--it looks like the cluster started up and formed normally, we joined all nodes to n1, n1 was the leader, then... immediately thereafter, n1 started insisting that there was no leader, and rejecting requests, while the rest of the cluster went on normally.

20200311T174200.000-0400.zip

To start, n1 is the leader:

 ({:id 958289600, :role :leader, :node "n1"}
 {:id 183312529, :node "n2", :role :follower}
 {:id 561617232, :node "n3", :role :follower}
 {:id 314213252, :node "n4", :role :follower}
 {:id 1472507146, :node "n5", :role :follower})

n1 executes a few transactions

2020-03-11 17:55:40,336{GMT}	INFO	[jepsen worker 15] jepsen.util: 15	:ok	:txn	[[:append 2 2] [:r 2 [2]] [:r 1 []]]

But a couple seconds later, it flips to NOTLEADER, then NOLEADER:

2020-03-11 17:42:13,330{GMT}	INFO	[jepsen worker 15] jepsen.util: 15	:fail	:txn	[[:r 2 nil]]	:notleader
2020-03-11 17:42:13,330{GMT}	INFO	[jepsen worker 30] jepsen.util: 30	:fail	:txn	[[:r 1 nil]]	:notleader
2020-03-11 17:42:13,336{GMT}	INFO	[jepsen worker 40] jepsen.util: 40	:fail	:txn	[[:r 0 nil] [:r 1 nil] [:r 0 nil] [:append 2 23]]	:noleader
2020-03-11 17:42:13,336{GMT}	INFO	[jepsen worker 25] jepsen.util: 25	:fail	:txn	[[:append 1 3] [:append 0 5]]	:noleader

After a few hundred seconds of this, we start getting socket timeouts on n1:

2020-03-11 17:42:22,466{GMT}	INFO	[jepsen worker 43] jepsen.util: 43	:info	:txn	[[:append 2 4]]	:socket-timeout

[aphyr]

We're still seeing this in 73ad833--nodes run fine for a while, then without faults, some of them decide there's no leader any more. Here's another test run

[yossigo]

Should be solved by #34.

[aphyr]

This looks fixed! We're still seeing occasional NOLEADER outages every minute or two with b9ee410, but they're short-lived now. :)