CounterServiceDemo: concurrent mass increment blocks clients

[belaban]

When running 2 (or more) instances of CounterServiceDemo and incrementing the same counter from 2 instances at the same time (e.g. press [9] and pick 1000 increments), then both instances block.
A preliminary investigation showed that the issue is with REDIRECT.

[belaban]

Looks like REDIRECT doesn't block, but is just very slow when the leader itself performs a mass increment of the counter. If A increments by 100 and B 100'000, or vice versa, concurrently, then - as soon as the first instance is done - the other instance increments very quickly.

[belaban]

Looks like the culprit is RAFT.resend_interval which is 1000 by default. Lowering it (e.g. to 100) speeds things up dramatically. Investigating why resend_interval is causing this.

[belaban]

The reason why resend_interval is important is the following scenario:

• A sends messages with indices [34..38], B's prev_index is 33
• B receives 34, sets prev_index to 34
• B receives 35, sets prev_index to 35
• B receives 37, drops it as its prev_index of 36 doesn't match the current prev_index of 35
  ** The reason for the out of order message is that the threads in A grab indices and then the order of message delivery depends on which thread reached NAKACK2 first
• B receives 36, sets prev_index to 36
• B receives 38, but drops it as its prev_index of 37 doesn't match the current prev_index of 36

So messages with indices 37 and 38 will not get appended to the log of B until the resend_task kicks in (by default every 1000 ms).

Suggested solutions:

1. When sending a message in RAFT, extend the scope of the lock where the index is incremented until after the message has been sent. Downside: bad concurrency
2. When receiving a message that's out of order, return false to the sender, but still append the message to the log.

[belaban]

OK, I chose solution 1