Repository navigation
v0.9.1
This release is about behaviour under heavy load and when nodes are down: no more dropped connections from long queues, and no more requests left without a reply.
Upgrade notes
- A node no longer closes a client, datastore or peer connection because too many requests are queued on it. Instead, a client or peer sending requests is paused at 20000 outstanding requests (TCP pushes back) and resumed at 10000.
max_msgsnow limits only new client requests: a client that would go over it is closed, with a warning in the log. Replies and requests from peers are always accepted.- A peer connection with 20000 requests queued refuses new ones with the new error
-ERR Peer: Peer Node has too many requests queued, counted in the new statspeer_queue_full_requestsandremote_peer_queue_full_requests. - Log changes: the startup lyrics and ASCII logo are gone (they put seven lines without a timestamp into journald and docker logs), and the exit line
done, rabbit doneis nowdynomite stopped. Update anything that matches on them.
Fixes
DC_EACH_SAFE_QUORUM: with a remote DC down, or too few of its racks left for a quorum, requests never got a reply, and pipelined replies after them were stuck too. They now getPeer Node is not connected(or the error that stopped the forward) at once.- One client pipelining more than 20000 requests was reset after about 100 replies; many clients pipelining through one node got
Datastore refused connection. Both now get every reply. - A stalled peer no longer makes its whole queue fail: requests already queued are replicated once it runs again.
- Replicas that disagree under
DC_SAFE_QUORUMandDC_EACH_SAFE_QUORUMcould crash the node when no message was free; the "Failed to achieve Quorum" error is now always built. - At the
max_msgslimit, replies from the datastore and from peers were refused too, which stalled or closed connections with unrelated errors. - Every close of a datastore or peer connection, and every failed reconnect, logged a spurious
epoll ctl ... failed: No such file or directoryerror.
Testing
- New unit tests (
make check) for the quorum error and themax_msgslimit at an empty free list. - New functional test that stops one rack of a DC and then all of it under
DC_EACH_SAFE_QUORUM.
Full Changelog: v0.9.0...v0.9.1