Skip to content

v0.9.1

Choose a tag to compare

@zcalusic zcalusic released this 27 Oct 18:39
· 61 commits to master since this release
v0.9.1

This release is about behaviour under heavy load and when nodes are down: no more dropped connections from long queues, and no more requests left without a reply.

Upgrade notes

  • A node no longer closes a client, datastore or peer connection because too many requests are queued on it. Instead, a client or peer sending requests is paused at 20000 outstanding requests (TCP pushes back) and resumed at 10000.
  • max_msgs now limits only new client requests: a client that would go over it is closed, with a warning in the log. Replies and requests from peers are always accepted.
  • A peer connection with 20000 requests queued refuses new ones with the new error -ERR Peer: Peer Node has too many requests queued, counted in the new stats peer_queue_full_requests and remote_peer_queue_full_requests.
  • Log changes: the startup lyrics and ASCII logo are gone (they put seven lines without a timestamp into journald and docker logs), and the exit line done, rabbit done is now dynomite stopped. Update anything that matches on them.

Fixes

  • DC_EACH_SAFE_QUORUM: with a remote DC down, or too few of its racks left for a quorum, requests never got a reply, and pipelined replies after them were stuck too. They now get Peer Node is not connected (or the error that stopped the forward) at once.
  • One client pipelining more than 20000 requests was reset after about 100 replies; many clients pipelining through one node got Datastore refused connection. Both now get every reply.
  • A stalled peer no longer makes its whole queue fail: requests already queued are replicated once it runs again.
  • Replicas that disagree under DC_SAFE_QUORUM and DC_EACH_SAFE_QUORUM could crash the node when no message was free; the "Failed to achieve Quorum" error is now always built.
  • At the max_msgs limit, replies from the datastore and from peers were refused too, which stalled or closed connections with unrelated errors.
  • Every close of a datastore or peer connection, and every failed reconnect, logged a spurious epoll ctl ... failed: No such file or directory error.

Testing

  • New unit tests (make check) for the quorum error and the max_msgs limit at an empty free list.
  • New functional test that stops one rack of a DC and then all of it under DC_EACH_SAFE_QUORUM.

Full Changelog: v0.9.0...v0.9.1