Skip to content

v2.21.0

Choose a tag to compare

@bpacholek bpacholek released this 08 Oct 07:36
· 113 commits to main since this release

Minor release. One of the library's operations whose own read brings a server's lame-duck INFO now waits for the failover only within its own deadline (#191). Before, the failover ran inline in that read, so a request() waiting for its reply, a SubscriptionQueue poll, a fetch or a flush() waited for the whole failover, dials, handshake, subscription replay and backoff included, before it met its own timeout.

Upgrading is recommended if you run a NATS cluster with rolling restarts or upgrades (a server put into lame duck mode sends that INFO) and connect with more than one server in the pool. There are no API changes. A few behaviours change, all listed in the upgrade notes of the CHANGELOG:

  • The operation times out at its deadline, or returns what a delivery brings meanwhile, while the failover carries on and announces the new connection when it is over.
  • With waitForReconnect: false the operation fails at once with Connection is not open, unless what it waits for came in the same read.
  • The messages read with the INFO are delivered by that read before the failover leaves the connection. A publish from one of their handlers is buffered and goes to the new server.
  • A requestMany() or a JetStream fetchBatch() whose read fails because the connection is going now returns what it has received instead of throwing; it throws only when it has received nothing.

Your own processIncoming() and readIncoming(), a serving loop's read and the heartbeat still run the failover inline. symfony-nats-messenger connects with reconnect off, so no failover runs there; its get() now receives the partial batch when the connection drops in the middle of a fetch.

Fixed

  • An operation whose own read brought a lame-duck INFO overran its deadline by the whole failover (#191). The read now starts the failover in a fiber of its own and waits for it only within the operation's deadline and wake-up, as a read that meets a lost connection does since 2.14.0 (#178). The rest of the chunk the INFO came in is still the leaving server's (#182): its PING and INFO are dropped, its -ERR is reported, and its PONG still completes the leaving connection's slot, so a flush() whose PONG came behind the INFO succeeds. On nats-server 2.12, with the second server's dial hanging until connectTimeoutMs, a request() with a 300 ms timeout whose read brought the INFO returned after 3.29 s; it now times out after 0.30 s with the failover under way.
  • A requestMany() or a JetStream fetchBatch() whose read failed because the connection was going threw away what it had received. This happened with waitForReconnect: false when the connection was lost or failed over, with any setting when the reconnect gave up, and with reconnect off when the connection dropped. A fetched message the server counted as delivered was redelivered only after the ack wait, or never on a consumer without acks. They now return the replies or the partial batch, as requestMany() already did when the connection closed and fetchBatch() when the heartbeats stopped.
  • A lame-duck failover could fail over a connection that a reconnect had opened while a DiscoveredServers or LameDuck listener was suspended. It now fails over only the connection the INFO came on, inline or in a fiber of its own.

Quality gates

  • PHPStan level 8.
  • 2948 unit tests (with data sets) on PHP 8.2 to 8.5, 149 live integration tests and 48 Behat scenarios.
  • All 45 runnable examples executed against a live server.
  • Statement coverage 98.83% from the unit suite (95% floor enforced in CI) and 98.89% combined (97% floor).
  • Infection covered MSI 93.0% over the 3736 mutants it ran (90% floor enforced in CI); it skipped 4311 of the 8047 it generated as too slow, so the score does not include them.
  • Every change has tests that fail on the old code. 60 new test cases, decided by the order of events with the failover's dial held: 41 fail on 2.20.0, and the other 19 pin what must not change.
  • The change was reviewed for semantics and for interleavings by two independent reviewers, and their findings were applied.
  • symfony-nats-messenger's main branch against this release: PHPStan level max, 528 unit tests, 52 functional scenarios against live NATS and 5 examples pass.

The full record is in the CHANGELOG.