Skip to content

Morphium v6.3.6

Choose a tag to compare

@sboesebeck sboesebeck released this 21 Aug 18:04
· 30 commits to develop since this release

Fixed

PoppyDB: ordinary client disconnects no longer flood the log with ERROR lines (#331)

All three Netty exceptionCaught handlers (decoder, encoder, command handler) logged every
exception unconditionally at ERROR - including a plain Connection reset by peer whenever a
client dropped its connection. Deploys, restarts and load balancers do that all day: on the ACC
acceptance cluster a single reconnect-looping client produced 140 ERROR lines in 40 minutes, and
unconditional ERROR logging is a good part of how a poppy.log grew into the gigabytes. The three
sites now share one rule: the IOException family (reset by peer, broken pipe, timeouts) is
logged at DEBUG, everything else stays at ERROR with the full stack trace. Close behaviour per
handler is unchanged.

PooledDriver: a rolling restart can no longer erode the topology into permanent silence (#330)

During the ACC rolling restart exactly one of ~30 clients ended up permanently bus-dead while
looking perfectly healthy: zero heartbeat threads, zero log lines, HTTP alive. The chain behind
it: the membership-removal path compared the hosts map's NORMALIZED keys against
UN-normalized names from the hello - so a hello advertising a member in a different case
(SERV-MSG1 vs serv-msg1, the exact constellation normalizeHostKey's own comment documents)
removed the very host it had just added. A few such hellos during the takeover window eroded the
hosts map AND the running host seed to empty - and an empty seed made reseedIfAllHostsEvicted
a silent no-op: the heartbeat kept cycling over nothing, spawning nothing, logging nothing,
forever.

Four layers of fix, innermost first: the removal comparison now uses the exact same
normalization as the add path; membership REMOVAL is only accepted from the PRIMARY's hello
(the code comment always claimed this, the code never checked - secondaries and in-election
nodes answering with partial lists during a restart can no longer eat the topology; additions
stay accepted from every hello); the originally configured host seed is captured at connect
and restored - loudly - when the running seed has been eroded to empty; and the heartbeat
itself became self-rescheduling with a watchdog (silent-cycle detection with forced reseed,
dead-task revival, orphaned per-host bookkeeping cleanup), so even an unforeseen way of
stalling discovery now logs and recovers instead of freezing silently.

Test results

Phase Tests Passed Skipped Flaky Broken Runner Tested commit When (UTC)
inmem 2164 2150 14 0 0 testrunner a72a2c7 2026-08-21T21:15:38Z
mongodb_rs 1249 1235 14 0 0 testrunner a72a2c7 2026-08-21T21:15:47Z
poppydb_rs 1231 1222 9 0 0 testrunner a72a2c7 2026-08-21T21:15:54Z
mongodb_single 1249 1229 20 1 0 testrunner a72a2c7 2026-08-21T21:16:25Z
poppydb_single 1231 1217 14 0 0 testrunner a72a2c7 2026-08-21T21:16:32Z