Morphium v6.3.6
Fixed
PoppyDB: ordinary client disconnects no longer flood the log with ERROR lines (#331)
All three Netty exceptionCaught handlers (decoder, encoder, command handler) logged every
exception unconditionally at ERROR - including a plain Connection reset by peer whenever a
client dropped its connection. Deploys, restarts and load balancers do that all day: on the ACC
acceptance cluster a single reconnect-looping client produced 140 ERROR lines in 40 minutes, and
unconditional ERROR logging is a good part of how a poppy.log grew into the gigabytes. The three
sites now share one rule: the IOException family (reset by peer, broken pipe, timeouts) is
logged at DEBUG, everything else stays at ERROR with the full stack trace. Close behaviour per
handler is unchanged.
PooledDriver: a rolling restart can no longer erode the topology into permanent silence (#330)
During the ACC rolling restart exactly one of ~30 clients ended up permanently bus-dead while
looking perfectly healthy: zero heartbeat threads, zero log lines, HTTP alive. The chain behind
it: the membership-removal path compared the hosts map's NORMALIZED keys against
UN-normalized names from the hello - so a hello advertising a member in a different case
(SERV-MSG1 vs serv-msg1, the exact constellation normalizeHostKey's own comment documents)
removed the very host it had just added. A few such hellos during the takeover window eroded the
hosts map AND the running host seed to empty - and an empty seed made reseedIfAllHostsEvicted
a silent no-op: the heartbeat kept cycling over nothing, spawning nothing, logging nothing,
forever.
Four layers of fix, innermost first: the removal comparison now uses the exact same
normalization as the add path; membership REMOVAL is only accepted from the PRIMARY's hello
(the code comment always claimed this, the code never checked - secondaries and in-election
nodes answering with partial lists during a restart can no longer eat the topology; additions
stay accepted from every hello); the originally configured host seed is captured at connect
and restored - loudly - when the running seed has been eroded to empty; and the heartbeat
itself became self-rescheduling with a watchdog (silent-cycle detection with forced reseed,
dead-task revival, orphaned per-host bookkeeping cleanup), so even an unforeseen way of
stalling discovery now logs and recovers instead of freezing silently.
Test results
| Phase | Tests | Passed | Skipped | Flaky | Broken | Runner | Tested commit | When (UTC) |
|---|---|---|---|---|---|---|---|---|
| inmem | 2164 | 2150 | 14 | 0 | 0 | testrunner | a72a2c7 | 2026-08-21T21:15:38Z |
| mongodb_rs | 1249 | 1235 | 14 | 0 | 0 | testrunner | a72a2c7 | 2026-08-21T21:15:47Z |
| poppydb_rs | 1231 | 1222 | 9 | 0 | 0 | testrunner | a72a2c7 | 2026-08-21T21:15:54Z |
| mongodb_single | 1249 | 1229 | 20 | 1 | 0 | testrunner | a72a2c7 | 2026-08-21T21:16:25Z |
| poppydb_single | 1231 | 1217 | 14 | 0 | 0 | testrunner | a72a2c7 | 2026-08-21T21:16:32Z |