Morphium v6.3.15

Fixed

PoppyDB: a resume whose replay does not fit the cursor queue is answered ChangeStreamHistoryLost instead of looping kill/resume

A client that fell more than one queue (10,000 events or the byte budget) behind was killed, resumed with its last token, got the whole backlog replayed into the fresh cursor, overflowed again - one kill per second, no events delivered, the primary's heap spent on replays. The replay-backlog gate that answers 286 up front existed only for the replication watch and only checked bytes. MongoCommandHandler.replayBacklogGate now covers every resume by a PoppyDB _data token and checks events and bytes; a client stream is judged by its own namespace (InMemoryDriver.estimateReplayBacklog). Every gate reply's errmsg starts with ChangeStreamHistoryLost: so ChangeStreamMonitor discards the token and starts fresh. Pinned by the client cases in ReplayBacklogGateTest.

PoppyDB: cursor-kill log lines name namespace and client

cursor 1174 (geniosMessageBus.msg client 10.101.6.13:51736) instead of a bare cursor id, for the count-cap, byte-budget and global-budget kills; replication watch of <member> for a secondary.

Added

PoppyDB: cursor-queue-events

The per-cursor event count cap (previously a constant 10,000) as an option: --cursor-queue-events <n> / config key cursor-queue-events, default 10000, >= 1. The replay-backlog gate uses the same value.

Changed

listDatabases() throws when the driver is not connected instead of returning null

DriverBase.listDatabases() answered null for "not connected" - indistinguishable from "there are no databases". The method already declares MorphiumDriverException, so every call site was forced to null-check an API that can simply say what is wrong; Morphium.listDatabases() inherited the problem one layer up. Not being connected is a normal transient state (startup before primary discovery, a pool drained during a failover, connections the server just dropped), but it is not an empty result, and a caller that printed the list or counted it saw a plausible-looking nothing instead of the outage. It now throws MorphiumDriverException("Not connected - cannot list databases, no connection to <seed> yet"), and a reply that carries no document raises MorphiumDriverNoReplyException instead of NPE-ing on res.get("databases"). Callers that already treat the checked exception as an error (BasicAdminTests.getDatabaseListTest asserts non-null, the PoppyDB restore tests assert contents) need no change; code that relied on the null short-circuit must now catch the exception, which is the point of the change. Covered by PooledDriverDroppedConnectionTest.listDatabasesOnADisconnectedDriverThrowsInsteadOfReturningNull.

Fixed

WAIT_FOR_ALL_SLAVES on a one-member replica set writes w:majority, so the entity's own majority reads see the write

@WriteSafety(level = WAIT_FOR_ALL_SLAVES) derives w from the members the driver can reach. On a one-node replica set that collapsed to a plain w:1, which the server acknowledges as soon as the write is applied in memory - before the majority commit point moves. @Entity defaults to readConcernLevel = majority, and Query sends it on find and count, so a read right after store() could legitimately miss the write: on the single-node rs0 of the test matrix, ListTests.listStoringTest inserted 100 documents, countAll() 200 ms later answered 0, no error anywhere, and the documents were still there afterwards - the count had read a majority snapshot that the w:1 write had not reached yet (visible only under load, when journaling lags). With three members w is 3 and therefore majority anyway, which is why only the single-node phase ever showed it. Morphium.getWriteConcernForClass now sends w:"majority" when the set is one member: that is the same member, journaled - what "all available copies" means there anyway, and the only acknowledgement a majority read is guaranteed to see. The substitution is keyed on the configured set size, not on the reachable count, via the new MorphiumDriver.getReplicaSetSize() (default: seed size; PooledDriver: the hosts list from the last hello, arbiters excluded, dead members included). A larger set that is temporarily down to one reachable member - PSA with the secondary gone, or the detection window after a member loss - keeps the documented "reachable count" behaviour: the server computes majority against its configuration, so it is unsatisfiable there, the write would only run into wtimeout (and, see #408, come back as a success), and the majority commit point does not move either, so nothing becomes more visible. Pinned by WriteConcernStandaloneTest with a stub driver that sets set size and reachable count independently (1/1 → majority, 3/1 → w:1, 3/3 → w:3).

ok is judged by value, not by Java type, at every remaining call site - CommandStatus replaces a dozen literal comparisons that read plain wrong

Servers encode a command reply's ok as a double (1.0/0.0) on most paths and as int32 on others, PoppyDB's own error paths included. A dozen call sites compared it literally, and both failure modes were live at once: ret.get("ok").equals(0.0) reads an int32 ok: 0 as success, so MorphiumWriterImpl reported a failed update as done (seven call sites: update, push/pull, the chunked/bulk paths and createIndex), and checkForError()'s !get("ok").equals(1.0) read an int32 ok: 1 as failure, inventing Error: null - null for a perfectly good read reply (getAnswerFor(), i.e. every find/getMore). The new CommandStatus (three-way OK/FAILURE/UNKNOWN - the only place that knows how ok is encoded) now decides at all of them: MorphiumWriterImpl (7), ReadMongoCommand's error-batch check, ListIndexesCommand, SingleMongoConnection.checkForError(), InMemoryDriver's bulkWrite per-op results and - already covered by the entry below - readSingleAnswer(); SaslAuthCommand is deliberately stricter than the read paths: the authentication handshake fails closed on an UNKNOWN status, while reads let an inconclusive reply's payload through (checkForError stays silent there instead of warning on every single read). Pinned by CommandStatusTest (every numeric representation of 1 and 0, plus the wrong types, NaN, 2 and a missing field) and SingleMongoConnectionOkSemanticsTest (int32 and double ok: 0 is an error carrying its mongo code on both the single-answer and the read path, int32 ok: 1 is accepted on both, a reply without a usable ok is handed up rather than thrown).

A missing reply is a retryable network error instead of null - no more NPEs (and no more silently empty results) on a connection the server dropped

A connection that sits idle in the pool until the server drops it (its own connection-pool expiry, a restart, a load balancer's idle timeout) still looks connected locally, so the next borrower's request goes out on a dead socket and no reply ever arrives. SingleMongoConnection.readReplyFor() returned null for that - and for a reply that missed the caller's deadline - which the command wrappers turned into NPEs on the answer: listDatabases() (DriverBase), Morphium.getReplicaSetStatus(), currentOp, distinct, the SASL handshake, and the explain variants all dereferenced readSingleAnswer()'s result unguarded. On plain reads it was worse than a crash: getAnswerFor() mapped "no reply" to an empty cursor, so a find or a getMore silently ended a result set that the server had answered fine a moment earlier. The read path now closes the connection and throws MorphiumDriverNoReplyException, a MorphiumDriverNetworkException subtype, so every caller gets the retryable-network-error treatment it already had for a broken pipe - the write path re-resolves a connection and re-sends (its own null check was the only one that ever existed and stays as a belt for non-wire drivers), and NetworkCallHelper retries. Two callers that legitimately read the transport layer directly keep the old contract and got an explicit null guard instead: Query.tail() (a tail now ends when its connection is gone instead of NPE-ing on answer.getFirstDoc()) and the legacy single-connection driver's readBatches/getDbStats, which throw the same exception for their NetworkCallHelper callers. getHelloResult()'s sleep-and-retry loop went with it - after a missing hello reply the socket is closed, so all it ever did was sleep a second before failing on the closed connection, which delayed dead-host detection by exactly that second. While there, readSingleAnswer()'s ok check became type-safe: it used to be get("ok").equals(0.0), which NPE'd on a reply without ok and let an integer 0 pass as success - servers do produce an int32 ok: 0 on error paths, e.g. PoppyDB's election commands when election is not enabled on the node or handling a vote/append-entries request threw (a merely rejected vote or append is ok: 1 with voteGranted/success: false), and ElectionNetworkClient reads those replies through readSingleAnswer(); now any numeric zero is the error it is, and a reply with a missing or non-numeric ok is handed up as-is with a WARN naming the host, request id and keys instead of silently counting as success. Pinned by SingleMongoConnectionNoReplyTest (peer FIN, missed deadline, reply still delivered) and PooledDriverDroppedConnectionTest (a canned backend drops the pooled connections the way a server does: listDatabases() raises a network error instead of NPE-ing, a write recovers on a fresh connection, the pool refills).

Test results

Phase	Tests	Passed	Skipped	Flaky	Broken	Runner	Tested commit	When (UTC)
inmem	—	—	—	—	—	missing
mongodb_rs	—	—	—	—	—	missing
poppydb_rs	—	—	—	—	—	missing
mongodb_single	—	—	—	—	—	missing
poppydb_single	—	—	—	—	—	missing