Morphium v6.3.7

Added

Documented: InMemoryDriver is unsuitable for on-disk format tests (#336)

A value that reaches a driver unmapped - a raw LocalDate handed to InsertMongoCommand, a
hand-built $set operand - is normalised on the wire path and stored verbatim in memory. The wire
drivers serialise every command through BsonEncoder, so a real server holds whatever
decode(encode(v)) produces; InMemoryDriver has no encoder in that path and keeps the Java
object. It affects far more than java.time: Character->Integer, enum->String,
Short/Byte->Integer, Float->Double, int[]->List, Calendar->Date,
ObjectId->MorphiumId.

The trap is that it is invisible from query results: the in-memory driver leaves the stored
value and the filter unnormalised, so equality still matches. A format test asserting on query
outcomes passes against InMemoryDriver for the wrong reason - worse than failing, because
nothing points at the gap. It cost time twice while reviewing #333.

Nothing written through the normal Morphium API is affected: the ObjectMapper maps those values
before they reach any driver, and since #335 the update APIs do too. docs/howtos/inmemory-driver.md
now carries the full type table and the guidance to pin on-disk shapes against a real MongoDB (or
PoppyDB, which decodes off the wire and is unaffected). InMemoryWireShapeParityTest pins the
divergence - @Disabled until the in-memory write path is normalised in 6.4.0, verified red on
InMemDriver and green on a real mongod before being parked.

Opt-in: java.time types can be stored as native BSON Date (useBsonDateForJavaTime)

ObjectMappingSettings#setUseBsonDateForJavaTime(boolean) (default false) makes
LocalDate, LocalTime, LocalDateTime and Instant marshal to a native BSON Date
(type 0x09) instead of Morphium's own per-type formats — epoch-day / nano-of-day longs for
LocalDate/LocalTime, Doc sub-documents for LocalDateTime/Instant. The written value is
bit-compatible with the official MongoDB Java driver's org.bson.codecs.jsr310 codecs.

LocalDate is anchored at UTC start-of-day and LocalTime at epoch day 0 UTC, the same
convention the official driver's codecs use. Sub-millisecond precision is lost when the flag is
on, which is the same trade-off the driver makes for these types.

Scalar fields only. A scalar field becomes a bare BSON Date, so mongosh shows ISODate and
native date range/sort queries and TTL indexes work directly on it. Elements of a
List/array/Map field do not: they keep the {"value": …} wrapper the generic serialization
path produces for every scalar-returning custom mapper, with a native Date inside. Those values
round-trip correctly, but a native date query against a container has to address field.value,
and an index has to be declared on that sub-path.

The update APIs (set(), push(), addToSet()) consult the custom mappers with the same shape
store() uses, so they follow this flag as well (#335). Still not covered: raw
Doc.of("field", someLocalDateTime) calls that go directly through BsonEncoder; that low-level
encoder writes the legacy format regardless of this setting.

With the flag off — the default — nothing changes on disk. The write path is untouched at the
default, so documents stay byte-identical to previous versions and older versions keep reading
documents written by this one. Reading is tolerant either way: each of the four mappers accepts
both its legacy shape and a native Date, so a database written before or after flipping the flag
stays readable, and the flag can be switched at runtime on an already-constructed mapper (the
mappers read it through a supplier rather than copying it at construction time).

Changed

set() / push() / addToSet() now write the same on-disk shape as store() for custom-mapped fields (#335)

The update APIs routed values through MorphiumWriterImpl#marshallIfNecessary, which had no
custom-mapper branch: a custom-mapped value reached the driver unmapped. On the in-memory driver
the raw Java object was stored (unqueryable and unreadable), and on a real MongoDB the encoder's
hardcoded legacy branches masked it — but only for scalar fields at the default flag value.
Container elements split at both settings: store() wrote [{"value": 18997}] while
set("dateList", …) wrote [18997], so a query matching one document silently missed the other.

The update path now consults the custom mappers with exactly the shape store() produces per
structure position — bare mapper output for scalar fields, the {"value": …} wrapper (or the
map-with-class_name shape for map-returning mappers) for container elements, and it follows
useBsonDateForJavaTime dynamically. Documents written via set()/push()/addToSet() are
now byte-shape-identical to store()-written ones and read back fully typed.

Migration note: documents that were previously written through the update APIs into
container fields of custom-mapped types keep the old flat shape. Queries predicated on such
fields match store()-shaped documents; re-save affected documents once via store() if your data
contains them. Documents written by store() were always correct and need no action.

BigDecimal precision converges downward (deliberate). store() has always written
BigDecimal through its mapper as a lossy double; the update APIs previously bypassed that
mapper and wrote a lossless Decimal128 — so one field could hold two different BSON types that
both print as 12.34, an invisible split this fix removes. The cost: values written only
through set()/push() lose their extra precision from now on, matching store()'s long-standing
behaviour (tracked as symptom 2 of
#334, which widens from "affects store()" to
"affects every write path" with this change). Store amounts requiring exact decimal semantics
before relying on either path, or keep them out of custom-mapped marshalling until #334
addresses it.

Fixed

Dump-restored TTL indexes no longer crash peers' initial sync - full-cluster restart recovers again (#340 follow-up)

The #340 restore recreated indexes with the JSON parser's number types: expireAfterSeconds
(and every other numeric index option) was registered as Long instead of Integer. The
restore itself ran fine - the damage surfaced only when a PEER asked for the indexes:
listIndexes served the Long, the wire encoded Int64, and the syncing peer's
IndexDescription.fromMap threw IllegalArgumentException from its reflective field set,
failing the initial sync in an endless retry loop. After a FULL cluster restart on the
acceptance environment - every node restoring from its own dump, no healthy peer left to
sync indexes from - two of three nodes never left recovery and the cluster ran on a single
node. Every TTL index in the system (13 across all databases) was affected. A rolling
restart hides the bug completely, which is why no test caught it: restore and restart were
each covered alone, never the combination "restored from a dump, then queried by a peer".

Fixed on both sides, deliberately:

• Restore side: the recreated spec's known Int32 option fields (expireAfterSeconds,
  textIndexVersion, 2dsphereIndexVersion, bits, min, max) are normalized to
  Integer, so the wire serves Int32 again - which also keeps peers still running versions
  WITHOUT the hardening below alive in a mixed-version replica set.
• Receiving side: IndexDescription.fromMap now coerces numeric values against the
  declared field type (every Integer field had the same trap, and the wire can also carry
  Double there - mongosh sends plain number literals as doubles) instead of letting a
  harmless wrapper mismatch become a node that never comes back up. The Boolean-from-Int32
  tolerance is widened to any numeric wrapper; genuinely incompatible types still fail.
  En passant: fromMap now also finds fields whose leading underscore asMap() strips
  (2dsphereIndexVersion), which the round trip had silently dropped forever.
• Diagnosability: a node whose initial sync keeps failing with the IDENTICAL error now
  escalates to an unmissable NODE STUCK IN RECOVERY log line after five consecutive
  identical failures - the outage was diagnosable only from a per-attempt error scrolling
  past in one secondary's log.

The regression tests pin exactly the missing combination: restore from a dump fixture, then
run the indexes through ListIndexesCommand/fromMap the way a syncing peer does - plus a
full-replica-set E2E that stops ALL nodes at once, restarts them from their dumps, and
asserts every node returns to PRIMARY or a completed-sync SECONDARY, not merely that the
data is back.

InMemoryDriver: integral query values match across Integer/Long - a long field answers its own integer query again (#342)

find({counter: 2}) returned nothing for a stored 2L: equality compared by wrapper type,
so a long entity field never matched its own integer query literal - in everyday operation,
no restore involved. After a dump/restore it got worse: the JSON parser delivers every number
as Long, so even int fields stopped answering integer queries - no error, just empty
results. That made the #340 index fix only half effective: the index survived the restart,
but no integer query could hit it. MongoDB treats Int32/Int64 as numerically comparable, so
this was also a divergence from the backend being emulated.

The comparison now happens in the matcher (not by converting query values against the
declared field type - that would heal the long-field case but not restored data, which is
Long regardless of what the field declares), in every path that compared by wrapper type:
the interpreted matcher's direct-equality and multikey-contains branches, the compiled
matcher's equivalents, the compiled $in/$nin hash sets (which had silently diverged from
the interpreted $in already), and - critically - the index equality path: IndexKey now
canonicalizes Byte/Short/Integer to Long, so an index built over restored (all-Long) values
answers an integer probe instead of quietly shifting the bug from the scan path into the
index path. Comparison is exact via longValue(), never through double, so longs past 2^53
cannot collapse.

Scope, deliberately narrow: the new equivalence covers the integral wrapper types only -
Byte, Short, Integer, Long. Double/Float and BigDecimal are explicitly NOT included:
direct equality against a stored 2.0 behaves exactly as before (no match for {x: 2}),
because floating-point equivalence raises precision questions (1.0 vs 1) and the
BigDecimal side is #334 symptom 2, which is still open. Unchanged pre-existing behavior, for
the record: the $eq/$ne/$in(interpreted)/$lt..$gte operator paths have long compared
ALL numbers via doubleValue() and continue to; range scans over the ordered index side and
sorting were already numeric. This fix is not a general numeric-equivalence feature - it
closes the integral gap and nothing else.

setDatabase() no longer leaves stale index/TTL/capped bookkeeping of the replaced contents (#341)

setDatabase() - the wholesale replace under every dump restore - swapped a database's
collection map and touched nothing else. Seven derived per-namespace structures kept
describing the data that had just been replaced: index definitions, built index stores, the
TTL registration and its expiry queues, the capped config, the identity-keyed capped size
cache and the capped byte counters. A restore into a driver that already holds data (the
in-process PoppyDB restore case) then served indexed reads from documents that no longer
exist, kept listing indexes - TTL among them - that would never be enforced on the restored
data, and the identity-keyed size cache retained nothing but dead references to the replaced
document instances: a retention leak in the same shape as the poppydb commandResultsById
one fixed this week. It went unnoticed for so long because a restore into a FRESH driver
finds all seven structures empty.

The fix reuses the wholesale-invalidation contract that drop(String, WriteConcern) and
resetData() already follow (#290): discard the per-namespace structures for the replaced
database and bump the global indexStoreDropEpoch BEFORE removing the stores, so a
lock-free store build racing the swap cannot re-publish a pre-swap snapshot. The TTL queues
are REMOVED, never emptied in place - the sweep and the insert path only re-bootstrap a
queue that is null (#269), so an empty-but-present queue would pin restored documents in a
never-expires state. Index definitions are deliberately not carried over: setDatabase
cannot know whether they hold for the new contents; restore() recreates the ones its dump
carries right after the swap (#340), and a legacy dump now yields a driver state that is at
least CONSISTENT - no index listed that nothing enforces.

PoppyDB dump/restore carries index definitions - TTL indexes survive a full restart (#340)

A dump file held only the documents (data/_id/db), never the indexes. After a FULL
cluster restart - every node restoring from its own dump, no running peer left to copy indexes
from via initial sync - the data came back and every index was silently gone: TTL indexes
stopped expiring (on the ACC replica set the jef_servacc collections grew to ~9,500 documents
unnoticed), every query fell back to a collection scan on the hot messaging path. A rolling
restart hid the loss completely, which is why it survived so long: as long as one node stays
up, initial sync rebuilds the indexes on every restarted peer.

Dumps now carry an additional optional indexes section per collection, in the same
listIndexes/createIndexes wire shape the initial sync already replicates losslessly (#258) -
extracted into one shared describeIndexes() so the dump format and the wire format cannot
drift apart. The restore recreates the indexes after inserting the documents, deliberately:
createIndex seeds a TTL index's expiry queue from the documents present at that moment, and
the sweep never re-bootstraps a queue that merely came up empty - index-before-data would leave
every restored document permanently un-expirable, the same bug in a new disguise.

Compatibility holds in both directions, checked against the released readers: dumps without the
section (every pre-6.3.7 dump) restore exactly as before, and dumps with it are still readable
by 6.3.0-6.3.6, whose restore paths both ignore unknown top-level keys. The existing three keys
are untouched - a dump of a database without secondary indexes stays byte-shape identical to a
pre-#340 dump. A failed index recreation (e.g. a hand-edited dump) never costs the data or the
remaining indexes: the restore continues, reports the failures via
DirectoryRestoreResult.getFailedIndexes(), and PoppyDB logs an unmissable
INDEX RESTORE INCOMPLETE warning - an index set that looks complete but is not would be worse
than none.

A read preference stored via asProperties() silently reverted to nearest on reload

DriverSettings.defaultReadPreference was @Transient, and so was the defaultReadPreferenceType
string that could have carried it. A config that was written out with asProperties() and read
back with fromProperties() therefore lost the setting entirely and fell back to the class default
nearest() — no warning, no error, just reads drifting off to secondaries. On a replica set that
turns every read-after-write into a coin flip against replication lag: the write is acknowledged by
the primary, the immediately following read goes to a secondary that has not applied it yet and
comes back empty. Single-node deployments and the in-memory driver never showed it, which is
exactly why it could sit unnoticed.

The type name is now a normal, serializable field, and the preference object is rebuilt from it
whenever the two have drifted apart. That covers the properties round trip and createCopy()
alike: the latter goes through Settings.copy(), which drops transient fields just as
serialization does. Tag sets are still not part of the properties representation; a tagged
preference keeps its type across a round trip but loses its tags.

Morphium's own test suite was among the victims: TestConfig pins the read preference to primary
precisely so tests are deterministic, but MultiDriverTestBase builds each driver's config through
that same properties round trip and threw the setting away. Test reads in the MongoDB replica-set
phase ran against nearest wherever the entity did not carry its own @DefaultReadPreference —
the annotation wins over the config, which is why the effect stayed hidden for so long. It surfaced
with ScalarCustomMapperContainerTest (new in #334), an entity without such an annotation that
reads straight back after writing without any retry tolerance, and it failed only in that one
phase. MultiDriverTestBase now re-applies the preference explicitly, so the intent survives even
if the round trip loses something else in the future.

CHITSPERC/CMISSPERC reported NaN instead of 0 before any cached read had happened

Statistics.java computed CHITS/(CHITS+CMISS)*100 unconditionally; before any cached read has
happened both are 0, so the ratio was 0.0/0.0 = NaN. Prometheus/OTel exporters silently drop NaN
samples, so a fresh application's cache-hit-ratio metric appeared entirely missing instead of a
real "no data yet" 0%. Found while verifying the quarkus-morphium observability module against a
live otel-collector/Prometheus stack. Both percentages are now also computed by reading each
AtomicLong once instead of three times, so they come from one consistent snapshot.

PoppyDB: secondaries no longer leak ~800 bytes of heap per replicated event

Every InMemoryDriver.runCommand() stores its reply in an internal by-id map, and the entry
only ever leaves that map when the caller fetches it (readSingleAnswer et al.). The
ReplicationManager apply path called runCommand() and threw the returned message id away for
every non-bulk-insert operation — update/replace (the dominant type on a live bus), delete,
drop, dropDatabase, the idempotent replay-insert, plus the initial-sync insert batches and the
pre-sync database drops. The same pattern hid in WatchCursorManager.createWatchCursor,
which discarded the stub reply of every started change stream (one leaked entry per created
cursor — reconnect-looping messaging clients create them all day). On the primary the Netty
handler fetches every request's answer, so only secondaries leaked per-event — one abandoned
reply per replicated event, forever. Proven by measurement
on a local 3-node replica set: 20,000 update events on the primary grew the secondaries'
live-object count by exactly +1 java.lang.Double (the "ok": 1.0) per event after full GC,
while the primary stayed flat. At production rates (~800 bytes/event, 12 events/s) that is
roughly 0.8 GB/day until the node runs into the memory-watermark reject. All apply sites now
fetch their result the way the bulk-insert path always did — which also surfaces write errors
that used to be swallowed silently (logged, never thrown: an error reported inside a delivered
result must not make the apply path fail harder than before).

As defense in depth the driver itself no longer allows unbounded growth of the by-id result
store: command ids are strictly monotonic and a legitimate caller fetches its answer
synchronously in the same call stack, so an entry whose id lies more than a full window
(10,000 ids, -Dinmemory.maxPendingCommandResults) in the past is abandoned with certainty —
never "about to be read" — and gets evicted with a rate-limited WARN once the store exceeds
the window. resetData() now clears the store too (it was the one cleanup path that missed
it), and REPLY_IN_MEM in the driver stats finally counts these pending replies, which is
what the new regression tests assert on.

SingleMongoConnection: every heartbeat hello re-ran the full SASL handshake

getHelloResult() appended a complete SCRAM authentication to every hello, including
hellos sent over a connection that had authenticated long ago. MongoDB auth state is
bound to the socket and survives for its lifetime, so on an auth-enabled cluster this
produced one full SASL exchange per second per client on each pooled connection - all
of it pure overhead, and invisible as connection churn because the socket never
changed. Measured on a production replica set as ~7,200 Successfully authenticated
entries per hour per node on unchanged connection ids. Authentication state is now
tracked per connection and re-run only on a fresh socket (or after logout), which is
exactly when it is actually needed. SingleMongoConnectDriver was never affected - its
heartbeat uses a bare HelloCommand without the auth follow-up.

PooledDriver: idle long-lived clients no longer rebuild their connection pool every 30 seconds

A long-lived PooledDriver client with little or no application traffic tore down and rebuilt
its pooled connections permanently: measured in production on a 3-node replica set with ~22
long-lived Spring Boot clients, the nodes saw 1.48 (primary), 3.76 and 4.27 (secondaries) NEW
TCP connections per second - steady, for hours - amounting to 347,000 / 762,000 / 937,000
connection establishments over 61h while only 150-220 connections were ever open at a time.
The cause: lastUsed on a pooled connection is only refreshed by real application borrows,
not by the heartbeat hello that runs over it every second (deliberately so - otherwise the
heartbeat would keep every connection "warm" forever and maxConnectionIdleTime could never
shrink the pool after a burst). The idle sweep therefore declared every pooled connection of a
quiet client idle after maxConnectionIdleTime (30s default) and closed it - and the refill
loop immediately re-created it to satisfy minConnectionsPerHost. A full TCP handshake every
30s per pooled connection, forever, for a connection that was carrying healthy heartbeat
traffic the whole time. The hypothesis was verified experimentally against a local 3-node
PoppyDB RS: with 9 pooled connections and idle time 10s the reconnect rate was exactly
0.90/s (= pool size / idle time), a 10x longer idle time cut it to a tenth, and a 5x slower
heartbeat left it unchanged. The fix keeps both properties intact: idle eviction now only
shrinks the surplus above minConnectionsPerHost (bursts still drain back down), while the
base stock is recycled solely via maxConnectionLifeTime (10min default). Secondaries were
hit hardest because primaries stay warm through real borrows - matching the measured
primary/secondary asymmetry.

Container fields of scalar-mapped types (BigDecimal, Character, Atomic*, LocalDate, ...) now deserialize correctly (#334)

List/array/Map fields whose element type has a custom mapper with a scalar marshall()
result (BigDecimal, Character, AtomicBoolean/AtomicInteger/AtomicLong, LocalDate,
LocalTime, Timestamp, ...) are stored element-wise as a {"value": <scalar>} wrapper map
without class_name. The read path had no branch that recognised this shape: the raw wrapper
Map survived into the loaded container, so the first typed access
(BigDecimal.compareTo(...)) threw a ClassCastException — and typed arrays like
BigDecimal[] failed the whole entity read outright with array element type mismatch.

The fix is deliberately read-side only — the on-disk write format is bit-for-bit
unchanged. A write-side fix (dropping the wrapper, adding class_name) was tried in
PR #333 and measurably changed the stored document shape, which breaks rollbacks,
mixed-version operation against a shared collection, and indexes on field.value; a
read-side unwrap is purely additive: existing documents load correctly, new documents look
exactly like before, and older Morphium versions keep reading them. A new format-stability
test pins the written raw shape so any future write-side change fails loudly.

Unwrapping is generic over the registered custom mappers, not a hardcoded type list, and
deliberately narrow: a map is only treated as a wrapper if the declared element type has a
registered custom mapper and the map carries exactly the key value (plus at most a
class_name). Documents that legitimately contain a field named value — embedded objects,
untyped Map<String, Object> content — are left untouched, and if the mapper was
deregistered at runtime the read falls back to the previous behavior instead of throwing.

Test results

Phase	Tests	Passed	Skipped	Flaky	Broken	Runner	Tested commit	When (UTC)
inmem	2233	2217	16	0	0	testrunner	b8d68b8	2026-08-26T15:26:03Z
mongodb_rs	1289	1275	14	0	0	testrunner	b8d68b8	2026-08-26T15:26:11Z
poppydb_rs	1271	1262	9	0	0	testrunner	b8d68b8	2026-08-26T15:26:18Z
mongodb_single	1289	1269	20	0	0	testrunner	b8d68b8	2026-08-26T15:26:25Z
poppydb_single	1271	1257	14	0	0	testrunner	b8d68b8	2026-08-26T15:26:32Z