Repository navigation
Releases: titusai-io/kimmydb
Release list
0.51.0 - 2026-10-17
Release Notes
Read before upgrading, three things for Decimal128 and a fourth for vector search.
(a) Breaking: a Decimal128 is compared, ordered and keyed by its exact value (ADR-238). A stored decimal used to rank equal to every number, so {v: {$eq: 999999}} matched {v: Decimal128("1.5")} and $ne could never exclude it. It now equals exactly the numbers of the same value, across Int32, Int64, Double and Decimal128, and Decimal128("0.1") is not Double(0.1): the double is slightly above one tenth. Filters, sorts, $expr comparisons, $cmp, $in, $indexOfArray, the $min and $max accumulators, the update $min, $max, $pull and $pullAll, and the $ne and $nin of update, delete and count change their answers: deleteMany({v: {$ne: 999999}}) now deletes the decimal documents. $group, $addToSet, $lookup join keys and the set operators identify equal numbers and decimals as one member. A unique index reports a violation between equal values of different types. $sum and $avg answer 400 on a Decimal128 operand where they skipped it and answered a wrong total.
(b) The first start of 0.51.0 migrates the store before it serves, and is a rollback boundary to 0.50 and 0.49 (storage schema 5; kimmyd restore of a 0.50 backup runs the same migration inside the restore command, before it reports restored). Every number key on disk changes, so every document with an integer _id is re-keyed and every non-vector index is rebuilt. Measured on the lab host, cold, 2,000,000 documents with integer _ids and three indexes took 15.6 to 19.3 minutes from the container's start to a serving node (the first run: re-key 6.3, index rebuild 11.8, oplog walk 1.1); a store with string or ObjectId _ids has no re-key. The start logs the time to plan from as a range, estimate_secs_low to estimate_secs, the upper figure being a spinning disk with a cold page cache; plan on the upper one. It needs free space: the start refuses, naming the figures, when there is too little, and a refused start leaves every row, the sidecar and the schema as they were, so 0.50 still opens the store (the open's own table commit, which 0.50 makes too, may have touched the file; a store with no free page can hit ENOSPC at that first commit on a filesystem that allocates a file's extended length, such as HFS+ or ExFAT). From the moment the sidecar is raised, a 0.50 or 0.49 build refuses the store (it leaves kimmy.redb and kimmy.format as they were and adds no file, but rewrites kimmy.last-exit to record the failed start, which a later 0.51.0 start reads with no effect); the way back is a backup taken before the upgrade, so roll with BACKUP=true. A member that is down for the migration for longer than storage.oplog_retention_secs returns by a whole-database snapshot. The rebuild reports every unique collision it finds, so a replication-merged duplicate that a schema 4 store already reported is reported again, once, on every member (/violations deduplicates it; change streams and webhooks do not). The first start also walks the oplog once. See operations.md.
(c) A Decimal128 _id, and a partial index with a Decimal128 bound, are refused until the operator declares cluster.decimal128. Set it only once every member runs 0.51.0 and has finished its migration, and as a change of its own, never in the change that ships the binary: a 0.50 or 0.49 member cannot decode a decimal _id and never retries a definition it refused. It defaults to true with clustering off and false with it on. A durable marker (kimmy_decimal128_id_present, and decimal128_id_present on /readyz) records that a decimal _id exists; once it does, a member can no longer be rolled back by wipe and resync. A 0.50 or 0.49 binary refuses a configuration that names cluster.decimal128.
Rolling. Member-at-a-time is safe. Every key on the wire, in a cursor token and in the cut digest keeps the schema 4 form, so a 0.50 and a 0.51 member compare and repair exactly as two 0.50 members do; a 0.51 member's tombstone horizon resets at its first start. Two differences outlast a member's own upgrade only until it migrates: a 0.51 member refuses or reports a decimal pair in a partial unique index that a 0.50 member accepts, and TTL deletes follow whichever member owns the collection's expiry.
Read before upgrading: the first start re-embeds every document with a binary _id, once. Vector chunks were stored under the _id's display text, under which every binary _id of one length was the same; they are now stored under an encoding that tells every two documents apart (ADR-239), and each server-embedded collection is rechecked: the owner re-embeds every document keyed by a binary _id (every UUID) once, one provider call's worth each, and moves the chunks of a String _id that begins with ~, is 24 lowercase hex digits or is an integer's decimal with no provider call. ObjectId, integer and every other String _id keep their chunks where they are. A single node rechecks at its first start; a cluster member rechecks what it owns at its first start, once its ownership has held, and again when every live member runs 0.51.0. Client-supplied (byo) collections keep their vectors, read as they are, until the client's next PUT. One loss during a roll: a 0.50.x member that deletes a document whose _id is a String equal to an integer's decimal or an ObjectId's hex ("7" beside 7) removes the integer's or ObjectId's vectors too, on every member; that document is re-embedded by the recheck when the roll finishes. operations.md has the roll and the rollback.
Added
- A
Decimal128has a place in the order. It is compared exactly against every number, sorted among them, indexable (aDecimal128at an indexed path is keyed, and answers range queries), usable as a filter operand, a sort key, an expression literal, a partial-index bound and an update operand, and can be an_id(an integral one that fitsi64is theInt64id of that value, with its stored_idunchanged). cluster.decimal128and the capabilityFacts.decimal128, with the gaugekimmy_decimal128_id_present(OTLPkimmy.decimal128_id_present), the/readyzfielddecimal128_id_present, and a WARN once per peer and every ten minutes while the marker is set and a live peer cannot hold such an id. A creation refused by the switch answers400and names it and the peers that block it./readyzreports each step of the migration with its own phase and count:preflight(the documents with a number_idthat the first 0.51.0 start on a 0.50 store reads to size the free-space check it makes before it changes anything; a store whose_ids are not numbers skips it, and a stop during it leaves the store as 0.50 left it),migrating(rows re-keyed),draining(entries removed from an index) andrefilling(documents walked to rebuild a collection's indexes).
Changed
- A number key is exact and variable length (ADR-238):
exp ‖ esc(M) ‖ END, whereMis the first 128 bits of the value's binary expansion with trailing zero bytes stripped andENDsays whether any later bit is nonzero. AnInt64key is 7 bytes where it was 12. See key-encoding.md. - A decimal at a partial filter's path is decided by the filter, by its exact value. A partial index holds exactly what its filter selects. A TTL partial index can now expire documents holding a decimal at its filter path that 0.50 never expired (
{tier: {$lt: 3}}overtier: Decimal128("1")), and the reverse: a document a$gtefilter selected only because its decimal ranked equal to the bound is now kept. The deletes are ordinary replicated deletes, irreversible except from a pre-upgrade backup, and the migration logsnow_selectedandnow_not_selectedfor each TTL index before the first expiry pass. A partial unique index enforces on decimals at its filter path: a write 0.50 accepted can be refused, and standing duplicates are reported on upgrade (/violations,kimmy_unique_violations_total, a log line) and then refuse an update that keeps the key. - An operator that takes a whole number takes a
Decimal128of whole value, as it takes the double of it.$arrayElemAt,$slice,$range,$substr,$size,$mod(its operand, and a stored decimal it is tested against, truncated toward zero as a double is),$limit,$skip,$push's$positionand$slice, sort directions and projection flags read a decimal as the double of the same value; a fraction, NaN or an infinity is refused where a double's would be. A decimal component of a$vectorSearchquery vector stays refused (it would have to be rounded to a 32-bit float). - A unique index can now report a violation between two values of different types that are equal (
Int64(1)andDecimal128("1.0")). - Creating a partial index with a
Decimal128bound is refused untilcluster.decimal128is set (see (c) above); a replicated, restored or repaired definition is always applied. /readyzgains a key (decimal128_id_present), and itsphasegainspreflight,drainingandrefilling: a client that pins the exact key set or enumerates the phases must add them.- The HNSW snapshot on disk is format 3, keyed by the id each chunk is stored under. A format-2 snapshot is refused and the collection's index rebuilt on its first search after the upgrade, once per vector collection on each member; a 0.50.x member refuses a format-3 one and rebuilds in turn.
- During a roll a 0.51.0 member reads a 0.50.x member's chunks for binary and reserved-String documents and moves or re-embeds them, while a 0.50.x member finds a 0.51.0 me...
0.50.0 - 2026-10-16
Release Notes
Read before upgrading, two things.
(a) Breaking: a database's drop now keeps the writes stamped after it (ADR-236), as a collection's drop does since 0.49.0. A write made into a collection of the database after the drop's stamp, through a member that had not yet heard of the drop, is kept, re-dated just above the drop, so the collection, and its database, can come back holding only the writes stamped after the drop. Clock skew can cause it. The way to avoid it is unchanged: confirm a drop on the members before writing through another one. A collection with nothing stamped after the drop is buried as before.
(b) The divergence check now repairs what it finds (ADR-237), on by default. A confirmed cut-digest finding plans a document repair, which writes a row only where it is strictly newer than the one held (and a young lost insert); a confirmed definition finding plans the build of one plain index this member lacks and a peer holds. Everything else stays a finding for you, in words, in docs/operations.md ("The cut digest" and "The definition digest"). A loop guard judges every repair and backs off one that changes nothing. cluster.divergence_repair = false makes a member detect only, as 0.49.0 does. Nine series report it.
Rolling and rolling back. Member-at-a-time is safe, and 0.50.0 is not a rollback boundary: no kimmy.format schema (still 4), I_EPOCH, table, backup tag or entry kind changed.
- During a roll a 0.49.x member buries a collection a 0.50.0 member keeps under a database drop, and refuses it from a 0.50.0 member until it is upgraded; it repairs nothing, and a 0.50.0 member repairing from it writes no young lost insert.
- Before rolling back to 0.49.x, remove
cluster.divergence_repair,cluster.repair_max_documentsandcluster.repair_max_index_documentsfrom the configuration: 0.49.x refuses to start on a configuration that names any of them. - A store rolled back to 0.49.x buries, at its next start, every collection 0.50.0 kept under a database drop, with the writes it kept, so the members differ until they run one version.
docs/compatibility.mdhas the two differences that outlast a roll.
Added
- The divergence check now repairs what it finds, in two cases. (
cluster.divergence_repair, on by default;falsemakes a member detect only, as 0.49.0 does. The count half's repair is unchanged and has no switch.)- A confirmed cut-digest finding plans a document repair: a scoped snapshot of the one collection applied as rows over rows the member already holds. A row is written only if strictly newer than the row held (live or tombstone); an id the member holds nothing for is written only if it is young (stamped after every tombstone cutoff either member has applied, so no delete of it can have been collected) and is otherwise counted
leftand logged, with the operator rule in operations.md. No definition is restored, nothing is dropped, no cursor is stored, no coverage is granted. - A confirmed definition finding plans a definition repair for one case: a plain index this member lacks by name and a peer holds, built from the first page of the peer's scoped snapshot, at most one per repair. A unique index, a TTL index, a rival under the same name, a vector configuration, an index the peer dropped, an index of a buried life, an index this member dropped, and an index at or below the collected-through horizon stay findings for the operator, each counted in
kimmy_sync_definition_repair_indexes_total{result}, and a repair that skips any index logs aWARN(a definition repair left indexes it will not build) with the count of each kind and the first ten index names. The build holds the single writer for its length;cluster.repair_max_index_documentsbounds it. - A loop guard keeps a repair that cannot change anything from running for ever and a repair that works from hiding a bug that keeps producing the divergence: a repair is judged at the first compared digest after it ends and backs off 5 min, 15 min, 45 min, 2 h 15 min and 6 h when it did not help (a repair that wrote a row at or below the finding's cut lowers the level to 1); more than three clear-and-recur episodes in an hour stop the collection for six hours, with an
ERRORandkimmy_sync_digest_repair_stopped_collections. - A repair never takes the peer's ordinary replication. After any document or definition repair pull to a peer, the next contact with it is an ordinary one; a document repair's round is at most 20 pages or 10 seconds, and a document repair ends, and is judged, after the collection's size at planning plus
cluster.repair_max_documents.kimmy_sync_repair_rounds_totalcounts the count half's repair rounds only; the two new repairs have their own series. The loop guard keeps a member's state across a restart or a short cut of the peer (a member is forgotten only after 12 hours out of the live member list), stops a one-peer loop however many collections the member holds, and backs off at once after an abandon (three rounds that landed nothing) or adroppedpage, and after three repairs in a row that end without a result for any other reason (a whole-database takeover, a cancellation while queued, a definition page without the collection). A plan the gates refuse (disabled,too_large) is counted once per standing finding, not at every contact. A refused definition build counts inkimmy_sync_ddl_refused_total. - New keys
cluster.divergence_repair,cluster.repair_max_documents(2,000,000) andcluster.repair_max_index_documents(250,000). New serieskimmy_sync_divergence_repair_enabled,kimmy_sync_digest_repairs_total{outcome},kimmy_sync_digest_repair_judgements_total{result},kimmy_sync_digest_repair_rows_total{result},kimmy_sync_digest_repair_stopped_collections,kimmy_sync_definition_repairs_total{outcome},kimmy_sync_definition_repair_judgements_total{result},kimmy_sync_definition_repair_indexes_total{result}, andkimmy_gc_tombstone_horizon_seconds. - A cut digest that compares equal quiets a suspended count finding of the same peer and collection, so
kimmy_sync_divergent_collectionsdoes not stay above 0 on a member taking steady writes after the two members hold the same documents (a count finding whose probe the gate keeps deferring is otherwise neither renewed nor cleared). The finding leaves the gauge and plans nothing, but the cooldown of its repair stands, so a peer that stops and moves in turn is not count-repaired at every stop; the next compared count probe clears it or confirms it again. - A collected-through horizon is kept: the cutoff of the last retention pass that removed a tombstone, raised inside each collector's removing transaction, reset to the start time on every start under a different build than the last (every upgrade and rollback) and by a restore, adopted by a completed whole-database snapshot, and offered on every snapshot page. After each upgrade there is a young-repair dead time: a lost insert older than the member's start is
leftfor good until an operator re-inserts it. - Rolling. Member-at-a-time is safe. A 0.49.0 sender sends no horizon, so a member repairing from it admits nothing young and leaves every id it holds nothing for. A 0.49.0 member repairs nothing. Rolling back:
ClusterConfigdenies unknown keys, so a 0.49.0 binary refuses to start on a config that names any of the three keys, at any value; remove them first. No stored data depends on them.
- A confirmed cut-digest finding plans a document repair: a scoped snapshot of the one collection applied as rows over rows the member already holds. A row is written only if strictly newer than the row held (live or tombstone); an id the member holds nothing for is written only if it is young (stamped after every tombstone cutoff either member has applied, so no delete of it can have been collected) and is otherwise counted
Changed
- Breaking: a database's drop keeps the writes stamped after it, as a collection's drop does (ADR-236, amending ADR-229's "writes and all"). A write made into a collection of the database after the drop's stamp, through a member that had not yet received the drop, was buried with the collection on that member and kept by a member holding a newer collection of the same name, and the count repair then replayed it back to some members and pages never did: the outcome depended on which member and which route, and two members could end holding different versions of one document for good. On every member and every route (entries, the count repair's replay, snapshot pages), for documents and for index and vector definitions alike, a collection the drop covers now keeps what is stamped after the drop and loses what is stamped at or before it, re-dated just above the drop. So a collection, and its database, can come back after a database drop, holding only the writes stamped after the drop, wherever such a write reached a member that had not heard of the drop; clock skew can cause it, as for a collection's drop since 0.49.0. To avoid it, confirm the drop on the members before writing through another one. A creation stamped at or before the drop is still refused everywhere, a collection with nothing stamped after the drop is buried as before, and the
written_after_the_dropwarning is gone (the write is kept, and loggedkept=true). No table, schema, wire field, backup tag or entry kind changed. Rolling back to 0.49.x: a 0.49.x member buries, at its next start or database sweep, every collection this release kept under a database drop, with the writes it kept, while 0.50.0 members keep them, so the members differ until they run one version; and during a roll a 0.49.x member that holds such a collection when the drop arrives buries it with the write, and refuses the kept collection from a 0.50.0 member's snapshot, until it is upgraded and takes it back from a peer.
Fixed
- A member taking a snapshot was count-compared by its peers, which confirmed a divergence and pulled a repair from it. A member part-way through a whole-database snapshot holds the pages it has applied as state its version vector does not name, and its vector stands still until the last pa...
0.49.0 - 2026-10-14
Release Notes
Read before upgrading, two things.
(a) A collection can now reappear after its drop. A drop that reaches a member holding writes stamped after it now keeps those writes (ADR-231, extended), where 0.48.x discarded them with the collection and left the members different for good. So when a write stamped after a collection's drop reaches a member that had not heard of the drop, the collection comes back on every member holding only the writes stamped after the drop. This is for collection drops only: a database's drop still buries a collection created at or before it together with the writes made into it afterwards (ADR-229). Clock skew can cause it: a write made earlier in real time by a member whose clock runs ahead counts as "after". The way to avoid it is unchanged: confirm a drop on the members before writing through another one. See docs/http-api.md, "A write made after a collection's drop is kept".
(b) The divergence gauge can read above 0 where it read 0, with a WARN. 0.49.0 adds the definition digest, a fourth comparison between members, of what a collection's indexes and vector configuration say. A finding moves the new kimmy_sync_definition_divergent_collections and is also counted in kimmy_sync_divergent_collections, so an alert on the existing gauge can fire where it did not. It is detect-only. Before acting on a finding, read docs/operations.md, "The definition digest": recreating a TTL index at a shorter value deletes the documents older than it on every member, and recreating a unique index files violations for the duplicates that exist. The first round can show one true positive, an index or vector configuration with no creation stamp that a late drop's sweep leaves (documented there).
Rolling and rolling back. Member-at-a-time is safe, and 0.49.0 is not a rollback boundary: no kimmy.format schema (still 4), I_EPOCH, table, backup tag or entry kind changed.
- A 0.48.x member ignores the definition request and is counted
unanswered, so definitions are compared between two 0.49.0 members only. - The two optional fields this release adds,
redated(in a collection's stored metadata) andredated_from(on a snapshot page's collection definition), are ignored by 0.48.x. Rolling back to 0.48.x is safe for the stored data. The refusal hints areMETArows 0.48.x never reads. - During a roll, old and new members still diverge in the case above (a drop meeting later writes): the 0.48.x member discards them as it did, and a write an old member has already destroyed is not recovered.
- Two things 0.48.0 cannot do for a store 0.49.0 has worked on. It has no start-time
reconcile_late_drops, so a crash in the middle of a late-drop sweep followed by a 0.48.0 start serves a half-swept life until 0.49.0 runs again. And it does not know where a re-dated life's first kept entry arrived (a 0.48.0 that rewrites the metadata drops the marker), so a change stream opened without a resume token on such a collection reads the retained log from the start to find it, the walk round 0500 measured at 62 s on a 1.8-million-row log.
Added
- The definition digest. Two members can hold the same documents and different indexes for good: a member that refused a replicated index lacks it while its peers hold it, and two members that created one name with different filters each keep their own, and nothing compared it. The divergence check now also compares a hash of each collection's index and vector definitions, at the cut the cut digest uses, on every checked contact, between two members that both have it. It is detect only: a confirmed finding moves
kimmy_sync_definition_divergent_collectionsandkimmy_sync_divergent_collections, logs aWARN(at most twenty per peer per tick, then a summary), and plans no repair. A member now remembers what it refused, in its metadata (a backup carries it), so a difference a refusal explains is counted asexplainedand never ascompared. New series:kimmy_sync_definition_divergent_collectionsandkimmy_sync_divergence_definition_probes_total{outcome,reason}(outcomescompared,explained,unanswered,deferred), with the OTLP series and the golden list. Read before upgrading: the divergence gauge can read above 0 where it read 0, because it now also counts a definition difference. A mixed 0.48.0/0.49.0 roll is safe: a 0.48.0 member ignores the request and is countedunansweredfor definitions, and is never compared. Not a rollback boundary: optional fields on the two divergence messages that an older member ignores, and rows in the metadata that an older build never reads. What to do with a finding, including that recreating a TTL index at a shorter value deletes documents, is indocs/operations.md, "The definition digest".
Fixed
- A collection dropped on one member and created again on another before the drop replicated left the members different for good, and the repair never finished. The second life had applied documents of the first before it learned of the drop, and the drop, judged aimed at a life already over, removed nothing. It now sets the life's incarnation floor at the drop and removes what that life took in of the one before (stamped at or before the drop), on every member that learns of it, by an entry, a snapshot page, a database drop's sweep or a start; a store left with such documents is healed before it serves. This reverses the residual ADR-229 accepted for database drops. The indexes and vector configuration of the first life are removed the same way.
- A write made after a collection's drop, through a member that had not yet applied the drop, was lost on that member and kept on the others, and the members stayed different. A drop that reaches a member holding the life it is aimed at now removes the records and definitions stamped at or before it and keeps the later ones, with the floor at the drop, so every member ends with the same documents and the repair ends (ADR-231, extended). A collection can therefore reappear after its drop: when a write stamped after the drop reaches a member that had not heard of it, the collection comes back on every member holding only those writes. Clock skew can cause it: a write made earlier in real time by a member whose clock runs ahead counts as "after". A database's drop is unchanged: a collection created at or before it is buried with the writes made into it afterwards (ADR-229). Receiver-local, no required wire field, table, entry kind or schema change, and not a rollback boundary: a build before it discards the later writes and ignores the two optional fields it adds (
redatedin a collection's stored metadata,redated_fromon a snapshot page's collection definition). During a roll the old divergence persists between old and new members, and a write an old member has already destroyed cannot be recovered. - An index a client was told existed could be swept away with the life before it. When a collection is dropped on one member and made again on another before the drop replicates, the first life's
create_index Xcan reach the second member before its own client createsX; that create found the definition standing and answered that it exists, and the first life's drop then removed it (ADR-231), leaving the client without an index and no entry replicating one. A local create of an identical definition over one stamped before the collection's own creation is now a new creation: it takes a fresh stamp and replicates aCreateIndex, and the members that hold the definition raise their stamp to it. The answer is the same as before (idempotent when the definition is identical), and an identical create over an index of the collection's own life still does nothing, so a client that creates its indexes on every boot mints no entry. A vector configuration already took a fresh stamp on every local configure and needed no change. The digest also no longer compares a collection that still owes ADR-231's sweep for a drop the cut covers. - The cut digest compared a recreated collection or database only after the drop's tombstone was collected. A drop's tombstone is kept for
storage.tombstone_retention_secs(24 hours by default), and 0.48.0 deferred a collection asincarnation(ordatabase_tombstone) for as long as either member held one, so on a cluster that drops and recreates collections most digest outcomes were deferrals and those collections were not compared (round 0510: 81% of outcomes). It now defers only while the drop is above the cut two members share, or while the standing collection is not newer than the drop. No false finding came of the old rule; this restores coverage.
Security
- rustls is on 0.23.46, which fixes four advisories (RUSTSEC-2026-0346, RUSTSEC-2026-0347, RUSTSEC-2026-0348 and RUSTSEC-2026-0349) against 0.23.45: a Brotli certificate decompression that could allocate up to 1 GiB before authentication, an unauthenticated ServerHello that could downgrade later connections to classical key exchange, application data buffered and sent to the outer server after an ECH rejection, and
Debugoutput of client session values that exposed TLS secrets. rustls carries the HTTPS listener, the member links and the outbound clients; the change is the lockfile only (no API, wire, store or configuration change), and rolling members one at a time is safe.
Download kimmyd 0.49.0
| File | Platform | Checksum |
|---|---|---|
| kimmyd-aarch64-unknown-linux-musl.tar.xz | ARM64 MUSL Linux | checksum |
| [kimmyd-x86_64-unknown-linux-musl.tar.xz](https://github.com/titusai-io/kimmydb/releases/dow... |
0.48.0 - 2026-10-13
Release Notes
Read before upgrading: the divergence gauge can now read above 0 where it read 0, and pages an existing alert. 0.48.0 adds the cut digest, a third comparison between members. It finds a document updated on one member and not on the other, which leaves every document count equal and which no earlier check saw. It is detect only: a confirmed finding moves kimmy_sync_digest_divergent_collections and kimmy_sync_divergent_collections, logs a WARN, and plans no repair. Roll the members one at a time. Nothing here is Breaking: the wire gains optional fields that a build before it ignores, two derived tables are added that a build before it ignores, and no kimmy.format schema (still 4), redb version, backup tag, snapshot field or entry kind changed, so a member rolls back by starting 0.47.2 on its store.
The first start of 0.48.0 walks the documents once, in the same pass that rebuilds the live counts, and so does the start after any crash or kill of 0.48.x (and after a clean stop of an older build, and after a start that was stopped before its walk completed). The walk is chunked, each chunk is committed, it honours a stop, and /readyz reports phase: counting with done and total. A stop during it resumes on the next start; a crash or a kill during it starts it again from the beginning. On a spinning disk, cold, it ran at about 10 MB/s, so a large store takes minutes. It also fixes a standing defect: the old rebuild ignored a stop and ran in one transaction, so a SIGTERM during a long counting phase became a SIGKILL after the stop budget.
Added
- The cut digest. One row per collection, kept in the transaction of every write: the XOR of a hash of each live record's
(_id, stamp), the highest record stamp per origin, and the highest stamp of a tombstone no oplog entry stands behind. Two members compare it at a cut, the highest stamp per origin both have processed; what either holds above the cut is left out of both, so a write one member has and the other has not yet received is never a finding. Finding the uncovered keys of a busy collection is one scan of the oplog above the cut, bounded to 200 ms and 8 MiB per request, one probe answered at a time per member. Before a mismatch is reported the member verifies its own row against a walk of the collection and corrects it if it drifted (kimmy_sync_digest_drift_total, anERROR). A collection is not compared, and the reason is counted, when a write landed between the plan and the comparison, when the scan spent its budget, or when either side holds a drop tombstone for its id (deferred{reason="incarnation"}, until the tombstone ages out). New series:kimmy_sync_digest_divergent_collections,kimmy_sync_divergence_digest_probes_total{outcome,reason}andkimmy_sync_digest_drift_total, with the same series in the OTLP export. What to do with a finding, including when the API cannot tell which member is behind and neither should be wiped, is indocs/operations.md, "The cut digest".
Changed
kimmy_process_resident_peak_bytes'sHELPtext says what the gauge is: the most this process has seen, not a reading that can fall.
Fixed
- A stop during the open's rebuild of the live counts was ignored. The rebuild is now chunked and honours the stop (above).
Download kimmyd 0.48.0
| File | Platform | Checksum |
|---|---|---|
| kimmyd-aarch64-unknown-linux-musl.tar.xz | ARM64 MUSL Linux | checksum |
| kimmyd-x86_64-unknown-linux-musl.tar.xz | x64 MUSL Linux | checksum |
0.47.2 - 2026-10-12
Release Notes
This release fixes kimmy_process_resident_peak_bytes, which could fall between two scrapes of a
running node. Roll the members one at a time. This release is not a rollback boundary: no
configuration key, table, kimmy.format schema (still 4), redb version, backup tag, snapshot field
or entry kind changed, and 0.47.1 opens a store 0.47.2 has run on, so a member rolls back by
starting it on 0.47.1.
Fixed
kimmy_process_resident_peak_bytesfell between two scrapes of a running node. The gauge was
read fromVmHWM, which the kernel reports as the larger of its high-water mark and the current
resident size and updates lazily, so a node whose resident size had just been its maximum could read
lower a few seconds later (249360384, then 248778752, with no restart). Every reading ofVmHWM
andVmRSSis now folded into a running maximum for the process, so the gauge never falls within
one process life, as the documentation says, and is never belowkimmy_process_resident_bytes.
Download kimmyd 0.47.2
| File | Platform | Checksum |
|---|---|---|
| kimmyd-aarch64-unknown-linux-musl.tar.xz | ARM64 MUSL Linux | checksum |
| kimmyd-x86_64-unknown-linux-musl.tar.xz | x64 MUSL Linux | checksum |
0.47.1 - 2026-10-12
Release Notes
This release fixes two things in the database-drop tombstone of 0.47.0: a WARN on every clean
drop, and a change stream opened at the start of a recreated collection that read the whole
retained log. Roll the members one at a time. This release is not a rollback boundary: no
configuration key, table, kimmy.format schema (still 4), redb version, backup tag, snapshot field
or entry kind changed, and 0.47.0 opens a store 0.47.1 has run on, so a member rolls back by
starting it on 0.47.0. Nothing here changes what a client is sent except that the second fix
answers the opens it used to answer too late.
Fixed
- A database drop warned for every collection it buried that held documents. The sweep that
buries what a database's drop covers loggeda database drop buried a collection that holds documentsatWARNfor any collection with a live document, so an ordinary clean drop warned on
every replica, once per collection. It now warns only when a document was written after the drop
(a database drop buried a collection that holds documents written after the drop, with
written_after_the_drop): a write a member that had not yet received the drop answered200and
that the drop then took. A collection whose documents all predate the drop is buried without a
warning. The look at the documents reads the fixed-width header of at most a thousand records and
copies nothing, so the warning sees the first thousand documents of the collection by id. - A change stream opened at the start on a recreated collection read the whole retained log.
Finding where the collection's current life begins walked the arrival index from its start to the
first entry stamped after the collection's drop, once per stream: on a log of 1.8 million entries
that was 1.8 million rows and a minute, and aGET .../watch?from_start=trueon a recreated
collection answered no events within a client's timeout while the node loggeda change stream took over 10 s to find where it starts. The start of a recreated collection is now found just after its
drop's entry and reads a few rows however long the log. The events delivered are the same, and a
resume token issued before a database's drop still resumes. A collection that survived a database's
drop (created here before the drop reached this member), and one this member holds no creation or
drop entry for, are placed by the walk as before.
Download kimmyd 0.47.1
| File | Platform | Checksum |
|---|---|---|
| kimmyd-aarch64-unknown-linux-musl.tar.xz | ARM64 MUSL Linux | checksum |
| kimmyd-x86_64-unknown-linux-musl.tar.xz | x64 MUSL Linux | checksum |
0.47.0 - 2026-10-12
Release Notes
This release makes a dropped database stay dropped. DELETE /v1/db/{db} used to drop the
collections the member answering it held, and a collection a peer had created and that member had
not yet received brought the database back on every member, with the deleted data in it. The drop
is now a stamped tombstone of the database that replicates, and every member drops what the
database held when it was stamped. That changes what the route means, so this release has one
Breaking change, below. There is no rollback boundary: no configuration key, table,
kimmy.format schema (still 4), redb version, backup tag, snapshot field or entry kind changed, and
0.46.0 applies the new drop as the drop of a collection it never held (Rolling
back).
Read before upgrading:
- Breaking: after a database drop, recreate the database on the member that dropped it, or wait
until every member lists it gone.DELETE /v1/db/{db}now also deletes, on every member, the
collections created before it that the answering member had not received, with their documents.
A creation on a member that has not received the drop can be stamped before it and is dropped
with it, with the documents written into it afterwards, and those writes were answered200.
Details in the first entry under Changed. - A roll or a rollback must finish within
storage.tombstone_retention_secs(default 24 h) of
any database drop made during it. A member on 0.46.0 keeps a collection the dropper never
held, and keeps serving it, until it restarts on 0.47.0; it needs the drop's tombstone to then
bury it, and every member collects that tombstone after the retention. Past it the collection,
and the database, come back on every member. - A
partially_appliedanswer from a database drop can carry an emptydroppedlist: the
drop's tombstone is its first commit, so the drop is recorded and replicates even when no
collection has been buried yet. - A start buries what a database's drop covers before it serves, and
/readyzshows
phase: reconcilingwhile it does (the OpenAPI document lists the phase). A store left by a
member that applied a drop as a plain collection drop, or that stopped between the tombstone and
the burials, is finished then. A start that cannot bury for a reason other than a stop fails with
the cause: runkimmyd check-storeon the stopped member, and if the store is damaged wipe its
data directory and let it catch up from its peers.
Changed
- Breaking:
DELETE /v1/db/{db}also deletes, on every member, the collections created
before it that the member answering it had not received (ADR-229). The drop is a stamped
tombstone of the database: on every member, every collection of it created at or before that
stamp is dropped with its documents, and none can be created at or before it. A collection
created after it stands. Recreate a dropped database on the member that dropped it, or wait
until every member lists it gone: a member that has not received the drop can stamp a creation
before it, and that creation is dropped too, with the documents written into it afterwards. The
same holds for a write acknowledged into such a collection on a member that has not applied the
drop, and for one on a 0.46.0 member in a roll. Resurrection is no longer possible except for a
creation concurrent with the drop (within clock skew, replication lag or the time the drop takes
to bury its collections) and for a member behind longer than the tombstone's retention. The sweep
logs each collection it buries that held documents (a database drop buried a collection that holds documents). A member that receives the drop of a very large database converges as fast
as it did on 0.46.0, and a stop during the sweep is clean.
Fixed
- A collection a member never held, with vectors, stopped replication at its first vector
chunk after its drop until a snapshot: the drop recorded the collection's tombstone and not
its shadow's, so the chunk was an entry for an unknown collection (ADR-229). - A change stream or the vector cache on a collection buried by a snapshot page or a
replicated drop with no entry of its own was never ended, or never forgot the graph; both
are now told (ADR-229).
Download kimmyd 0.47.0
| File | Platform | Checksum |
|---|---|---|
| kimmyd-aarch64-unknown-linux-musl.tar.xz | ARM64 MUSL Linux | checksum |
| kimmyd-x86_64-unknown-linux-musl.tar.xz | x64 MUSL Linux | checksum |
0.46.0 - 2026-10-11
Release Notes
This is a minor release with no rollback boundary: no configuration key, table,
kimmy.format schema (still 4) or redb version (4.3.0) changed, and the wire gained only
optional fields and one list that 0.45.0 skips (Rolling
back). What is new is that a member that caught up by a
whole-database snapshot gives its peers, and receives, what a snapshot used to leave out, and
serves only the history it actually holds:
- A snapshot-caught-up member no longer relays a window it cannot serve (a vector
configuration, a collection drop, a collection creation), and the snapshot now carries
index drops, a vector configuration turned off and deletes of documents a member
never held (ADR-224, 225, 226, 228). Newkimmy_sync_horizon_state_held_total. - A change stream never skips events silently across a collected range or a snapshot's
hole: it is refused withResumeTokenExpiredinstead (ADR-173's addendum). - A member could be stuck catching up indefinitely on a snapshot page that changed a
client-supplied vector width; it no longer is (ADR-227). dns:andk8s:seeds resolve with the resolverdns-srv:uses, so a stop does not
wait for a lookup (ADR-223).
Read before upgrading (none is Breaking in the API, wire, disk or configuration sense):
- A missing
resolv.conf, or one with no nameserver, is now an error fordns:and
k8s:seeds (ADR-223). - A member behind a snapshot-caught-up peer's grant may take a whole snapshot from it,
where it was served entries across history the peer did not hold: the peer answers
BeyondHorizon, the member asks its other contacts first, and snapshots only if none can
serve it, at most one tick late (ADR-224). - A retried
disableof a vector configuration that is already off now writes and
replicates an entry, and the entries path is last-writer-wins on the configuration's stamp
(ADR-226). - Change-stream clients resuming across a member that caught up by snapshot may get
ResumeTokenExpiredand must re-list (the refusal is conservative: one maximum per
origin) (ADR-173's addendum). - Holes from snapshots taken on 0.45.0 (or before this release) are not known, so a member
that caught up that way serves across history it holds only as state until it is rebuilt from
a peer.
Changed
- A
kimmydstopped while adns:ork8s:seed is being looked up no longer
waits for the lookup (it held the stop for up to the 22 s budget when the name
server did not answer). These seeds now resolve with the librarydns-srv:
already uses, one resolver per lookup, so/etc/hostsand/etc/resolv.confare
read each time and nothing is cached between discovery ticks, as with
getaddrinfo. What differs, so that a seed that resolved still does:
resolv.confis honoured for its nameservers,search/domainandoptions ndots/timeout/attempts/edns0; a missingresolv.conf, or none with a
nameserver, is now an error (it was a resolver on127.0.0.1);nsswitch.conf
is not read (the static musl build and so the container never read it; a glibc or
macOS build losesmdns/sssd/scoped resolvers); and a seed is given the timeresolv.conf'stimeoutandattemptsallow for the
names it tries (at least 15 s) and is then reported and tried again at the next
tick, where it used to wait on the resolver's own timeouts. Seeds are resolved
together, so several that do not answer cost one bound in all. No rollback boundary and no configuration
change. The webhook and embedding-provider lookups still usegetaddrinfo.
ADR-223; amends ADR-202. - New test switch
KIMMY_TEST_DNS_SERVER=<ip>:<port>(docs/testing.md,
docs/operations.md): it reachesdns:andk8s:seeds, notdns-srv:. Never set
it in a deployment; it logs aWARNat start.
Fixed
-
A change stream resumed across a member that caught up by snapshot could silently miss a
delete. A member that took a whole-database snapshot holds the sender's documents but not the
events inside the span, so a client resuming on it with a token below the span was sent the
current versions and never the delete of a document the member had not held: its cache stayed
wrong with nothing to say so. Such a token (bare or vector) is now refused with
ResumeTokenExpired, and the client re-lists, as for any history a member no longer holds. A
token that covers the span still resumes. Clients resuming across a snapshot-caught-up member
may now getResumeTokenExpired; the refusal is conservative (one maximum per origin). ADR-173's
addendum amends its "no gap on another member" claim. No API, wire or disk change. -
A bare-stamp resume token inside a range retention had collected could resume and skip
events silently after an old entry arrived late. Such a token (issued before 0.30.0, or
start_ata stamp) was refused when it was older than the oldest entry the oplog retains, and
an entry stamped before everything retention collected (a delete of a document the member never
held, by entries) could arrive late and become that oldest entry, lowering the bar; the token
then resumed with the collected events missing. It is now also refused when retention has
collected an entry above it, from the recorded per-origin horizon, which a late arrival cannot
lower. A token above everything collected still resumes. ADR-173's addendum. No rollback
boundary and no wire or disk change. -
A delete now reaches a member through a member that never held the document. A
member that took a whole-database snapshot recorded nothing for a document it had never
held and the sender had deleted, so a member that then took its snapshot kept the
document: detected by the count check, never repaired. The snapshot receiver now records
the tombstone, and aDeleteentry that arrives for a tombstone it already holds under
the same stamp is appended to the oplog when it is missing (one extra read, only on that
path) instead of being taken as already applied. It does not change a member whose pull
a completed snapshot's grant already covers; the hole (ADR-224, below) is what handles that.
ADR-228. No rollback boundary: the tombstone is an ordinary one. -
A member could be stuck catching up indefinitely when a snapshot page changed the width
of a client-supplied vector configuration, and now it is not. A member holding abyo
configuration of one width, whose peer turned it off and configured another (the only way to
change it), refused the snapshot page in every round: the snapshot never completed, the
catching-up marker stayed set, and the member refused requests until the origin changed the
width back or the member's store was wiped. A page's dated configuration now applies as the
off-then-on it stands for, in one transaction, keeping the stored vectors (which the new
width skips); any other configuration a page carries that this build cannot hold is now
skipped, logged and counted inkimmy_sync_ddl_refused_totallike an index it cannot
build, instead of failing the page. ADR-227. No rollback boundary: a 0.45 receiver still
refuses the page. -
A snapshot now carries the indexes the sender has dropped. A member that took a
snapshot (or a scoped repair) from a member that had dropped an index kept the index,
so a unique index the cluster had dropped went on refusing inserts on that one member,
a TTL index went on expiring documents, and nothing detected it. The page now lists the
sender's index-drop tombstones; the receiver judges each as a replicatedDropIndex
(an index created after the drop is left alone), records it, and serves it onward.
Together with the entry below about a member that caught up by snapshot (ADR-224), a member behind a snapshot-caught-up peer gets the drop
too. An older sender writes nothing and an older receiver ignores the list, so a mixed
cluster keeps the old behaviour until every member has rolled. ADR-225. No rollback boundary. -
A snapshot now carries a vector configuration that was turned off. A member that took a
snapshot (or a scoped repair) from a member that had turned a collection's vector
configuration off kept it, and went on embedding, refusing chunk writes and answering vector
search from a configuration the cluster had dropped; nothing detected it. A configuration now
carries the stamp of its latest change, a page says when it was turned off, and the receiver
applies the later of its own and the page's. Two behaviours change with it: the entries
path is last-writer-wins on that stamp (two concurrent changes from two members now converge on
the later one, and a re-served older entry no longer flips a configuration back), and a
disableof a configuration that is already off is now stamped and logged, so a retried
disable writes (and replicates) where it used to do nothing. An unstamped state written before
this release counts as older than any stamped change, so a 0.45-era change made after the
origin's off is lost when a page arrives. Together with the entry below about a member that caught up by snapshot (ADR-224), a member behind a
snapshot-caught-up peer gets the off too. An older sender writes no stamp and an older receiver
ignores it. ADR-226. No rollback boundary. -
A member that caught up by a whole-database snapshot no longer serves a peer a
window across history it holds only as state. It held the sender's documents as
entries and its schema changes, drops and deletes as state only, yet advertised
the whole span, so a peer behind that span was served the documents without the
changes beside them, witnessed past them, and never asked again. Seen on a real
cluster loop, silently, for a vector configuration turned on or off, a collection
drop, an index drop and a delete the member never...
0.45.0 - 2026-10-06
Release Notes
This is a minor release with no breaking change and no rollback boundary: no
configuration key, table, kimmy.format schema (still 4), redb version (4.3.0)
or wire frame that 0.44.0 cannot read changed, and every answer that differs
from 0.44.0's is a request that was refused before and works now. What is new:
- A
$lookupsub-pipeline's$matchreads theletnames through$expr, and
a correlated equality is a hash join (10,000 inputs against 100,000 foreign
documents: 104 s to 86 ms), withexplainonaggregateto see the plan. - A member whose single writer is wedged yields what it owns and says so in
/v1/topology(degraded), and does not take the work back while the writer
is still held (ADR-220). - A restored member reads its lost writes back over its peers' contacts as well
as its own, so a filter that drops only what it opens no longer leaves it owing
the replay (ADR-212's addendum);kimmy_replay_totalcounts it. - A
$letvariable the body does not read can no longer fail the expression
(ADR-219), and a write a batch or a unique index refuses no longer advances the
clock.
Rolling back to 0.44.0: nothing to do. A three-member cluster on this build wrote
documents, held a writer wedge, restored a member from a backup (and replayed its
lost writes), and 0.44.0 started on the same data directories and configuration
with no repair and no refusal, held the same counts and replicated writes; 0.45.0
then started on them again and did the same. What 0.44.0 does not know of: the
new optional replay_from field on Vectors and writer_wedged in the facts
block, which it ignores; the ReplayEntries and ReplayAck frames, which a
member sends only to a peer that asked, and 0.44.0 never asks; degraded in
/v1/topology, which a client ignores. explain on aggregate is a 422 on
0.44.0, and a let name in a sub-pipeline $expr a 400, as before. A mixed
roll answers by member until every member is rolled.
Added
- A
$lookupsub-pipeline's$matchcan read theletnames, and a correlated
equality is a hash join.{"$match": {"$expr": {"$eq": ["$order", "$$oid"]}}}
now works in a sub-pipeline (it was a400), at any depth of the filter and in any
$matchof it, and a$matchwhose conjunction holds such an$eq(either
argument first, any expressions) is answered from a table of the foreign
collection instead of a nested loop: 10,000 inputs against 100,000 foreign
documents went from 104 s to 86 ms in a release build. The first input document
is run as the nested loop, anything that would raise falls back to it, and the
join never answers or fails differently from it. The equality is$expr's$eq:
numbers across types are equal,nulland a missing field are the same value
(unlike thelocalFieldform, where a missing key joins nothing), arrays compare
whole. Shapes that do not join ($or,$gt,$elemMatch) run as the nested
loop. A$$nameas a plain value of a clause is still refused.explainreports
foreignScanHashJoin, with the key, the residual clauses and the fallback. No
action on upgrade; an older member answers400during a roll. See
aggregation.md, "Thelet/pipelineform", and ADR-222. /v1/topologysays when a node cannot serve. A node's entry gains an
optionaldegraded:"writer"while its single writer is wedged (below) and
"runtime"while its async runtime has stalled. It is advisory (prefer a node
without it), appears only on alivenode, and its absence is not a promise of
health. New series:kimmy_writer_wedgedandkimmy_writer_hold_age_seconds.
A new test switch,KIMMY_TEST_HOLD_WRITER=<ms>,<holder>. See operations.md,
"A wedged writer", and ADR-220.aggregatetakesexplain: trueand answers with a plan, running nothing.
{"pipeline": [...], "explain": true}answers{"explain": {"executed": false, "source": ..., "stages": [...]}}with no documents and no count: the leading
$match's access path asfindwould take it (strategy,index,
indexFieldsUsed,probes, without the counts a scan finds), one entry for each
later stage, a$lookup's strategy, and a$vectorSearch's configuration. It
needs the access a run needs,$lookupcollections included. It was a422
(explain: unknown field); an older member still answers that during a roll.
The MCP aggregate tool takes it too. See aggregation.md, "Performance", and
ADR-221.
Changed
- A member whose writer is wedged now yields what it owns, and no peer hands it
work. One hold of the single writer that lasts more than 60 seconds used to
delay every class of owned work (expiry, webhooks, embeddings) on the member and
be detected by nothing. It is now a bad item for every class that owns work, so
the member yields them to a peer by the usual rules, and a member that advertises
it (writer_wedged, optional on the wire) is refused as a target for every
class. Index builds and index drops are never read as a wedge, which is a stated
gap. An index drop now has a holder of its own,index_drop, in
kimmy_write_lock_*{holder}(it wasdrop).responsivein a member's block is
now absent until six ticks have been judged, where it read false. No rollback
boundary. - A restored member now reads its lost writes back over its peers' contacts
too. A member restored from a backup asked the peers that hold its later
writes only on connections it opened, so a firewall that dropped what it opened
and let in what the peers opened left it owing the replay (serving as
unknown,owed_replay) with the writes one hop away. It now asks on the
contacts its peers open as well. No configuration, and no action on upgrade:
a 0.44 peer ignores the ask and the member waits as before. Counted on
kimmy_replay_total{via,result}. A member that can dial nobody still cannot
clear. See operations.md, "Serving asunknownbecause a member owes the
replay", and ADR-212's addendum.KIMMY_TEST_REFUSE_DIALis a new test switch. - A
$letvariable the body does not read can no longer fail the expression.
$letevaluated every variable before its body, so a guard that kept the body
from reading a variable did not keep the variable from failing:{$let: {vars: {r: {$divide: [1, "$n"]}}, in: {$cond: [{$eq: ["$n", 0]}, 0, "$$r"]}}}was a
400overn: 0and now answers0. A variable is evaluated when the body
first reads it, once, and a variable nothing reads is not evaluated at all.
Only requests that failed are affected: an answer that was given is the same.
When two variables would fail, the error is now the first one read rather than
the first one written. A$lookupletstill evaluates every variable it
declares. See aggregation.md, "Variables", and ADR-219.
Documented
- When a schema-change confirmation answers
backoff. A confirmation that
arrives during a member's back-off is answered at once, as ADR-191 says; one
already waiting behind the push that failed is answeredbackoffwhen that
push fails, so the answer can take up to the request timeout. See
operations.md,kimmy_ddl_confirmations_total.
Fixed
- A write a batch or a unique index refuses no longer advances the clock.
A batch insert whose later document collided, or repeated a unique value,
still ticked the hybrid clock for the documents before it, and an insert, a
replace or afind_and_modifythat a unique index refused ticked it too,
because the refusal was found after the stamp was minted (the two leftovers
of 0.44.0's entry on writes that find nothing to do). A batch now checks every
document, in order, against the stored documents, the ids before it and the
indexes, and mints a stamp for each, in submission order, only once all have
passed; a single write files its index entries before it mints. Nothing a
caller sees changes except the clock: the same request fails with the same
answer. Still leftover: amultiupdate chunk, or a scoped batch, whose
later document fails (an update operator that cannot apply to it, or a unique
value) aborts after the documents before it in the same chunk were minted
for, and so still ticks the clock; the abort writes, logs and publishes
nothing.
Download kimmyd 0.45.0
| File | Platform | Checksum |
|---|---|---|
| kimmyd-aarch64-unknown-linux-musl.tar.xz | ARM64 MUSL Linux | checksum |
| kimmyd-x86_64-unknown-linux-musl.tar.xz | x64 MUSL Linux | checksum |
0.44.0 - 2026-10-05
Release Notes
This is a minor release with breaking changes. Several requests that used to
answer 200 with a quietly wrong result are now refused with a 400, and a few
answers that were wrong are now right, so a client that depended on the old
answer sees a change. Read the Breaking: entries below before rolling.
Breaking changes in this release:
- An array written inside an expression is evaluated, so a field path in it is the field's value.
$incand$mulkeep a 32-bit integer 32-bit, andPUTkeeps_id's type.modifiedcounts the documents an update changed, and a document left as it was is not written (and has no change event).- A
$lookupjoins on every element of an array its key crosses or holds, and its attachments are held to the ceiling. - A projection through an array keeps the array and every element.
- An update that writes one path twice is a
400and writes nothing. - A filter's
$exprthat cannot be evaluated against a document fails the request with a400instead of dropping the document. - A
$regexpattern, and the patterns of one filter together, have a size limit. - A regular expression inside
$in,$ninor$all, or as the operand of$ne, is a400. - An
$elemMatchinside$allis a400. - An empty
$allis a400.
A member that cannot do a class of work (expiry, webhook delivery or
embedding) now hands it to a healthy peer that can, and takes it back after a
quiet spell; it keeps owning the class until every live peer has confirmed, so
the overlap is a duplicate and never a gap (ADR-213). It adds no Breaking
entry: nothing that was accepted is now refused. A webhook subscription taken
over behind a long run of older entries keeps delivering (ADR-218).
Rolling back to 0.43.0: no configuration key was added (every knob of this
release is an environment variable or a constant), no table, kimmy.format
schema (still 4) or wire frame changed, so 0.43.0 starts on a 0.44.0 data
directory and its config file without edits. What it loses or ignores:
kimmy.replay-floorgainsowed = <node id>lines after its two floor
lines. 0.43.0 reads the floor lines and skips every other key, so it reads
the same floor. It keeps the file as it is when it does not move the floor,
so staleowedlines can survive a 0.43.0 run; back on 0.44.0 each holds a
marked member at503forcluster.catch_up_wait_secs, and is released
after the dwell if that member never returns.- A collection owed an embedding scan is marked in the store's metadata table
(vector_rescan:<collection id>). 0.43.0 looks metadata keys up by name and
never lists them, so it ignores the marks: it does not run those scans, and
documents the scan would have embedded wait. The marks stay in the store, and
0.44.0 scans the marked collections at its next start. - Answers revert to 0.43.0's: the
400s above answer200again, with the
results described in each entry. A mixed roll answers by member until every
member is rolled. kimmyd check-storeandkimmy_store_repairs_totaldo not exist in 0.43.0;
nothing is stored for them.- Ownership yielding (ADR-213) stores and sends nothing 0.43.0 reads
differently.kimmy.last-startis an advisory file besidekimmy.last-exit;
0.43.0 never opens it, so it is ignored (an interrupted write can leave a
kimmy.last-start.tmp.<pid>that 0.43.0 does not clean up, harmlessly). The
replication frames gain optionalfacts_genandechofields, and the facts
block optional fields for each class's state, cause and responsiveness; the
frames and the block are decoded leniently, and 0.43.0 skips fields it does
not know, so a mixed cluster works, with a 0.44.0 member that yields owning
the class until no 0.43.0 member is live, as an older peer never confirms.
There is no table or schema change. The webhook fix (ADR-218) changes no wire,
schema or configuration.
Added
-
$vectorSearch, an aggregation stage. The first stage of a pipeline can
run a vector search and hand the hits to the rest of it, so a search can be
filtered further, joined and grouped inside the database. It takes the fields
of thevector_searchendpoint (queryorvector,k,per_document,
filter), checks the same things and answers the same errors, and each hit
enters the pipeline as its source document with_scoreand_chunkadded.
It needsreadon the collection, which carriessearch, so a token granted
onlysearchis refused by aggregate'sreadcheck;kbounds what enters
the pipeline,aggregatestill has noexplain, and anywhere but first is a
400. The names other systems use,queryVector,
numCandidates,limitandindex, are refused, each naming the spelling to
use or saying there is no such knob. See Aggregation, Vectors and ADR-216. -
An
arrayFiltersentry takes$expr.{"$expr": {"$lt": ["$$line.qty", "$$line.min"]}}selects the elements whose quantity is below their own
minimum, where an entry could only compare a field with a constant. The
expression reads the element as$$<identifier>, at the entry's top level or
under$and,$orand$nor, onupdateandfind_and_modify; a scalar
element is$$nlike any other. A field,$$ROOTand$$CURRENTin it are
refused, as is an$exprthat names no identifier or a second one, and an
element it cannot be evaluated against fails the update, naming the document,
as a filter's does. No request that works changes. MongoDB refuses$expr
here, so this is an extension and is recorded in the Deviations register. See
Query language, "Positional updates", and ADR-209. -
The set operators,
$objectToArrayand$arrayToObjectin the
expression language.$setUnion,$setIntersection,$setDifference,
$setEquals,$setIsSubset,$anyElementTrueand$allElementsTrueread
each array as the set of its distinct members, where two values are one
member exactly when$eqcalls them equal (1and1.0are one), and a set
result comes back first-seen: members in the order they first appear,
arguments left to right.$objectToArrayturns a document into
[{k, v}, ...]in field order;$arrayToObjectreads[[k, v], ...]or
[{k, v}, ...], and a repeated key keeps its first place and its last
value. Null or missing in is null out; a non-array (or, for
$objectToArray, a non-document) is a400. ADecimal128in a set input
is a400rather than compared, and$arrayToObjectrefuses a mixed form,
a pair of the wrong shape, a non-string key, and a key holding a NUL, each
naming the element; any other string is a key, empty, dotted and
$-prefixed ones included, so a document turned into pairs and back is the
same document. Every one of these names was an unknown operator before, so
nothing that used to be accepted is refused.
$zipand$sortArraystay unknown operators. See aggregation.md, "Sets"
and "Documents as pairs", and ADR-207. -
kimmyd check-storeverifies a stopped member's store against redb's page
checksums, writing nothing. A serving node never verifies them: redb checks
them only in its repair after an unclean stop and in its own integrity check,
which needs the database to itself, so on a store closed cleanly a flipped bit
in a page that still decodes was served as data and nothing ever looked. The
command runs redb's integrity check over the store in the configured data
directory, through a view that keeps every write redb makes in memory, under
the store's lock, and prints one JSON line (verdict,detail,
unclean_close,rolled_back,elapsed_ms,bytes_read) with its log on
stderr. It exits 0 when the store verifies, 65 when it is damaged, and 1 when
it could not be checked (a node holds the store, there is no store, a newer
build wrote it, a read failed, or it was stopped). It needs no serving
configuration, reads the whole store, and does not readhnsw/. See
operations.md, "Checking a store", and ADR-204. Not a rollback boundary:
nothing stored changes. -
A start that repairs the store says whether the repair rolled back the
latest commit. redb's repair after an unclean stop discards the latest
commit when it fails verification, and did so silently. TheWARNline
database repaired after an unclean stopnow carriesrolled_back, and a
new counter,kimmy_store_repairs_total{rolled_back}, reads 0 or 1 for the
process. A rollback is the commit the stop interrupted, or damage to it;
after a clean shutdown it means the store is damaged. -
A member that cannot do a class of work gives it to a peer that can. Each
of expiry, webhook delivery and embedding now beats a heartbeat, and a
dedicated thread judges those heartbeats and the runtime's own stall probe
every five seconds. A class that has failed locally in 6 of the last 12 ticks
(expiry: three passes of five), or has not moved for longer than its own bound, is
advertised as yielded to a peer that is healthy, responsive and would own the
work, and the member goes on owning it until every live peer has confirmed.
It takes the class back after two quiet minutes, doubling that for a class
that yields again, up to 32 minutes. A provider that hangs or answers 5xx, a
webhook endpoint that is down and a missing key never make a member yield, a
paused process or a loaded host reads as nothing, at most one member yields a
class at a time at a few members, and when two members are stalled in the same
class none yields it (a fault that follows the data costs one hand-off). A
start afterstorage_failedortask_died, or after an unclean end that the
disk's free space, a short previous run or an earlier unclean end corroborates,
begins with every class yielded, so a node in a full-disk restart loop stops
holding its share. A stuck writer is not detected yet. A member t...