Skip to content

Releases: titusai-io/kimmydb

0.51.0 - 2026-10-17

Choose a tag to compare

@github-actions github-actions released this 17 Oct 11:27
72559c3

Release Notes

Read before upgrading, three things for Decimal128 and a fourth for vector search.

(a) Breaking: a Decimal128 is compared, ordered and keyed by its exact value (ADR-238). A stored decimal used to rank equal to every number, so {v: {$eq: 999999}} matched {v: Decimal128("1.5")} and $ne could never exclude it. It now equals exactly the numbers of the same value, across Int32, Int64, Double and Decimal128, and Decimal128("0.1") is not Double(0.1): the double is slightly above one tenth. Filters, sorts, $expr comparisons, $cmp, $in, $indexOfArray, the $min and $max accumulators, the update $min, $max, $pull and $pullAll, and the $ne and $nin of update, delete and count change their answers: deleteMany({v: {$ne: 999999}}) now deletes the decimal documents. $group, $addToSet, $lookup join keys and the set operators identify equal numbers and decimals as one member. A unique index reports a violation between equal values of different types. $sum and $avg answer 400 on a Decimal128 operand where they skipped it and answered a wrong total.

(b) The first start of 0.51.0 migrates the store before it serves, and is a rollback boundary to 0.50 and 0.49 (storage schema 5; kimmyd restore of a 0.50 backup runs the same migration inside the restore command, before it reports restored). Every number key on disk changes, so every document with an integer _id is re-keyed and every non-vector index is rebuilt. Measured on the lab host, cold, 2,000,000 documents with integer _ids and three indexes took 15.6 to 19.3 minutes from the container's start to a serving node (the first run: re-key 6.3, index rebuild 11.8, oplog walk 1.1); a store with string or ObjectId _ids has no re-key. The start logs the time to plan from as a range, estimate_secs_low to estimate_secs, the upper figure being a spinning disk with a cold page cache; plan on the upper one. It needs free space: the start refuses, naming the figures, when there is too little, and a refused start leaves every row, the sidecar and the schema as they were, so 0.50 still opens the store (the open's own table commit, which 0.50 makes too, may have touched the file; a store with no free page can hit ENOSPC at that first commit on a filesystem that allocates a file's extended length, such as HFS+ or ExFAT). From the moment the sidecar is raised, a 0.50 or 0.49 build refuses the store (it leaves kimmy.redb and kimmy.format as they were and adds no file, but rewrites kimmy.last-exit to record the failed start, which a later 0.51.0 start reads with no effect); the way back is a backup taken before the upgrade, so roll with BACKUP=true. A member that is down for the migration for longer than storage.oplog_retention_secs returns by a whole-database snapshot. The rebuild reports every unique collision it finds, so a replication-merged duplicate that a schema 4 store already reported is reported again, once, on every member (/violations deduplicates it; change streams and webhooks do not). The first start also walks the oplog once. See operations.md.

(c) A Decimal128 _id, and a partial index with a Decimal128 bound, are refused until the operator declares cluster.decimal128. Set it only once every member runs 0.51.0 and has finished its migration, and as a change of its own, never in the change that ships the binary: a 0.50 or 0.49 member cannot decode a decimal _id and never retries a definition it refused. It defaults to true with clustering off and false with it on. A durable marker (kimmy_decimal128_id_present, and decimal128_id_present on /readyz) records that a decimal _id exists; once it does, a member can no longer be rolled back by wipe and resync. A 0.50 or 0.49 binary refuses a configuration that names cluster.decimal128.

Rolling. Member-at-a-time is safe. Every key on the wire, in a cursor token and in the cut digest keeps the schema 4 form, so a 0.50 and a 0.51 member compare and repair exactly as two 0.50 members do; a 0.51 member's tombstone horizon resets at its first start. Two differences outlast a member's own upgrade only until it migrates: a 0.51 member refuses or reports a decimal pair in a partial unique index that a 0.50 member accepts, and TTL deletes follow whichever member owns the collection's expiry.

Read before upgrading: the first start re-embeds every document with a binary _id, once. Vector chunks were stored under the _id's display text, under which every binary _id of one length was the same; they are now stored under an encoding that tells every two documents apart (ADR-239), and each server-embedded collection is rechecked: the owner re-embeds every document keyed by a binary _id (every UUID) once, one provider call's worth each, and moves the chunks of a String _id that begins with ~, is 24 lowercase hex digits or is an integer's decimal with no provider call. ObjectId, integer and every other String _id keep their chunks where they are. A single node rechecks at its first start; a cluster member rechecks what it owns at its first start, once its ownership has held, and again when every live member runs 0.51.0. Client-supplied (byo) collections keep their vectors, read as they are, until the client's next PUT. One loss during a roll: a 0.50.x member that deletes a document whose _id is a String equal to an integer's decimal or an ObjectId's hex ("7" beside 7) removes the integer's or ObjectId's vectors too, on every member; that document is re-embedded by the recheck when the roll finishes. operations.md has the roll and the rollback.

Added

  • A Decimal128 has a place in the order. It is compared exactly against every number, sorted among them, indexable (a Decimal128 at an indexed path is keyed, and answers range queries), usable as a filter operand, a sort key, an expression literal, a partial-index bound and an update operand, and can be an _id (an integral one that fits i64 is the Int64 id of that value, with its stored _id unchanged).
  • cluster.decimal128 and the capability Facts.decimal128, with the gauge kimmy_decimal128_id_present (OTLP kimmy.decimal128_id_present), the /readyz field decimal128_id_present, and a WARN once per peer and every ten minutes while the marker is set and a live peer cannot hold such an id. A creation refused by the switch answers 400 and names it and the peers that block it.
  • /readyz reports each step of the migration with its own phase and count: preflight (the documents with a number _id that the first 0.51.0 start on a 0.50 store reads to size the free-space check it makes before it changes anything; a store whose _ids are not numbers skips it, and a stop during it leaves the store as 0.50 left it), migrating (rows re-keyed), draining (entries removed from an index) and refilling (documents walked to rebuild a collection's indexes).

Changed

  • A number key is exact and variable length (ADR-238): exp ‖ esc(M) ‖ END, where M is the first 128 bits of the value's binary expansion with trailing zero bytes stripped and END says whether any later bit is nonzero. An Int64 key is 7 bytes where it was 12. See key-encoding.md.
  • A decimal at a partial filter's path is decided by the filter, by its exact value. A partial index holds exactly what its filter selects. A TTL partial index can now expire documents holding a decimal at its filter path that 0.50 never expired ({tier: {$lt: 3}} over tier: Decimal128("1")), and the reverse: a document a $gte filter selected only because its decimal ranked equal to the bound is now kept. The deletes are ordinary replicated deletes, irreversible except from a pre-upgrade backup, and the migration logs now_selected and now_not_selected for each TTL index before the first expiry pass. A partial unique index enforces on decimals at its filter path: a write 0.50 accepted can be refused, and standing duplicates are reported on upgrade (/violations, kimmy_unique_violations_total, a log line) and then refuse an update that keeps the key.
  • An operator that takes a whole number takes a Decimal128 of whole value, as it takes the double of it. $arrayElemAt, $slice, $range, $substr, $size, $mod (its operand, and a stored decimal it is tested against, truncated toward zero as a double is), $limit, $skip, $push's $position and $slice, sort directions and projection flags read a decimal as the double of the same value; a fraction, NaN or an infinity is refused where a double's would be. A decimal component of a $vectorSearch query vector stays refused (it would have to be rounded to a 32-bit float).
  • A unique index can now report a violation between two values of different types that are equal (Int64(1) and Decimal128("1.0")).
  • Creating a partial index with a Decimal128 bound is refused until cluster.decimal128 is set (see (c) above); a replicated, restored or repaired definition is always applied.
  • /readyz gains a key (decimal128_id_present), and its phase gains preflight, draining and refilling: a client that pins the exact key set or enumerates the phases must add them.
  • The HNSW snapshot on disk is format 3, keyed by the id each chunk is stored under. A format-2 snapshot is refused and the collection's index rebuilt on its first search after the upgrade, once per vector collection on each member; a 0.50.x member refuses a format-3 one and rebuilds in turn.
  • During a roll a 0.51.0 member reads a 0.50.x member's chunks for binary and reserved-String documents and moves or re-embeds them, while a 0.50.x member finds a 0.51.0 me...
Read more

0.50.0 - 2026-10-16

Choose a tag to compare

@github-actions github-actions released this 16 Oct 06:30
98eef35

Release Notes

Read before upgrading, two things.

(a) Breaking: a database's drop now keeps the writes stamped after it (ADR-236), as a collection's drop does since 0.49.0. A write made into a collection of the database after the drop's stamp, through a member that had not yet heard of the drop, is kept, re-dated just above the drop, so the collection, and its database, can come back holding only the writes stamped after the drop. Clock skew can cause it. The way to avoid it is unchanged: confirm a drop on the members before writing through another one. A collection with nothing stamped after the drop is buried as before.

(b) The divergence check now repairs what it finds (ADR-237), on by default. A confirmed cut-digest finding plans a document repair, which writes a row only where it is strictly newer than the one held (and a young lost insert); a confirmed definition finding plans the build of one plain index this member lacks and a peer holds. Everything else stays a finding for you, in words, in docs/operations.md ("The cut digest" and "The definition digest"). A loop guard judges every repair and backs off one that changes nothing. cluster.divergence_repair = false makes a member detect only, as 0.49.0 does. Nine series report it.

Rolling and rolling back. Member-at-a-time is safe, and 0.50.0 is not a rollback boundary: no kimmy.format schema (still 4), I_EPOCH, table, backup tag or entry kind changed.

  • During a roll a 0.49.x member buries a collection a 0.50.0 member keeps under a database drop, and refuses it from a 0.50.0 member until it is upgraded; it repairs nothing, and a 0.50.0 member repairing from it writes no young lost insert.
  • Before rolling back to 0.49.x, remove cluster.divergence_repair, cluster.repair_max_documents and cluster.repair_max_index_documents from the configuration: 0.49.x refuses to start on a configuration that names any of them.
  • A store rolled back to 0.49.x buries, at its next start, every collection 0.50.0 kept under a database drop, with the writes it kept, so the members differ until they run one version. docs/compatibility.md has the two differences that outlast a roll.

Added

  • The divergence check now repairs what it finds, in two cases. (cluster.divergence_repair, on by default; false makes a member detect only, as 0.49.0 does. The count half's repair is unchanged and has no switch.)
    • A confirmed cut-digest finding plans a document repair: a scoped snapshot of the one collection applied as rows over rows the member already holds. A row is written only if strictly newer than the row held (live or tombstone); an id the member holds nothing for is written only if it is young (stamped after every tombstone cutoff either member has applied, so no delete of it can have been collected) and is otherwise counted left and logged, with the operator rule in operations.md. No definition is restored, nothing is dropped, no cursor is stored, no coverage is granted.
    • A confirmed definition finding plans a definition repair for one case: a plain index this member lacks by name and a peer holds, built from the first page of the peer's scoped snapshot, at most one per repair. A unique index, a TTL index, a rival under the same name, a vector configuration, an index the peer dropped, an index of a buried life, an index this member dropped, and an index at or below the collected-through horizon stay findings for the operator, each counted in kimmy_sync_definition_repair_indexes_total{result}, and a repair that skips any index logs a WARN (a definition repair left indexes it will not build) with the count of each kind and the first ten index names. The build holds the single writer for its length; cluster.repair_max_index_documents bounds it.
    • A loop guard keeps a repair that cannot change anything from running for ever and a repair that works from hiding a bug that keeps producing the divergence: a repair is judged at the first compared digest after it ends and backs off 5 min, 15 min, 45 min, 2 h 15 min and 6 h when it did not help (a repair that wrote a row at or below the finding's cut lowers the level to 1); more than three clear-and-recur episodes in an hour stop the collection for six hours, with an ERROR and kimmy_sync_digest_repair_stopped_collections.
    • A repair never takes the peer's ordinary replication. After any document or definition repair pull to a peer, the next contact with it is an ordinary one; a document repair's round is at most 20 pages or 10 seconds, and a document repair ends, and is judged, after the collection's size at planning plus cluster.repair_max_documents. kimmy_sync_repair_rounds_total counts the count half's repair rounds only; the two new repairs have their own series. The loop guard keeps a member's state across a restart or a short cut of the peer (a member is forgotten only after 12 hours out of the live member list), stops a one-peer loop however many collections the member holds, and backs off at once after an abandon (three rounds that landed nothing) or a dropped page, and after three repairs in a row that end without a result for any other reason (a whole-database takeover, a cancellation while queued, a definition page without the collection). A plan the gates refuse (disabled, too_large) is counted once per standing finding, not at every contact. A refused definition build counts in kimmy_sync_ddl_refused_total.
    • New keys cluster.divergence_repair, cluster.repair_max_documents (2,000,000) and cluster.repair_max_index_documents (250,000). New series kimmy_sync_divergence_repair_enabled, kimmy_sync_digest_repairs_total{outcome}, kimmy_sync_digest_repair_judgements_total{result}, kimmy_sync_digest_repair_rows_total{result}, kimmy_sync_digest_repair_stopped_collections, kimmy_sync_definition_repairs_total{outcome}, kimmy_sync_definition_repair_judgements_total{result}, kimmy_sync_definition_repair_indexes_total{result}, and kimmy_gc_tombstone_horizon_seconds.
    • A cut digest that compares equal quiets a suspended count finding of the same peer and collection, so kimmy_sync_divergent_collections does not stay above 0 on a member taking steady writes after the two members hold the same documents (a count finding whose probe the gate keeps deferring is otherwise neither renewed nor cleared). The finding leaves the gauge and plans nothing, but the cooldown of its repair stands, so a peer that stops and moves in turn is not count-repaired at every stop; the next compared count probe clears it or confirms it again.
    • A collected-through horizon is kept: the cutoff of the last retention pass that removed a tombstone, raised inside each collector's removing transaction, reset to the start time on every start under a different build than the last (every upgrade and rollback) and by a restore, adopted by a completed whole-database snapshot, and offered on every snapshot page. After each upgrade there is a young-repair dead time: a lost insert older than the member's start is left for good until an operator re-inserts it.
    • Rolling. Member-at-a-time is safe. A 0.49.0 sender sends no horizon, so a member repairing from it admits nothing young and leaves every id it holds nothing for. A 0.49.0 member repairs nothing. Rolling back: ClusterConfig denies unknown keys, so a 0.49.0 binary refuses to start on a config that names any of the three keys, at any value; remove them first. No stored data depends on them.

Changed

  • Breaking: a database's drop keeps the writes stamped after it, as a collection's drop does (ADR-236, amending ADR-229's "writes and all"). A write made into a collection of the database after the drop's stamp, through a member that had not yet received the drop, was buried with the collection on that member and kept by a member holding a newer collection of the same name, and the count repair then replayed it back to some members and pages never did: the outcome depended on which member and which route, and two members could end holding different versions of one document for good. On every member and every route (entries, the count repair's replay, snapshot pages), for documents and for index and vector definitions alike, a collection the drop covers now keeps what is stamped after the drop and loses what is stamped at or before it, re-dated just above the drop. So a collection, and its database, can come back after a database drop, holding only the writes stamped after the drop, wherever such a write reached a member that had not heard of the drop; clock skew can cause it, as for a collection's drop since 0.49.0. To avoid it, confirm the drop on the members before writing through another one. A creation stamped at or before the drop is still refused everywhere, a collection with nothing stamped after the drop is buried as before, and the written_after_the_drop warning is gone (the write is kept, and logged kept=true). No table, schema, wire field, backup tag or entry kind changed. Rolling back to 0.49.x: a 0.49.x member buries, at its next start or database sweep, every collection this release kept under a database drop, with the writes it kept, while 0.50.0 members keep them, so the members differ until they run one version; and during a roll a 0.49.x member that holds such a collection when the drop arrives buries it with the write, and refuses the kept collection from a 0.50.0 member's snapshot, until it is upgraded and takes it back from a peer.

Fixed

  • A member taking a snapshot was count-compared by its peers, which confirmed a divergence and pulled a repair from it. A member part-way through a whole-database snapshot holds the pages it has applied as state its version vector does not name, and its vector stands still until the last pa...
Read more

0.49.0 - 2026-10-14

Choose a tag to compare

@github-actions github-actions released this 14 Oct 17:01
feb275e

Release Notes

Read before upgrading, two things.

(a) A collection can now reappear after its drop. A drop that reaches a member holding writes stamped after it now keeps those writes (ADR-231, extended), where 0.48.x discarded them with the collection and left the members different for good. So when a write stamped after a collection's drop reaches a member that had not heard of the drop, the collection comes back on every member holding only the writes stamped after the drop. This is for collection drops only: a database's drop still buries a collection created at or before it together with the writes made into it afterwards (ADR-229). Clock skew can cause it: a write made earlier in real time by a member whose clock runs ahead counts as "after". The way to avoid it is unchanged: confirm a drop on the members before writing through another one. See docs/http-api.md, "A write made after a collection's drop is kept".

(b) The divergence gauge can read above 0 where it read 0, with a WARN. 0.49.0 adds the definition digest, a fourth comparison between members, of what a collection's indexes and vector configuration say. A finding moves the new kimmy_sync_definition_divergent_collections and is also counted in kimmy_sync_divergent_collections, so an alert on the existing gauge can fire where it did not. It is detect-only. Before acting on a finding, read docs/operations.md, "The definition digest": recreating a TTL index at a shorter value deletes the documents older than it on every member, and recreating a unique index files violations for the duplicates that exist. The first round can show one true positive, an index or vector configuration with no creation stamp that a late drop's sweep leaves (documented there).

Rolling and rolling back. Member-at-a-time is safe, and 0.49.0 is not a rollback boundary: no kimmy.format schema (still 4), I_EPOCH, table, backup tag or entry kind changed.

  • A 0.48.x member ignores the definition request and is counted unanswered, so definitions are compared between two 0.49.0 members only.
  • The two optional fields this release adds, redated (in a collection's stored metadata) and redated_from (on a snapshot page's collection definition), are ignored by 0.48.x. Rolling back to 0.48.x is safe for the stored data. The refusal hints are META rows 0.48.x never reads.
  • During a roll, old and new members still diverge in the case above (a drop meeting later writes): the 0.48.x member discards them as it did, and a write an old member has already destroyed is not recovered.
  • Two things 0.48.0 cannot do for a store 0.49.0 has worked on. It has no start-time reconcile_late_drops, so a crash in the middle of a late-drop sweep followed by a 0.48.0 start serves a half-swept life until 0.49.0 runs again. And it does not know where a re-dated life's first kept entry arrived (a 0.48.0 that rewrites the metadata drops the marker), so a change stream opened without a resume token on such a collection reads the retained log from the start to find it, the walk round 0500 measured at 62 s on a 1.8-million-row log.

Added

  • The definition digest. Two members can hold the same documents and different indexes for good: a member that refused a replicated index lacks it while its peers hold it, and two members that created one name with different filters each keep their own, and nothing compared it. The divergence check now also compares a hash of each collection's index and vector definitions, at the cut the cut digest uses, on every checked contact, between two members that both have it. It is detect only: a confirmed finding moves kimmy_sync_definition_divergent_collections and kimmy_sync_divergent_collections, logs a WARN (at most twenty per peer per tick, then a summary), and plans no repair. A member now remembers what it refused, in its metadata (a backup carries it), so a difference a refusal explains is counted as explained and never as compared. New series: kimmy_sync_definition_divergent_collections and kimmy_sync_divergence_definition_probes_total{outcome,reason} (outcomes compared, explained, unanswered, deferred), with the OTLP series and the golden list. Read before upgrading: the divergence gauge can read above 0 where it read 0, because it now also counts a definition difference. A mixed 0.48.0/0.49.0 roll is safe: a 0.48.0 member ignores the request and is counted unanswered for definitions, and is never compared. Not a rollback boundary: optional fields on the two divergence messages that an older member ignores, and rows in the metadata that an older build never reads. What to do with a finding, including that recreating a TTL index at a shorter value deletes documents, is in docs/operations.md, "The definition digest".

Fixed

  • A collection dropped on one member and created again on another before the drop replicated left the members different for good, and the repair never finished. The second life had applied documents of the first before it learned of the drop, and the drop, judged aimed at a life already over, removed nothing. It now sets the life's incarnation floor at the drop and removes what that life took in of the one before (stamped at or before the drop), on every member that learns of it, by an entry, a snapshot page, a database drop's sweep or a start; a store left with such documents is healed before it serves. This reverses the residual ADR-229 accepted for database drops. The indexes and vector configuration of the first life are removed the same way.
  • A write made after a collection's drop, through a member that had not yet applied the drop, was lost on that member and kept on the others, and the members stayed different. A drop that reaches a member holding the life it is aimed at now removes the records and definitions stamped at or before it and keeps the later ones, with the floor at the drop, so every member ends with the same documents and the repair ends (ADR-231, extended). A collection can therefore reappear after its drop: when a write stamped after the drop reaches a member that had not heard of it, the collection comes back on every member holding only those writes. Clock skew can cause it: a write made earlier in real time by a member whose clock runs ahead counts as "after". A database's drop is unchanged: a collection created at or before it is buried with the writes made into it afterwards (ADR-229). Receiver-local, no required wire field, table, entry kind or schema change, and not a rollback boundary: a build before it discards the later writes and ignores the two optional fields it adds (redated in a collection's stored metadata, redated_from on a snapshot page's collection definition). During a roll the old divergence persists between old and new members, and a write an old member has already destroyed cannot be recovered.
  • An index a client was told existed could be swept away with the life before it. When a collection is dropped on one member and made again on another before the drop replicates, the first life's create_index X can reach the second member before its own client creates X; that create found the definition standing and answered that it exists, and the first life's drop then removed it (ADR-231), leaving the client without an index and no entry replicating one. A local create of an identical definition over one stamped before the collection's own creation is now a new creation: it takes a fresh stamp and replicates a CreateIndex, and the members that hold the definition raise their stamp to it. The answer is the same as before (idempotent when the definition is identical), and an identical create over an index of the collection's own life still does nothing, so a client that creates its indexes on every boot mints no entry. A vector configuration already took a fresh stamp on every local configure and needed no change. The digest also no longer compares a collection that still owes ADR-231's sweep for a drop the cut covers.
  • The cut digest compared a recreated collection or database only after the drop's tombstone was collected. A drop's tombstone is kept for storage.tombstone_retention_secs (24 hours by default), and 0.48.0 deferred a collection as incarnation (or database_tombstone) for as long as either member held one, so on a cluster that drops and recreates collections most digest outcomes were deferrals and those collections were not compared (round 0510: 81% of outcomes). It now defers only while the drop is above the cut two members share, or while the standing collection is not newer than the drop. No false finding came of the old rule; this restores coverage.

Security

  • rustls is on 0.23.46, which fixes four advisories (RUSTSEC-2026-0346, RUSTSEC-2026-0347, RUSTSEC-2026-0348 and RUSTSEC-2026-0349) against 0.23.45: a Brotli certificate decompression that could allocate up to 1 GiB before authentication, an unauthenticated ServerHello that could downgrade later connections to classical key exchange, application data buffered and sent to the outer server after an ECH rejection, and Debug output of client session values that exposed TLS secrets. rustls carries the HTTPS listener, the member links and the outbound clients; the change is the lockfile only (no API, wire, store or configuration change), and rolling members one at a time is safe.

Download kimmyd 0.49.0

File Platform Checksum
kimmyd-aarch64-unknown-linux-musl.tar.xz ARM64 MUSL Linux checksum
[kimmyd-x86_64-unknown-linux-musl.tar.xz](https://github.com/titusai-io/kimmydb/releases/dow...
Read more

0.48.0 - 2026-10-13

Choose a tag to compare

@github-actions github-actions released this 13 Oct 17:43
f964707

Release Notes

Read before upgrading: the divergence gauge can now read above 0 where it read 0, and pages an existing alert. 0.48.0 adds the cut digest, a third comparison between members. It finds a document updated on one member and not on the other, which leaves every document count equal and which no earlier check saw. It is detect only: a confirmed finding moves kimmy_sync_digest_divergent_collections and kimmy_sync_divergent_collections, logs a WARN, and plans no repair. Roll the members one at a time. Nothing here is Breaking: the wire gains optional fields that a build before it ignores, two derived tables are added that a build before it ignores, and no kimmy.format schema (still 4), redb version, backup tag, snapshot field or entry kind changed, so a member rolls back by starting 0.47.2 on its store.

The first start of 0.48.0 walks the documents once, in the same pass that rebuilds the live counts, and so does the start after any crash or kill of 0.48.x (and after a clean stop of an older build, and after a start that was stopped before its walk completed). The walk is chunked, each chunk is committed, it honours a stop, and /readyz reports phase: counting with done and total. A stop during it resumes on the next start; a crash or a kill during it starts it again from the beginning. On a spinning disk, cold, it ran at about 10 MB/s, so a large store takes minutes. It also fixes a standing defect: the old rebuild ignored a stop and ran in one transaction, so a SIGTERM during a long counting phase became a SIGKILL after the stop budget.

Added

  • The cut digest. One row per collection, kept in the transaction of every write: the XOR of a hash of each live record's (_id, stamp), the highest record stamp per origin, and the highest stamp of a tombstone no oplog entry stands behind. Two members compare it at a cut, the highest stamp per origin both have processed; what either holds above the cut is left out of both, so a write one member has and the other has not yet received is never a finding. Finding the uncovered keys of a busy collection is one scan of the oplog above the cut, bounded to 200 ms and 8 MiB per request, one probe answered at a time per member. Before a mismatch is reported the member verifies its own row against a walk of the collection and corrects it if it drifted (kimmy_sync_digest_drift_total, an ERROR). A collection is not compared, and the reason is counted, when a write landed between the plan and the comparison, when the scan spent its budget, or when either side holds a drop tombstone for its id (deferred{reason="incarnation"}, until the tombstone ages out). New series: kimmy_sync_digest_divergent_collections, kimmy_sync_divergence_digest_probes_total{outcome,reason} and kimmy_sync_digest_drift_total, with the same series in the OTLP export. What to do with a finding, including when the API cannot tell which member is behind and neither should be wiped, is in docs/operations.md, "The cut digest".

Changed

  • kimmy_process_resident_peak_bytes's HELP text says what the gauge is: the most this process has seen, not a reading that can fall.

Fixed

  • A stop during the open's rebuild of the live counts was ignored. The rebuild is now chunked and honours the stop (above).

Download kimmyd 0.48.0

File Platform Checksum
kimmyd-aarch64-unknown-linux-musl.tar.xz ARM64 MUSL Linux checksum
kimmyd-x86_64-unknown-linux-musl.tar.xz x64 MUSL Linux checksum

0.47.2 - 2026-10-12

Choose a tag to compare

@github-actions github-actions released this 12 Oct 20:30
62a5a03

Release Notes

This release fixes kimmy_process_resident_peak_bytes, which could fall between two scrapes of a
running node. Roll the members one at a time. This release is not a rollback boundary: no
configuration key, table, kimmy.format schema (still 4), redb version, backup tag, snapshot field
or entry kind changed, and 0.47.1 opens a store 0.47.2 has run on, so a member rolls back by
starting it on 0.47.1.

Fixed

  • kimmy_process_resident_peak_bytes fell between two scrapes of a running node. The gauge was
    read from VmHWM, which the kernel reports as the larger of its high-water mark and the current
    resident size and updates lazily, so a node whose resident size had just been its maximum could read
    lower a few seconds later (249360384, then 248778752, with no restart). Every reading of VmHWM
    and VmRSS is now folded into a running maximum for the process, so the gauge never falls within
    one process life, as the documentation says, and is never below kimmy_process_resident_bytes.

Download kimmyd 0.47.2

File Platform Checksum
kimmyd-aarch64-unknown-linux-musl.tar.xz ARM64 MUSL Linux checksum
kimmyd-x86_64-unknown-linux-musl.tar.xz x64 MUSL Linux checksum

0.47.1 - 2026-10-12

Choose a tag to compare

@github-actions github-actions released this 12 Oct 17:37
e96527d

Release Notes

This release fixes two things in the database-drop tombstone of 0.47.0: a WARN on every clean
drop, and a change stream opened at the start of a recreated collection that read the whole
retained log. Roll the members one at a time. This release is not a rollback boundary: no
configuration key, table, kimmy.format schema (still 4), redb version, backup tag, snapshot field
or entry kind changed, and 0.47.0 opens a store 0.47.1 has run on, so a member rolls back by
starting it on 0.47.0. Nothing here changes what a client is sent except that the second fix
answers the opens it used to answer too late.

Fixed

  • A database drop warned for every collection it buried that held documents. The sweep that
    buries what a database's drop covers logged a database drop buried a collection that holds documents at WARN for any collection with a live document, so an ordinary clean drop warned on
    every replica, once per collection. It now warns only when a document was written after the drop
    (a database drop buried a collection that holds documents written after the drop, with
    written_after_the_drop): a write a member that had not yet received the drop answered 200 and
    that the drop then took. A collection whose documents all predate the drop is buried without a
    warning. The look at the documents reads the fixed-width header of at most a thousand records and
    copies nothing, so the warning sees the first thousand documents of the collection by id.
  • A change stream opened at the start on a recreated collection read the whole retained log.
    Finding where the collection's current life begins walked the arrival index from its start to the
    first entry stamped after the collection's drop, once per stream: on a log of 1.8 million entries
    that was 1.8 million rows and a minute, and a GET .../watch?from_start=true on a recreated
    collection answered no events within a client's timeout while the node logged a change stream took over 10 s to find where it starts. The start of a recreated collection is now found just after its
    drop's entry and reads a few rows however long the log. The events delivered are the same, and a
    resume token issued before a database's drop still resumes. A collection that survived a database's
    drop (created here before the drop reached this member), and one this member holds no creation or
    drop entry for, are placed by the walk as before.

Download kimmyd 0.47.1

File Platform Checksum
kimmyd-aarch64-unknown-linux-musl.tar.xz ARM64 MUSL Linux checksum
kimmyd-x86_64-unknown-linux-musl.tar.xz x64 MUSL Linux checksum

0.47.0 - 2026-10-12

Choose a tag to compare

@github-actions github-actions released this 12 Oct 05:54
8217ba8

Release Notes

This release makes a dropped database stay dropped. DELETE /v1/db/{db} used to drop the
collections the member answering it held, and a collection a peer had created and that member had
not yet received brought the database back on every member, with the deleted data in it. The drop
is now a stamped tombstone of the database that replicates, and every member drops what the
database held when it was stamped. That changes what the route means, so this release has one
Breaking change
, below. There is no rollback boundary: no configuration key, table,
kimmy.format schema (still 4), redb version, backup tag, snapshot field or entry kind changed, and
0.46.0 applies the new drop as the drop of a collection it never held (Rolling
back
).

Read before upgrading:

  • Breaking: after a database drop, recreate the database on the member that dropped it, or wait
    until every member lists it gone.
    DELETE /v1/db/{db} now also deletes, on every member, the
    collections created before it that the answering member had not received, with their documents.
    A creation on a member that has not received the drop can be stamped before it and is dropped
    with it, with the documents written into it afterwards, and those writes were answered 200.
    Details in the first entry under Changed.
  • A roll or a rollback must finish within storage.tombstone_retention_secs (default 24 h) of
    any database drop made during it.
    A member on 0.46.0 keeps a collection the dropper never
    held, and keeps serving it, until it restarts on 0.47.0; it needs the drop's tombstone to then
    bury it, and every member collects that tombstone after the retention. Past it the collection,
    and the database, come back on every member.
  • A partially_applied answer from a database drop can carry an empty dropped list: the
    drop's tombstone is its first commit, so the drop is recorded and replicates even when no
    collection has been buried yet.
  • A start buries what a database's drop covers before it serves, and /readyz shows
    phase: reconciling while it does (the OpenAPI document lists the phase). A store left by a
    member that applied a drop as a plain collection drop, or that stopped between the tombstone and
    the burials, is finished then. A start that cannot bury for a reason other than a stop fails with
    the cause: run kimmyd check-store on the stopped member, and if the store is damaged wipe its
    data directory and let it catch up from its peers.

Changed

  • Breaking: DELETE /v1/db/{db} also deletes, on every member, the collections created
    before it that the member answering it had not received
    (ADR-229). The drop is a stamped
    tombstone of the database: on every member, every collection of it created at or before that
    stamp is dropped with its documents, and none can be created at or before it. A collection
    created after it stands. Recreate a dropped database on the member that dropped it, or wait
    until every member lists it gone
    : a member that has not received the drop can stamp a creation
    before it, and that creation is dropped too, with the documents written into it afterwards. The
    same holds for a write acknowledged into such a collection on a member that has not applied the
    drop, and for one on a 0.46.0 member in a roll. Resurrection is no longer possible except for a
    creation concurrent with the drop (within clock skew, replication lag or the time the drop takes
    to bury its collections) and for a member behind longer than the tombstone's retention. The sweep
    logs each collection it buries that held documents (a database drop buried a collection that holds documents). A member that receives the drop of a very large database converges as fast
    as it did on 0.46.0, and a stop during the sweep is clean.

Fixed

  • A collection a member never held, with vectors, stopped replication at its first vector
    chunk after its drop
    until a snapshot: the drop recorded the collection's tombstone and not
    its shadow's, so the chunk was an entry for an unknown collection (ADR-229).
  • A change stream or the vector cache on a collection buried by a snapshot page or a
    replicated drop with no entry of its own
    was never ended, or never forgot the graph; both
    are now told (ADR-229).

Download kimmyd 0.47.0

File Platform Checksum
kimmyd-aarch64-unknown-linux-musl.tar.xz ARM64 MUSL Linux checksum
kimmyd-x86_64-unknown-linux-musl.tar.xz x64 MUSL Linux checksum

0.46.0 - 2026-10-11

Choose a tag to compare

@github-actions github-actions released this 11 Oct 12:28
f1e725f

Release Notes

This is a minor release with no rollback boundary: no configuration key, table,
kimmy.format schema (still 4) or redb version (4.3.0) changed, and the wire gained only
optional fields and one list that 0.45.0 skips (Rolling
back
). What is new is that a member that caught up by a
whole-database snapshot gives its peers, and receives, what a snapshot used to leave out, and
serves only the history it actually holds:

  • A snapshot-caught-up member no longer relays a window it cannot serve (a vector
    configuration, a collection drop, a collection creation), and the snapshot now carries
    index drops, a vector configuration turned off and deletes of documents a member
    never held
    (ADR-224, 225, 226, 228). New kimmy_sync_horizon_state_held_total.
  • A change stream never skips events silently across a collected range or a snapshot's
    hole
    : it is refused with ResumeTokenExpired instead (ADR-173's addendum).
  • A member could be stuck catching up indefinitely on a snapshot page that changed a
    client-supplied vector width; it no longer is (ADR-227).
  • dns: and k8s: seeds resolve with the resolver dns-srv: uses, so a stop does not
    wait for a lookup (ADR-223).

Read before upgrading (none is Breaking in the API, wire, disk or configuration sense):

  • A missing resolv.conf, or one with no nameserver, is now an error for dns: and
    k8s: seeds (ADR-223).
  • A member behind a snapshot-caught-up peer's grant may take a whole snapshot from it,
    where it was served entries across history the peer did not hold: the peer answers
    BeyondHorizon, the member asks its other contacts first, and snapshots only if none can
    serve it, at most one tick late (ADR-224).
  • A retried disable of a vector configuration that is already off now writes and
    replicates an entry
    , and the entries path is last-writer-wins on the configuration's stamp
    (ADR-226).
  • Change-stream clients resuming across a member that caught up by snapshot may get
    ResumeTokenExpired and must re-list
    (the refusal is conservative: one maximum per
    origin) (ADR-173's addendum).
  • Holes from snapshots taken on 0.45.0 (or before this release) are not known, so a member
    that caught up that way serves across history it holds only as state until it is rebuilt from
    a peer.

Changed

  • A kimmyd stopped while a dns: or k8s: seed is being looked up no longer
    waits for the lookup
    (it held the stop for up to the 22 s budget when the name
    server did not answer). These seeds now resolve with the library dns-srv:
    already uses, one resolver per lookup, so /etc/hosts and /etc/resolv.conf are
    read each time and nothing is cached between discovery ticks, as with
    getaddrinfo. What differs, so that a seed that resolved still does:
    resolv.conf is honoured for its nameservers, search/domain and options ndots/timeout/attempts/edns0; a missing resolv.conf, or none with a
    nameserver, is now an error
    (it was a resolver on 127.0.0.1); nsswitch.conf
    is not read (the static musl build and so the container never read it; a glibc or
    macOS build loses mdns/sssd/scoped resolvers); and a seed is given the time resolv.conf's timeout and attempts allow for the
    names it tries (at least 15 s) and is then reported and tried again at the next
    tick, where it used to wait on the resolver's own timeouts. Seeds are resolved
    together, so several that do not answer cost one bound in all. No rollback boundary and no configuration
    change. The webhook and embedding-provider lookups still use getaddrinfo.
    ADR-223; amends ADR-202.
  • New test switch KIMMY_TEST_DNS_SERVER=<ip>:<port> (docs/testing.md,
    docs/operations.md): it reaches dns: and k8s: seeds, not dns-srv:. Never set
    it in a deployment; it logs a WARN at start.

Fixed

  • A change stream resumed across a member that caught up by snapshot could silently miss a
    delete.
    A member that took a whole-database snapshot holds the sender's documents but not the
    events inside the span, so a client resuming on it with a token below the span was sent the
    current versions and never the delete of a document the member had not held: its cache stayed
    wrong with nothing to say so. Such a token (bare or vector) is now refused with
    ResumeTokenExpired, and the client re-lists, as for any history a member no longer holds. A
    token that covers the span still resumes. Clients resuming across a snapshot-caught-up member
    may now get ResumeTokenExpired;
    the refusal is conservative (one maximum per origin). ADR-173's
    addendum amends its "no gap on another member" claim. No API, wire or disk change.

  • A bare-stamp resume token inside a range retention had collected could resume and skip
    events silently after an old entry arrived late.
    Such a token (issued before 0.30.0, or
    start_at a stamp) was refused when it was older than the oldest entry the oplog retains, and
    an entry stamped before everything retention collected (a delete of a document the member never
    held, by entries) could arrive late and become that oldest entry, lowering the bar; the token
    then resumed with the collected events missing. It is now also refused when retention has
    collected an entry above it, from the recorded per-origin horizon, which a late arrival cannot
    lower. A token above everything collected still resumes. ADR-173's addendum. No rollback
    boundary and no wire or disk change.

  • A delete now reaches a member through a member that never held the document. A
    member that took a whole-database snapshot recorded nothing for a document it had never
    held and the sender had deleted, so a member that then took its snapshot kept the
    document: detected by the count check, never repaired. The snapshot receiver now records
    the tombstone, and a Delete entry that arrives for a tombstone it already holds under
    the same stamp is appended to the oplog when it is missing (one extra read, only on that
    path) instead of being taken as already applied. It does not change a member whose pull
    a completed snapshot's grant already covers; the hole (ADR-224, below) is what handles that.
    ADR-228. No rollback boundary: the tombstone is an ordinary one.

  • A member could be stuck catching up indefinitely when a snapshot page changed the width
    of a client-supplied vector configuration, and now it is not.
    A member holding a byo
    configuration of one width, whose peer turned it off and configured another (the only way to
    change it), refused the snapshot page in every round: the snapshot never completed, the
    catching-up marker stayed set, and the member refused requests until the origin changed the
    width back or the member's store was wiped. A page's dated configuration now applies as the
    off-then-on it stands for, in one transaction, keeping the stored vectors (which the new
    width skips); any other configuration a page carries that this build cannot hold is now
    skipped, logged and counted in kimmy_sync_ddl_refused_total like an index it cannot
    build, instead of failing the page. ADR-227. No rollback boundary: a 0.45 receiver still
    refuses the page.

  • A snapshot now carries the indexes the sender has dropped. A member that took a
    snapshot (or a scoped repair) from a member that had dropped an index kept the index,
    so a unique index the cluster had dropped went on refusing inserts on that one member,
    a TTL index went on expiring documents, and nothing detected it. The page now lists the
    sender's index-drop tombstones; the receiver judges each as a replicated DropIndex
    (an index created after the drop is left alone), records it, and serves it onward.
    Together with the entry below about a member that caught up by snapshot (ADR-224), a member behind a snapshot-caught-up peer gets the drop
    too. An older sender writes nothing and an older receiver ignores the list, so a mixed
    cluster keeps the old behaviour until every member has rolled. ADR-225. No rollback boundary.

  • A snapshot now carries a vector configuration that was turned off. A member that took a
    snapshot (or a scoped repair) from a member that had turned a collection's vector
    configuration off kept it, and went on embedding, refusing chunk writes and answering vector
    search from a configuration the cluster had dropped; nothing detected it. A configuration now
    carries the stamp of its latest change, a page says when it was turned off, and the receiver
    applies the later of its own and the page's. Two behaviours change with it: the entries
    path is last-writer-wins on that stamp (two concurrent changes from two members now converge on
    the later one, and a re-served older entry no longer flips a configuration back), and a
    disable of a configuration that is already off is now stamped and logged, so a retried
    disable writes (and replicates) where it used to do nothing. An unstamped state written before
    this release counts as older than any stamped change, so a 0.45-era change made after the
    origin's off is lost when a page arrives. Together with the entry below about a member that caught up by snapshot (ADR-224), a member behind a
    snapshot-caught-up peer gets the off too. An older sender writes no stamp and an older receiver
    ignores it. ADR-226. No rollback boundary.

  • A member that caught up by a whole-database snapshot no longer serves a peer a
    window across history it holds only as state.
    It held the sender's documents as
    entries and its schema changes, drops and deletes as state only, yet advertised
    the whole span, so a peer behind that span was served the documents without the
    changes beside them, witnessed past them, and never asked again. Seen on a real
    cluster loop, silently, for a vector configuration turned on or off, a collection
    drop, an index drop and a delete the member never...

Read more

0.45.0 - 2026-10-06

Choose a tag to compare

@github-actions github-actions released this 06 Oct 11:42
28afec4

Release Notes

This is a minor release with no breaking change and no rollback boundary: no
configuration key, table, kimmy.format schema (still 4), redb version (4.3.0)
or wire frame that 0.44.0 cannot read changed, and every answer that differs
from 0.44.0's is a request that was refused before and works now. What is new:

  • A $lookup sub-pipeline's $match reads the let names through $expr, and
    a correlated equality is a hash join
    (10,000 inputs against 100,000 foreign
    documents: 104 s to 86 ms), with explain on aggregate to see the plan.
  • A member whose single writer is wedged yields what it owns and says so in
    /v1/topology (degraded), and does not take the work back while the writer
    is still held (ADR-220).
  • A restored member reads its lost writes back over its peers' contacts as well
    as its own
    , so a filter that drops only what it opens no longer leaves it owing
    the replay (ADR-212's addendum); kimmy_replay_total counts it.
  • A $let variable the body does not read can no longer fail the expression
    (ADR-219), and a write a batch or a unique index refuses no longer advances the
    clock.

Rolling back to 0.44.0: nothing to do. A three-member cluster on this build wrote
documents, held a writer wedge, restored a member from a backup (and replayed its
lost writes), and 0.44.0 started on the same data directories and configuration
with no repair and no refusal, held the same counts and replicated writes; 0.45.0
then started on them again and did the same. What 0.44.0 does not know of: the
new optional replay_from field on Vectors and writer_wedged in the facts
block, which it ignores; the ReplayEntries and ReplayAck frames, which a
member sends only to a peer that asked, and 0.44.0 never asks; degraded in
/v1/topology, which a client ignores. explain on aggregate is a 422 on
0.44.0, and a let name in a sub-pipeline $expr a 400, as before. A mixed
roll answers by member until every member is rolled.

Added

  • A $lookup sub-pipeline's $match can read the let names, and a correlated
    equality is a hash join.
    {"$match": {"$expr": {"$eq": ["$order", "$$oid"]}}}
    now works in a sub-pipeline (it was a 400), at any depth of the filter and in any
    $match of it, and a $match whose conjunction holds such an $eq (either
    argument first, any expressions) is answered from a table of the foreign
    collection instead of a nested loop: 10,000 inputs against 100,000 foreign
    documents went from 104 s to 86 ms in a release build. The first input document
    is run as the nested loop, anything that would raise falls back to it, and the
    join never answers or fails differently from it. The equality is $expr's $eq:
    numbers across types are equal, null and a missing field are the same value
    (unlike the localField form, where a missing key joins nothing), arrays compare
    whole. Shapes that do not join ($or, $gt, $elemMatch) run as the nested
    loop. A $$name as a plain value of a clause is still refused. explain reports
    foreignScanHashJoin, with the key, the residual clauses and the fallback. No
    action on upgrade; an older member answers 400 during a roll. See
    aggregation.md, "The let / pipeline form", and ADR-222.
  • /v1/topology says when a node cannot serve. A node's entry gains an
    optional degraded: "writer" while its single writer is wedged (below) and
    "runtime" while its async runtime has stalled. It is advisory (prefer a node
    without it), appears only on a live node, and its absence is not a promise of
    health. New series: kimmy_writer_wedged and kimmy_writer_hold_age_seconds.
    A new test switch, KIMMY_TEST_HOLD_WRITER=<ms>,<holder>. See operations.md,
    "A wedged writer", and ADR-220.
  • aggregate takes explain: true and answers with a plan, running nothing.
    {"pipeline": [...], "explain": true} answers {"explain": {"executed": false, "source": ..., "stages": [...]}} with no documents and no count: the leading
    $match's access path as find would take it (strategy, index,
    indexFieldsUsed, probes, without the counts a scan finds), one entry for each
    later stage, a $lookup's strategy, and a $vectorSearch's configuration. It
    needs the access a run needs, $lookup collections included. It was a 422
    (explain: unknown field); an older member still answers that during a roll.
    The MCP aggregate tool takes it too. See aggregation.md, "Performance", and
    ADR-221.

Changed

  • A member whose writer is wedged now yields what it owns, and no peer hands it
    work.
    One hold of the single writer that lasts more than 60 seconds used to
    delay every class of owned work (expiry, webhooks, embeddings) on the member and
    be detected by nothing. It is now a bad item for every class that owns work, so
    the member yields them to a peer by the usual rules, and a member that advertises
    it (writer_wedged, optional on the wire) is refused as a target for every
    class. Index builds and index drops are never read as a wedge, which is a stated
    gap. An index drop now has a holder of its own, index_drop, in
    kimmy_write_lock_*{holder} (it was drop). responsive in a member's block is
    now absent until six ticks have been judged, where it read false. No rollback
    boundary.
  • A restored member now reads its lost writes back over its peers' contacts
    too.
    A member restored from a backup asked the peers that hold its later
    writes only on connections it opened, so a firewall that dropped what it opened
    and let in what the peers opened left it owing the replay (serving as
    unknown, owed_replay) with the writes one hop away. It now asks on the
    contacts its peers open as well. No configuration, and no action on upgrade:
    a 0.44 peer ignores the ask and the member waits as before. Counted on
    kimmy_replay_total{via,result}. A member that can dial nobody still cannot
    clear. See operations.md, "Serving as unknown because a member owes the
    replay", and ADR-212's addendum. KIMMY_TEST_REFUSE_DIAL is a new test switch.
  • A $let variable the body does not read can no longer fail the expression.
    $let evaluated every variable before its body, so a guard that kept the body
    from reading a variable did not keep the variable from failing: {$let: {vars: {r: {$divide: [1, "$n"]}}, in: {$cond: [{$eq: ["$n", 0]}, 0, "$$r"]}}} was a
    400 over n: 0 and now answers 0. A variable is evaluated when the body
    first reads it, once, and a variable nothing reads is not evaluated at all.
    Only requests that failed are affected: an answer that was given is the same.
    When two variables would fail, the error is now the first one read rather than
    the first one written. A $lookup let still evaluates every variable it
    declares. See aggregation.md, "Variables", and ADR-219.

Documented

  • When a schema-change confirmation answers backoff. A confirmation that
    arrives during a member's back-off is answered at once, as ADR-191 says; one
    already waiting behind the push that failed is answered backoff when that
    push fails, so the answer can take up to the request timeout. See
    operations.md, kimmy_ddl_confirmations_total.

Fixed

  • A write a batch or a unique index refuses no longer advances the clock.
    A batch insert whose later document collided, or repeated a unique value,
    still ticked the hybrid clock for the documents before it, and an insert, a
    replace or a find_and_modify that a unique index refused ticked it too,
    because the refusal was found after the stamp was minted (the two leftovers
    of 0.44.0's entry on writes that find nothing to do). A batch now checks every
    document, in order, against the stored documents, the ids before it and the
    indexes, and mints a stamp for each, in submission order, only once all have
    passed; a single write files its index entries before it mints. Nothing a
    caller sees changes except the clock: the same request fails with the same
    answer. Still leftover: a multi update chunk, or a scoped batch, whose
    later document fails (an update operator that cannot apply to it, or a unique
    value) aborts after the documents before it in the same chunk were minted
    for, and so still ticks the clock; the abort writes, logs and publishes
    nothing.

Download kimmyd 0.45.0

File Platform Checksum
kimmyd-aarch64-unknown-linux-musl.tar.xz ARM64 MUSL Linux checksum
kimmyd-x86_64-unknown-linux-musl.tar.xz x64 MUSL Linux checksum

0.44.0 - 2026-10-05

Choose a tag to compare

@github-actions github-actions released this 05 Oct 02:19
457fa87

Release Notes

This is a minor release with breaking changes. Several requests that used to
answer 200 with a quietly wrong result are now refused with a 400, and a few
answers that were wrong are now right, so a client that depended on the old
answer sees a change. Read the Breaking: entries below before rolling.

Breaking changes in this release:

  • An array written inside an expression is evaluated, so a field path in it is the field's value.
  • $inc and $mul keep a 32-bit integer 32-bit, and PUT keeps _id's type.
  • modified counts the documents an update changed, and a document left as it was is not written (and has no change event).
  • A $lookup joins on every element of an array its key crosses or holds, and its attachments are held to the ceiling.
  • A projection through an array keeps the array and every element.
  • An update that writes one path twice is a 400 and writes nothing.
  • A filter's $expr that cannot be evaluated against a document fails the request with a 400 instead of dropping the document.
  • A $regex pattern, and the patterns of one filter together, have a size limit.
  • A regular expression inside $in, $nin or $all, or as the operand of $ne, is a 400.
  • An $elemMatch inside $all is a 400.
  • An empty $all is a 400.

A member that cannot do a class of work (expiry, webhook delivery or
embedding) now hands it to a healthy peer that can, and takes it back after a
quiet spell; it keeps owning the class until every live peer has confirmed, so
the overlap is a duplicate and never a gap (ADR-213). It adds no Breaking
entry: nothing that was accepted is now refused. A webhook subscription taken
over behind a long run of older entries keeps delivering (ADR-218).

Rolling back to 0.43.0: no configuration key was added (every knob of this
release is an environment variable or a constant), no table, kimmy.format
schema (still 4) or wire frame changed, so 0.43.0 starts on a 0.44.0 data
directory and its config file without edits. What it loses or ignores:

  • kimmy.replay-floor gains owed = <node id> lines after its two floor
    lines. 0.43.0 reads the floor lines and skips every other key, so it reads
    the same floor. It keeps the file as it is when it does not move the floor,
    so stale owed lines can survive a 0.43.0 run; back on 0.44.0 each holds a
    marked member at 503 for cluster.catch_up_wait_secs, and is released
    after the dwell if that member never returns.
  • A collection owed an embedding scan is marked in the store's metadata table
    (vector_rescan:<collection id>). 0.43.0 looks metadata keys up by name and
    never lists them, so it ignores the marks: it does not run those scans, and
    documents the scan would have embedded wait. The marks stay in the store, and
    0.44.0 scans the marked collections at its next start.
  • Answers revert to 0.43.0's: the 400s above answer 200 again, with the
    results described in each entry. A mixed roll answers by member until every
    member is rolled.
  • kimmyd check-store and kimmy_store_repairs_total do not exist in 0.43.0;
    nothing is stored for them.
  • Ownership yielding (ADR-213) stores and sends nothing 0.43.0 reads
    differently. kimmy.last-start is an advisory file beside kimmy.last-exit;
    0.43.0 never opens it, so it is ignored (an interrupted write can leave a
    kimmy.last-start.tmp.<pid> that 0.43.0 does not clean up, harmlessly). The
    replication frames gain optional facts_gen and echo fields, and the facts
    block optional fields for each class's state, cause and responsiveness; the
    frames and the block are decoded leniently, and 0.43.0 skips fields it does
    not know, so a mixed cluster works, with a 0.44.0 member that yields owning
    the class until no 0.43.0 member is live, as an older peer never confirms.
    There is no table or schema change. The webhook fix (ADR-218) changes no wire,
    schema or configuration.

Added

  • $vectorSearch, an aggregation stage. The first stage of a pipeline can
    run a vector search and hand the hits to the rest of it, so a search can be
    filtered further, joined and grouped inside the database. It takes the fields
    of the vector_search endpoint (query or vector, k, per_document,
    filter), checks the same things and answers the same errors, and each hit
    enters the pipeline as its source document with _score and _chunk added.
    It needs read on the collection, which carries search, so a token granted
    only search is refused by aggregate's read check; k bounds what enters
    the pipeline, aggregate still has no explain, and anywhere but first is a
    400. The names other systems use, queryVector,
    numCandidates, limit and index, are refused, each naming the spelling to
    use or saying there is no such knob. See Aggregation, Vectors and ADR-216.

  • An arrayFilters entry takes $expr. {"$expr": {"$lt": ["$$line.qty", "$$line.min"]}} selects the elements whose quantity is below their own
    minimum, where an entry could only compare a field with a constant. The
    expression reads the element as $$<identifier>, at the entry's top level or
    under $and, $or and $nor, on update and find_and_modify; a scalar
    element is $$n like any other. A field, $$ROOT and $$CURRENT in it are
    refused, as is an $expr that names no identifier or a second one, and an
    element it cannot be evaluated against fails the update, naming the document,
    as a filter's does. No request that works changes. MongoDB refuses $expr
    here, so this is an extension and is recorded in the Deviations register. See
    Query language, "Positional updates", and ADR-209.

  • The set operators, $objectToArray and $arrayToObject in the
    expression language.
    $setUnion, $setIntersection, $setDifference,
    $setEquals, $setIsSubset, $anyElementTrue and $allElementsTrue read
    each array as the set of its distinct members, where two values are one
    member exactly when $eq calls them equal (1 and 1.0 are one), and a set
    result comes back first-seen: members in the order they first appear,
    arguments left to right. $objectToArray turns a document into
    [{k, v}, ...] in field order; $arrayToObject reads [[k, v], ...] or
    [{k, v}, ...], and a repeated key keeps its first place and its last
    value. Null or missing in is null out; a non-array (or, for
    $objectToArray, a non-document) is a 400. A Decimal128 in a set input
    is a 400 rather than compared, and $arrayToObject refuses a mixed form,
    a pair of the wrong shape, a non-string key, and a key holding a NUL, each
    naming the element; any other string is a key, empty, dotted and
    $-prefixed ones included, so a document turned into pairs and back is the
    same document. Every one of these names was an unknown operator before, so
    nothing that used to be accepted is refused.
    $zip and $sortArray stay unknown operators. See aggregation.md, "Sets"
    and "Documents as pairs", and ADR-207.

  • kimmyd check-store verifies a stopped member's store against redb's page
    checksums, writing nothing.
    A serving node never verifies them: redb checks
    them only in its repair after an unclean stop and in its own integrity check,
    which needs the database to itself, so on a store closed cleanly a flipped bit
    in a page that still decodes was served as data and nothing ever looked. The
    command runs redb's integrity check over the store in the configured data
    directory, through a view that keeps every write redb makes in memory, under
    the store's lock, and prints one JSON line (verdict, detail,
    unclean_close, rolled_back, elapsed_ms, bytes_read) with its log on
    stderr. It exits 0 when the store verifies, 65 when it is damaged, and 1 when
    it could not be checked (a node holds the store, there is no store, a newer
    build wrote it, a read failed, or it was stopped). It needs no serving
    configuration, reads the whole store, and does not read hnsw/. See
    operations.md, "Checking a store", and ADR-204. Not a rollback boundary:
    nothing stored changes.

  • A start that repairs the store says whether the repair rolled back the
    latest commit.
    redb's repair after an unclean stop discards the latest
    commit when it fails verification, and did so silently. The WARN line
    database repaired after an unclean stop now carries rolled_back, and a
    new counter, kimmy_store_repairs_total{rolled_back}, reads 0 or 1 for the
    process. A rollback is the commit the stop interrupted, or damage to it;
    after a clean shutdown it means the store is damaged.

  • A member that cannot do a class of work gives it to a peer that can. Each
    of expiry, webhook delivery and embedding now beats a heartbeat, and a
    dedicated thread judges those heartbeats and the runtime's own stall probe
    every five seconds. A class that has failed locally in 6 of the last 12 ticks
    (expiry: three passes of five), or has not moved for longer than its own bound, is
    advertised as yielded to a peer that is healthy, responsive and would own the
    work, and the member goes on owning it until every live peer has confirmed.
    It takes the class back after two quiet minutes, doubling that for a class
    that yields again, up to 32 minutes. A provider that hangs or answers 5xx, a
    webhook endpoint that is down and a missing key never make a member yield, a
    paused process or a loaded host reads as nothing, at most one member yields a
    class at a time at a few members, and when two members are stalled in the same
    class none yields it (a fault that follows the data costs one hand-off). A
    start after storage_failed or task_died, or after an unclean end that the
    disk's free space, a short previous run or an earlier unclean end corroborates,
    begins with every class yielded, so a node in a full-disk restart loop stops
    holding its share. A stuck writer is not detected yet. A member t...

Read more