Skip to content

v7.68.0

Choose a tag to compare

@github-actions github-actions released this 02 Oct 18:51
· 2496 commits to main since this release
c245abe

Pairs with engine-service-v0.1.143. The engine carries 14 new changesets (vectors-020 to vectors-026, fk-006-1, vectors-023-1 and staging-6-drop-landing-schema), the refusal of a chunk write that has no live manifest row (nexus-z0o2p.24, NX_OWNERLESS_WRITE_MODE: the managed service starts in log-only, and the local engine launch sets enforce), the engine's own hourly reaper of ownerless chunks and the live(c) read predicate (RDR-192), the removal of /v1/staging/* and the staging schema (nexus-z0o2p.27, irreversible by binary rollback, see Removed), the HNSW scan budget (nexus-wbfpw.47) and the census count fix (nexus-wbfpw.60). Two of its wire changes are not additive: a client older than this release has its chunk-before-owner writes refused once the engine enforces, and no 7.x client calls the removed staging routes. Local installs get the new engine with this release, and after nx upgrade every long-lived nx-mcp server and hook-spawned nx must be restarted (nx daemon restart-stale).

Added

  • nx catalog doctor --visible-outside-manifest flags a chunk that live(c) should hide and does not (nexus-wbfpw.26, RDR-192 Step 14). For each knowledge__ collection it takes the census superseded bucket (manifest-less chunks whose owning note is live) and asks the normal read path, never include_non_live, whether any is visible; each one that is names the collection, note and chash and fails the check. It is a regression canary: a healthy engine finds nothing. It reads at most 300 per call, probes each census page as it lands, probes at most 5,000 chunks and checks a 120-second budget between calls, and prints checked N of M. It probes get only, not search. A run cut short before it probed anything reads INCONCLUSIVE and exits 1; with the superseded bucket empty everywhere it reads "nothing to probe" (exit 0, no PASS word, vacuous: true in --json); it reads not applicable on a box with no knowledge__ collection or an engine without the census route.

  • store_put and nx store put say which chunks a re-put removed (nexus-wbfpw.25, RDR-192 Step 13). When re-putting a note under the same title deletes the old chunk, the result gains a second line, Superseded: N chunk(s) removed: [<chash>, ...] (at most three chashes spelled out, then (and M more)). A first put, an identical re-put, and a re-put whose dropped chunk another document still owns print exactly as before, because the line reports what the engine's sweep deleted, not what the manifest dropped. The engine half is an additive change to the sweep response: each sweep_detail entry gains swept_chashes (hex, ascending, at most 300) and swept_chashes_truncated; swept stays the exact count. Against an engine that predates the field the line carries the count only. When the sweep errors instead (a gate or statement timeout), the note is stored and the old chunk stays, hidden from search, and the engine reaper collects it later if its collection passes the reaper's census and floor; the result says so: Superseded: the sweep did not finish; up to N replaced chunk(s) were not removed. .... A put whose acknowledgement was lost and resent prints neither line, because the resend's sweep removes nothing and the first attempt's answer is gone.

  • nx doctor warns when the engine has seen a chunk write with no owner (nexus-20onx, RDR-223 Phase 3 Step 2). The new "Ownerless writes" row reads ownerless_writes_refused_total and ownerless_writes_would_refuse_total from the engine's /v1/status. A count above zero means a client older than the Phase 2 release is writing, a process upgraded on disk still runs its old code, or a caller that is not the nexus client (curl, a script) writes that way; the counters are since-boot and name no cause. After upgrading conexus, restart every long-lived nx-mcp server (one per Claude Code session) and any hook-spawned nx. The row is green and reads "not applicable" when the engine cannot be reached or predates the refusal. The cloud cutover order (log-only soak, then enforce by environment value and a same-tag redeploy) is docs/operations/ownerless-write-cutover.md.

  • nx t3 quarantine restore brings chunks back out of a quarantine- collection with no manifest row required, and makes the ones whose own metadata names a live document visible again (nexus-2x9xa and nexus-wbfpw.49, RDR-192 Step 9 Day-2). The only restore until now, gc_restore_rereferenced, needs a manifest row in the collection naming the chunk, and a chunk the reaper quarantined wrongly has none, so recovering it meant hand SQL from the gc_audit chash list. Name the collection the chunks came from and one source: --chash (repeatable), --audit-id (the chash list of a reaper_quarantine row; a gc_quarantine_orphans row, which is what nx index repo and nx t3 gc write, lists only a sample of what it moved and is refused with that reason), or --quarantined-since / --quarantined-before (the chunks in the sibling quarantined from that collection inside the window, which works for any quarantine). The chunks move in one engine statement under the exclusive sweep gate, with text, vector and created_at intact and the quarantine stamp and the reaper's own tags removed. A chunk the collection already holds is skipped and reported present, never overwritten; a chash found nowhere reports missing; --dry-run classifies and moves nothing. Moving bytes is not enough, because a chunk with no live owning manifest row is hidden from search and get, so --reattach, the default, also writes the manifest row when it can. Reach, stated plainly: it reaches chunks whose own metadata names their document (written before RDR-108, or by nx store put) and single-chunk legacy notes. A chunk the indexer cut from a file (docs__, code__, rdr__) carries no document key, reads no_live_owner even when its file's document is live, and comes back as bytes that stay hidden: the remedy there is nx index repo --force on the owning file, and the output says so instead of printing a re-put command. It writes nothing and reports superseded, with a reason, when the document is stamped complete (its manifest is authoritative, so a manifest-less chunk of it is stale, including a document emptied on purpose), is mid index run or its last index run failed (its manifest is partial), was cut from another content hash, already holds a chunk at that position, has the position past its registered chunk count or manifest rows under another collection, or when another stored chunk in the collection or its quarantine sibling names the same document and position (two versions of one position are both refused, even when they land on different pages or in different runs). The manifest is never changed to make room and documents.chunk_count is not touched; the output says M of N attached when a multi-chunk document ends with fewer rows than it registers. A chunk that is not attached stays hidden, and the verb says so with its exit status: exit 3 when every requested chunk is back but at least one stays hidden from search and get (a dry run exits 3 when a real run would), with hidden, owners and partial_owners in --json. In a knowledge__ collection the output prints nx store put - --collection C --title 'T' for a chunk with no live owner (nx t3 backfill-manifest does nothing for this class); a superseded chunk gets no recipe, because its document's current text is live. The reaper may take a hidden chunk again 30 days after the restore; the output prints that date. --no-reattach moves bytes only and still reports what reattach would have done; running the verb again attaches chunks that are present. Each restore or attach writes one gc_audit row, operation=quarantine_restore (nx catalog gc-audit list --operation quarantine_restore). New engine route POST /v1/vectors/gc/quarantine-restore, functions nexus.quarantine_restore_chunks and nexus.quarantine_reattach_plan (changeset vectors-025); the client verb needs an engine that carries it and exits 4 against one that does not. Exit status is 1 when a requested chunk was missing or had a different embedding width (a dry run too; wins over 3), 5 when the engine refused, and 6 when the engine was busy: a manifest writer held the collection's lock for more than 2 seconds, or the call lost a deadlock, the busy statement rolled back (each quarantine collection is restored in its own transaction, so the engine says when an earlier one had already committed: nothing_moved: false, with its audit ids), and the engine answered a typed retryable 503 (reason: quarantine_restore_busy, Retry-After: 5) rather than a 500. A failure after the first page, an unexpected one included, still prints the report and audit ids of the pages already committed. See docs/cli-reference.md and the procedure in docs/operations/engine-reaper.md § Getting a chunk back.

  • Engine: POST /v1/pipeline/reset_uploaded, and uploaded_chunks in GET /v1/pipeline/counts (nexus-z0o2p.11). A streaming PDF upload killed hard leaves the head of the document flagged uploaded, and the writer state that made sending only the tail safe died with the process. reset_uploaded puts every chunk of one run back to not-uploaded and zeroes chunks_uploaded in one transaction, and keeps the extracted pages, the chunks and their embeddings. It checks run_epoch like every other write route, so a run another process took over cannot reset the new owner's flags; a ref naming no row, or another tenant's, resets nothing. uploaded_chunks counts the embedded chunks flagged uploaded. The client refuses an engine that answers without the new field.

Changed

  • The engine's orphan GC waits 30 days after a chunk loses an owner, and for 30 days after this engine is first deployed nothing that already exists is old enough (nexus-wbfpw.15, .16, .17, RDR-192 Steps 7 and 8). nexus.chunk_is_reapable is the one predicate for "garbage collection may take this chunk": no own-collection manifest row in any owner state, the later of last_written_at and the chunk's orphaning time older than 30 days, and not in a quarantine collection. Database triggers on the manifest record the orphaning time in a new table, nexus.chunk_orphaned_at, whenever a manifest change drops a chunk's owner row (any owner, whether or not another remains; a chunk written within the last hour is not recorded), so the 30 days run from when the chunk became ownerless: a deleted file's chunks and the old tail of a multi-batch re-index stay for 30 days. The record is a separate table, so dropping 5000 chunks' manifest rows adds tens of milliseconds (one laptop-container measurement: 51 to 60 ms on a 35,000-chunk table, against 11 to 14 s when the record was a chunk-row update) and rewrites no chunk row or vector index. gc_quarantine_orphans and its bounded form use it, so the cleanup at the end of nx index repo moves less than it did (often nothing right after a delete). vectors-020 added last_written_at with DEFAULT now(), so every existing chunk takes the migration time and none is reapable until 30 days after the deploy; the first cleanups after an upgrade, and any reaper census, read zero for that long. New read-only route POST /v1/vectors/reapable lists what the predicate selects, with a keyset cursor (after_chash); no client verb uses it yet (nexus-wbfpw.18, .19).

  • The engine refuses a chunk write whose chashes have no live manifest row (nexus-z0o2p.24, RDR-223 Phase 3). POST /v1/vectors/upsert-chunks and /store-put answer 422 (reason: ownerless_chunk_write, naming /v1/catalog/manifest/write_many and /append) when any chash in the request has no live manifest row in the collection; writes of chunks a live document already owns (a collection re-embed, a metadata refresh) are unchanged. Not additive. A client older than this release gets 422 on its ownerless writes, so after the upgrade every long-lived nx-mcp server and every hook-spawned nx must be restarted, not merely upgraded. NX_OWNERLESS_WRITE_MODE is enforce or log-only; unset means log-only (the engine logs and counts such writes, ownerless_writes_would_refuse_total on GET /v1/status, and refuses nothing), and the local engine launch sets enforce itself, so a local install enforces from the first boot of the upgraded engine, with no soak: restart every nx-mcp server and hook-spawned nx right after nx upgrade (nx daemon restart-stale, then nx doctor's Process freshness row), or the first write from a process still running old code is refused. The log line names the client by User-Agent and the new X-Nexus-Client-Version header that the shared client classes now send (the vector client, the T2 stores, the catalog client, the token store and the scratch store; absent marks a client older than this release, or an engine request that goes through none of those classes, such as a hook call or a status, version or health probe, none of which writes a chunk); it is rate limited to one per route, tenant and collection per minute. The 422 text says to upgrade conexus and restart nx-mcp / Claude Code sessions. nx collection re-embed resends only the chunks that are still owned when a chunk loses its owner mid-run.

  • The engine reaps ownerless chunks on its own, once an hour: it moves them to quarantine, and deletes what it moved 14 days later (nexus-2x9xa, RDR-192 Step 9). A pass visits every tenant that has a service token, plus the default tenant, and every collection of every prefix (knowledge__, docs__, code__, rdr__). It first expires the tenant's quarantine- collections, but only the chunks the reaper itself moved: every moved chunk is tagged quarantined_by: engine-reaper, and the new function nexus.reaper_expire_quarantine deletes tagged chunks stamped more than NX_REAPER_QUARANTINE_RETENTION_DAYS (14) days ago, with no fraction floor and at most 5,000 per origin per pass, never a chunk a manifest row of the origin still names (those are counted as expiry_protected, which is not a refusal). Quarantine that nx index repo or nx t3 gc filled carries no tag; the engine never touches it, and it keeps the client's NX_GC_QUARANTINE_DAYS expiry (so a repository nobody has indexed for months keeps its quarantine). The split is symmetric: the client's expiry (nexus.gc_expire_quarantine, the call nx index repo makes, and the gc/expire-quarantine route with or without force) now skips the chunks the reaper tagged and counts only its own rows toward its floor, so a client NX_GC_QUARANTINE_DAYS below 14 cannot shorten what the reaper moved. Then it moves the chunks nexus.chunk_is_reapable selects (no owner row in any state, 30 days since the chunk last had an owner or was written) into the collection's quarantine- sibling, at most 300 per collection per pass. It is the safety net for the post-commit sweep's fail-open paths (a gate timeout, a statement timeout) and for a crashed multi-request run, which leave replaced chunks behind. A moved chunk is moved back when a later nx index repo or heal names it in a manifest row of a repository collection's origin; for a knowledge__ collection nothing does (a re-put embeds a fresh chunk in the origin) and a manifest row that names the chunk only protects the quarantine copy from expiry. nx t3 quarantine restore restores by chash, gc_audit id or window (see its entry). A pass writes gc_audit rows (engine-reaper, operation reaper_quarantine, the chashes, one per collection it moves from) and logs event=reaper_pass for every tenant, candidates=0 included. It refuses, moves nothing, logs a WARN with reason= and a running refused_total, and writes one gc_audit row (operation reaper_refused, with the reason, counts and up to five title/source_path samples, once per collection each time the reason changes) when: the tenant has no verified rdr192-manifest-backfill record; the collection is not registered or not live; the collection's whole reapable set is at least 100 chunks and more than 0.25 of the collection (judged before the census, so a refused collection pays for none); the manifest-less census, re-run in the engine and bounded to 60 seconds, reads any legacy-unmanifested or unclassified chunk, reads a chunk count other than the one being judged, or times out; or another statement (the dry run, the move, one sibling's expiry) hits its 25 second bound (statement_timed_out_total; an expiry's row is filed under the quarantine- collection's name). A collection whose census, dry run, move or expiry times out three passes running rests for 2, 4, 8 passes and so on, at most 24 hours (census_backoff_total; census_timed_out_total counts the timeouts), and a run that spends its wall-clock budget resumes where it stopped, so a slow collection cannot starve the others. A busy sweep gate and a row-lock timeout skip the collection until the next pass, with their own counters (gate_busy_total, lock_timeout_total); they are not refusals. Settings (engine environment): NX_REAPER_ENABLED (true/false; any other value warns and leaves it on), NX_REAPER_INTERVAL_SECONDS (3600, at least 60), NX_REAPER_BATCH_SIZE (300, the most it accepts), NX_REAPER_FLOOR_FRACTION (0.25; the floor on the move only, the expiry has none; 1.0 turns it off), NX_REAPER_FLOOR_MIN_CHUNKS (100), NX_REAPER_WALL_CLOCK_BUDGET_SECONDS (600, a whole run), NX_REAPER_CENSUS_TIMEOUT_SECONDS (60), NX_REAPER_QUARANTINE_RETENTION_DAYS (14, the engine's retention for what it moved), NX_REAPER_FLOOR_EXEMPT_COLLECTIONS (comma-separated tenant/collection entries, or bare collection names that match in every tenant, exempt from the MOVE floor only, for draining a collection that is legitimately mostly garbage; logged at boot and on every pass that uses it, and the move's audit row records floor_fraction 1.0); a malformed value logs event=reaper_setting_invalid and uses the default. There is no grace setting: it is the predicate's 30 days. The first pass runs 60 seconds after the engine starts. For 30 days after the engine carrying vectors-020 is deployed nothing that already exists is old enough, so the reaper moves nothing in that time; one cliff follows at deploy plus 30 days, when every existing orphan ages at once, drains at 300 per collection per hour, and is refused by the floor in a collection that is mostly orphans. A tenant with chunks and no live service token (a cloud tenant idle for more than seven days) is not visited until it holds a token again. New changesets vectors-024 (functions nexus.reaper_quarantine_chunks and nexus.reaper_expire_quarantine) and vectors-026 (nexus.gc_expire_quarantine skips the reaper's rows); no row changes at deploy. The operator runbook is docs/operations/engine-reaper.md.

  • nx t3 gc quarantines what the engine's reapable predicate selects instead of deleting by chunk id, and --orphan-window is gone (nexus-wbfpw.18, RDR-192 Step 8). Paired client change: it needs the next engine tag, which carries POST /v1/vectors/reapable and the reapable-aware gc_quarantine_orphans (nexus-wbfpw.16, .17); against an older engine the verb refuses and says so. The verb used to compute orphans in the client (catalog alive-set against indexed_at) and delete them by id, which could hard-delete a chunk a client re-wrote after the snapshot. It now lists with POST /v1/vectors/reapable (advisory, paged by keyset) and acts through POST /v1/vectors/gc/quarantine-orphans, whose own statement carries the predicate and takes the sweep gate, so a racing write wins. What you will see change: chunks are moved to the quarantine-* collection, not deleted, and the move frees no storage until the chunks expire. Each side expires only what it moved: the client expires the chunks the client moved once they are older than NX_GC_QUARANTINE_DAYS (default 14), and the engine expires only the chunks its own reaper moved. The client expiry is run by nx index repo for a repo's code, docs and rdr collections and by nx t3 gc for any collection it moves (after its own move, and on a run with nothing to move), so a knowledge__* quarantine, which no repo index sweeps, expires too; the verb prints Client expiry of <quarantine>: N expired, M refused. The operator restore verb nx t3 quarantine restore ships separately (nexus-wbfpw.49) and refuses this verb's audit rows (a sample, not every chash), so restore by window or --chash; quarantine-* collections now appear in collection listings; the clock is the later of last_written_at and when the chunk last lost an owner (30 days, fixed in the engine), not indexed_at, so a chunk with no indexed_at is now a candidate once old enough; --orphan-window is refused with a message naming its removal; the engine records one gc_audit row per batch and the verb's own t3_gc row, t3_gc_chunks_deleted event, Summary: deleted N output and Protected by pending tombstones line are gone; for 30 days after the engine upgrade nothing that already exists is reapable, so the verb reports 0 reapable chunk(s); nothing to do and exits 0, and there is no replacement for --orphan-window as a way to reclaim space sooner. The move is the bounded form of the route (2,000 rows per call, looped until drained), asking for the engine's full 5,000-chash audit sample so each gc_audit row records its batch. New refusals, each also reported by a dry run: the manifest-less census must read legacy-unmanifested = 0 and unclassified = 0 for the collection (read at the start of the run and again right before the move; a live legacy note with no manifest row reads reapable once old and the engine route takes no exclusion list), and a pass of at least 100 reapable chunks that is more than NX_GC_FLOOR_FRACTION (default 0.25) of the collection's stored chunks is refused unless NX_GC_FORCE=1 (the engine's own reading: the 100 minimum counts the reapable set, the fraction is compared by division). That floor is the verb's own for now: the route carries none, the reaper's floor has no HTTP route, and indexer._prune_deleted_files calls the same route with no floor (a floor on the route is nexus-wbfpw.52); NX_GC_FORCE=1 nx t3 gc is the way past a floor refusal. A --dry-run that names a refusal now exits 1, so a script can gate the real run on it. It reuses the GC family's variable names, not the engine reaper's NX_REAPER_FLOOR_FRACTION. A census response without scope_chunk_total is refused, and a bounded drain that stops short (no progress, the batch cap, or a failed batch after earlier ones moved) exits 1 naming how many chunks were already moved; each batch commits on its own, so a re-run is safe. A failed batch's own outcome is reported as unknown (a timeout can follow a commit), and a drain that moves more than the listing named says so. The RUNFENCE breaker, --allow-incomplete-index-state, the unknown-collection refusal and --allow-empty-manifest-set are unchanged. Removed as dead code with their last caller: HttpVectorClient.delete_by_chunk_ids, T3Database.delete_by_chunk_ids, HttpCatalogClient.chashes_for_collection_with_tombstone_protected and HttpCatalogClient.record_gc_audit.

  • nx store list --reapable -c COLLECTION lists the chunks the engine would reap (nexus-wbfpw.19, RDR-192 Step 10). Paired client change: needs the same next engine tag (POST /v1/vectors/reapable). Read-only, advisory, paged by keyset; prints chash, ownerless_since, days since that instant, last_written_at, created_at, title and catalog_doc_id per chunk, or 0 reapable chunks in <collection> when there are none. ownerless_since is the engine's grace clock, the later of last_written_at and when the chunk last lost an owner, so the age answers "how long ownerless"; POST /v1/vectors/reapable returns it as a new additive field, and against an engine that omits it the age falls back to the write age. It is not derived from created_at. --collection is required, and a name no collection carries is refused like nx t3 gc refuses it, not reported as 0 reapable.

  • nx store put, nx memory promote, nx catalog import and MCP store_put write a note in one request and say the same thing about every outcome (nexus-z0o2p.16, .17, .18, RDR-223 Phase 2). A note's chunks and its catalog owner rows go to the engine together, so a chunk can no longer be stored without an owner, a client that dies mid-write leaves nothing ownerless, and a failed write leaves the note's previous version as it was. The four commands share one wording (failure_message) and one firing of the post-store hooks (fire_note_chains). What you will see change: a refusal the client makes before it sends anything (a stale embedding profile, a missing Voyage key, a retired collection name) now reads as its own remedy followed by "The note was not written and its chunks and manifest are unchanged; run the command again once that is fixed" (and, if the note was stored before, that its index state may read 'failed' until a write succeeds, because the write began and then failed the index-run fence), where it used to say "retry is safe" or "may already have succeeded". An engine refusal reads "could not store ...; retry is safe" and is the only outcome that mentions chunk metadata having been refreshed. A write the engine accepted but would not stamp complete is its own outcome ("refused to stamp the document complete ... stays 'indexing'"). MCP store_put reports those three client-side refusals as not stored and removes the catalog row it minted; it reported them as "may already have succeeded" and kept the row. A landed but unstamped note no longer says it "could not confirm the catalog manifest landed". A note that could not be cataloged now names the cause. nx store put and nx memory promote refuse empty input with "nothing to store" instead of a traceback, as MCP store_put already did. An outcome a command does not recognise is an error, never "Stored:".

  • nx catalog import exits 1 when any note failed, is uncertain, or was written but not stamped complete (nexus-z0o2p.18). It exited 0 on a partial import. Unresolvable links alone still do not change the exit status. Imported chunks carry source_agent="recovery-import"; they carried none.

  • nx memory promote prints , N chunks for a note split to the embedding model's token window (nexus-z0o2p.17), as nx store put does. A promoted note used to be one chunk whatever its size, so a long entry could exceed the model's window.

  • nx index repo writes no chunk of a file that has no catalog document (RDR-223, nexus-z0o2p.20). Those chunks went through the legacy ownerless upsert, where the live(c) filter hid them from every read, and the engine will refuse them. The run now refuses such a file before chunking it, names it in the end-of-run summary with its cause (register_failed, catalog_hook_failed), records it in nx index failures under IdentityLessFile (a deliberate worktree or temp-dir refusal is not recorded, so nx doctor stays green for it), and holds the --since-head base back. The run exits non-zero in that case; a per-file register failure or a catalog outage used to exit 0. A chunk shared with a file that does have a document is now written only by that document's own write. Worktree and temp-dir refusals keep their own summary line and do not fail the run. The oversize-file fallbacks (code, prose, PDF) write nothing for such a file either (cause oversize_no_catalog_document), count it as dropped, hold the --since-head base, record it in nx index failures, and report it as not indexed rather than skipped: index fresh. nx doctor names each recorded file with its cause and the remedy. A failed durable write prints a warning. A non-service T3 (the in-memory test topology) keeps its write.

  • The pre-PG upgrade path says to run the 6.18.1 hop against a local engine, and gives the cloud as a second hop (nexus-z0o2p.27). The stranded-install banner, the retired-sqlite message, the nx doctor remedy, docs/migration-runbook.md, Getting Started, the CLI reference, the plugin's upgrade skill and AGENTS.md tell a pre-PG user to stop any local service a 7.x install left running and run nx upgrade at conexus==6.18.1 against a local engine: note your service_url and service_token (the cloud steps need both again), clear NX_SERVICE_URL, NX_SERVICE_TOKEN, NX_SERVICE_HOST and NX_SERVICE_PORT in the shell, delete service_url (and service_token) under credentials: in ~/.config/nexus/config.yml, which is a file edit because there is no nx config unset, and export NX_LOCAL=1 for the hop. Both halves are needed: NX_LOCAL=1 does not by itself override a configured service_url at the pin, and the pin's provisioning refuses when is_local_mode() is false (NX_LOCAL=0, install.mode: managed, or a ChromaDB Cloud key with no mode record), which NX_LOCAL=1 overrides. This is read from the 6.18.1 source, not run at the pin. At the cloud switch, unset NX_LOCAL before nx config set service_url. Never nx guided-upgrade --service-url, never a managed endpoint. Voyage-embedded data needs the local engine run with NX_VOYAGE_API_KEY, or it is refused or re-embedded to bge and is not expected to import into a Voyage cloud. To reach the managed cloud, make it a second hop with the current client. That hop has not been rehearsed end to end (nexus-xbqh9). Hand-carry any T2 memory you need BEFORE switching to the cloud (nx memory has no export verb, by hand it does not scale past tens of entries, and it is lossy: get prints content only, so tags, agent, TTL and timestamps are not carried; read by id, and pass the body to put on stdin). The hop uses nx store export --all / nx store import and nx catalog export / nx catalog import (catalog import last, after re-indexing from source), and nx stranded ack once on the cloud. It does not carry T2 plans, taxonomy (topics, assignments, links, centroids), document_aspects, the aspect queue, aspect_promotion_log, frecency and relevance_log (so a note's TTL is lost and an expiring note becomes permanent), the telemetry stores, DEVONthink highlights, or curated catalog metadata (author, year, corpus, meta, collection supersession); owner tumblers are re-minted. A local bge-768 collection is sent to the cloud engine under a bge name and what it answers is unverified (nexus-xbqh9); re-index it from source instead, which for code, docs and rdr content is the cheaper path anyway. nx catalog export --help and docs/catalog.md now name the real nx store export verb.

  • nx store import writes each chunk together with its owner row and stores the vectors in the file (nexus-z0o2p.19, RDR-223). The import used to store every page of chunks through /v1/vectors/upsert-chunks (the engine re-embedded the text) and write the owner manifests once at the end, so a client that died mid-file left chunks with no owner, and an imported document read index_state NULL. It now reads the file to hash it and to count each document's records (and once more to write), resolves each owner group to its document (one document however many groups resolve to it, with the summed count), then writes a document's first request as a manifest replace (a document of one request also carries its sweep and stamp in that request) and every later page as an append, in pages of up to 300 records whatever the collection's embed cap, each with its chunks and their exported vectors (the engine embeds nothing and checks the model and dimension; one that embeds anyway fails the import). Every payload chunk replaces a stored vector with the file's, including a chunk another live document shares, metadata is merged so keys another document's enrichment set survive, and the summary says how many stored vectors differed. A document is swept and stamped complete on its own last page, inside that request, so a crash leaves only the unfinished documents indexing, and the same command run again finishes them. Keep-existing (nexus-wbfpw.40) is unchanged with one exception: a document that owns chunks keeps its chunk list, and the file's chunks for it are now left out of the import and counted in "Skipped" (before, they were stored hidden); unowned_count / unowned_documents keep their names and now mean chunks left out. The exception: a document left indexing or failed whose recorded hash equals this file's hash is this file's own dead run, and a rerun finishes it with the append form (nothing dropped); a document with another hash, or already complete, stays kept; a kept document another run left indexing or failed is reported as unfinished (with the delete-and-reimport remedy), not as one "with a different chunk list". The summary also counts documents whose replaced chunks the engine's fail-open sweep could not delete (sweep_skipped; nx t3 gc removes them). --skip-existing now means "send no text or vector for a chunk the target already holds" (the record still gets its owner). taxonomy__* and quarantine-* targets are refused before anything is written. Two records of one document at one position keep both chunks, the later one past the document's highest position. Needs an engine with the RDR-223 routes (append_many with complete, begin-many with snapshot_manifest, supplied vectors); against an older one the import stops at its first request with "the engine is older than this client" and what to do (upgrade the local engine, or wait for the cloud deploy), marks the documents that request fenced, and has no fallback. A legacy two-segment target is written with the model its collection is registered with, not the export header's prefix-based guess (which a local install's engine refused).

  • nx index pdf writes each chunk together with its owner row, and a killed upload resumes without extracting the PDF again (nexus-z0o2p.11, .15, RDR-223). The streaming pipeline sends every 128-chunk batch with its manifest rows in one request, so a process killed part way leaves every chunk it wrote owned. The non-streaming branches (a small PDF, a repo PDF over the chunk cap) write the same way. A killed streaming upload re-sends from chunk 0 on the next run through reset_uploaded, without re-extracting, and the engine skips re-embedding chunks it holds. The completion stamp comes after the title, author and extraction-method passes, and a failed stage no longer releases the writer's last request. A PDF with no catalog document is refused before extraction, and --dry-run writes only to a throwaway in-memory store. The checkpoint writer and reader and ChunkLandingUnverifiedError, which nothing used any more, are deleted.

  • Every writer path stamps its document complete last, after its post-store hooks (nexus-z0o2p.34, RDR-223 Technical Design 6). A process killed in a hook (taxonomy assignment, aspect enqueue, catalog enrichment) used to leave a document that read complete, and the next run skipped it as fresh. It now leaves the run fence at indexing and the rerun redoes the write and fires the hooks again. This covers the oversize code, prose and PDF fallbacks, the batched flush in nx index repo, the note writers (MCP store_put, nx store put, nx memory promote, the recovery import) and the .nxexp import. A refused or failed stamp leaves the fence at indexing.

  • Engine: every HNSW search sets a scan budget (nexus-wbfpw.47, RDR-192). With most of an index filtered out by the live(c) predicate, pgvector's default scan limit stopped before it found ten live rows: measured recall at 10 was 0.85 to 0.89 at 98% dead chunks on shared 768- and 1024-dimension indexes, and 1.000 with the budget. Every search path (plain, both hybrid branches, the combined-query pair and the taxonomy centroid lookup) now sets hnsw.max_scan_tuples to 200,000 and a scan memory multiplier derived from a 16 MB per-search budget and the server's work_mem (4 at the stock 4 MB, 1 at 384 MB). NX_HNSW_MAX_SCAN_TUPLES and NX_HNSW_SCAN_MEM_BUDGET_MB override them on the engine; both are validated at boot and logged as hnsw_scan_budget. A filtered search that cannot fill its LIMIT can now scan up to the cap before it stops, and what that costs on a large local index is unmeasured.

Removed

  • POST /v1/vectors/upsert-reference-only (nexus-z0o2p.24, RDR-223 Phase 3). It wrote a chunk with no manifest row and had no client caller; the production edge log shows no request to it from 2026-07-03 to 2026-10-01. It answers 410 Gone. The repository method behind it stays for test fixtures.

  • nx t3 reidentify and scripts/migrate_art_papers.py (nexus-z0o2p.25, RDR-223 Phase 3). nx t3 reidentify was the RDR-108 verb that re-keyed legacy chunk ids; it re-upserted through col.upsert, which the service collection stub does not have, so --no-dry-run raised AttributeError on every real install, and a pgvector chunk id is its chash by schema. The ART migration script was a one-off over the retired SQLite T2 and catalog files. nx t3 backfill-manifest and the other nx t3 verbs are unchanged. The verify-fill code in db/reconcile and db/embed_migrate went too; neither had a caller.

  • nx collection backfill-hash and nx catalog backfill's pass 4 (RDR-223 Phase 3 follow-up). The hidden verb re-upserted each chunk through col.upsert to add chunk_text_hash metadata; the service collection handle has no upsert, so the verb raised AttributeError on every real install (the defect class that removed nx t3 reidentify), and a pgvector chunk id is its chash by schema so there is nothing to backfill. nx catalog backfill loses its chunk_text_hash pass, which was already a no-op on the service-backed T3.

  • The engine's /v1/staging/* routes and the staging schema (nexus-z0o2p.27, RDR-223 Phase 3). The RDR-180 land-then-transform landing zone (POST /v1/staging/load/{store}, /embed_fill, /promote, /finalize, /clear, GET /v1/staging/counts) is gone, along with its promote code and the sweep's guard over staging.document_chunks; changeset staging-6-drop-landing-schema drops the seven staging tables, discarding any rows landed there, and drops the schema unless something stands in the way: an object in the schema the changeset did not name, a view or foreign key outside it that depends on a named table, or a migrating role that owns neither. The boot goes on in each case and that object stays. The migration connection does not surface the changeset's NOTICEs, so the catalog is the only evidence of what was left (pg_namespace and pg_tables for schema staging return no rows after a complete drop). No 7.x client calls the routes. The one client that did was the Chroma-era migration in conexus 6.18.1 and earlier. Aimed at a managed cloud engine, that path is unsupported: after this change it has no routes to land through, and what it did against a current engine before is unmeasured (nexus-6g218). Against a local install it uses its own pinned engine and is unaffected. Irreversible by a binary rollback (nexus-z0o2p.44): once the walk has dropped the schema, an engine at v0.1.142 or older aborts at boot, because its always-run grants changeset grants-nexus-diag-3 grants on schema staging and does not check that it exists, wherever the role nexus_diag exists. To go back, create the schema first (CREATE SCHEMA staging, as the admin role), then start the older engine; that was rehearsed on a restore of production (the older engine boots, runs no new changeset, and re-grants nexus_diag), and restoring from a point in time is the fallback. A local client downgrade to 7.67.0 meets the same refusal only where a nexus_diag role exists, which has not been checked for the local bundle.

Fixed

  • Backfill and the reindex delete-guard read stored chunks, not live ones (nexus-wbfpw.35, RDR-192 Phase 2 gate). nx catalog backfill --from-t3, the RDR and paper backfills and the nx collection reindex pre-delete scan read through the live(c) filter, so a collection whose chunks are all unowned looked empty to them: the backfills found nothing to register, and reindex skipped its "would destroy every chunk" refusal and deleted the collection. They now pass include_non_live, and the backfills filter collections on stored_count. Reading stored chunks also brings back the chunks of documents deleted but not yet purged, so the verbs skip a path the catalog holds only as a deleted document (they used to register or re-index it as a new live one) and say how many they skipped. The match is exact: a source path equals the tombstone's file_path, made relative to its owner's repo_root where that applies, never a suffix, and each tombstone's repo_root makes only its own relative form. reindex keeps a source that any live catalog document in the same collection names, under any owner and in either path form (one file can be catalogued under two owners, and a deleted document's owner is only one of them), so it cannot purge chunks it then fails to rebuild; the backfills skip a deleted document's path unless a live document holds it under that same owner (delete a file's document, then index the file again). The guard fails closed: reindex refuses when the catalog cannot be read or when every source is a deleted document, and reindex, the three backfills and nx catalog orphan-backfill (dt-link, synthetic, apply-csv, which now leave a deleted document's title groups alone) refuse on an engine whose GET /v1/catalog/trash entries carry no file_path, instead of running without the guard. Needs an engine newer than engine-service-v0.1.142, which adds file_path to those entries and orders them by deleted_at then tumbler so a batch delete cannot hide a tombstone between pages; against v0.1.142 the verbs above refuse once the trash holds anything. The verb that writes, re-embed, stays on live rows. For a chunk no document ever owned, a client re-write would refresh the reap grace anchor and extend its life. For a chunk owned only by a deleted document that argument does not hold (under the planned nexus.chunk_orphaned_at design a refresh extends nothing); the reasons there are the Voyage cost of a re-embed and that patching a chunk of a deleted document is pointless. re-embed now says how many stored chunks stayed on their old vectors.

  • MCP store_list says "N stored" (nexus-wbfpw.35). Its header and footer read "showing 1-20 of N" with N the stored chunk count while the rows are live ones; they now say N stored as nx store list does, and the next-page hint appears only after a full page (a page of the engine's 300-row cap counts as full, so limit=500 no longer reads as the end).

  • nx collection reindex now refuses, and asks for --force, on any collection holding hidden chunks with no source (nexus-wbfpw.35). The scan reads stored chunks, so chunks with no live catalog document (unowned, or owned only by a deleted document) count as sourceless alongside manual store_put entries, where before they were invisible to it. The refusal says so and points at nx collection info, which reports live and stored counts; a collection of only such chunks is refused outright.

  • The single-file index failure no longer says the chunks are searchable (nexus-wbfpw.34, nexus-wbfpw.35). nx index pdf and nx index md on a file with no catalog document wrote nothing since RDR-223 but said "indexed ... searchable but orphaned"; the message now says nothing was written, and for chunks written without a manifest row it says they are hidden until repaired.

  • Engine: the hybrid search dispatch counts the rows search returns (nexus-wbfpw.35, changeset vectors-023). text_gate_probe_<dim> kept the old tenant-wide tombstone anti-join after live(c) moved every other read path, so a gate that was selective among live rows could read as dense and take the HNSW-first plan. It now uses live(c).

  • nx index repo writes an oversize file's chunks together with their owner rows, and fails the run when a file is deferred (nexus-z0o2p.14, RDR-223 P2.4). The fallbacks for a code file over 300 chunks, a prose or RDR file over the CCE batch and a PDF over one batch wrote chunks with upsert-chunks and the owner rows in a later manifest write, so a client that died between the two left chunks nobody owned. They now send chunks and owner rows in the same request, with the metadata merge the old upsert had (bib_* set by nx enrich bib survives a forced re-index; a key the indexer owns and dropped is cleared), and so does every normal nx index repo chunk flush. A file deferred on a transient write error (429, 502, 503, 504, the embed timeout, a connection failure) is now named in a warning and makes the command exit non-zero; it used to exit 0. Needs the engine that carries the metadata_merge route fields; an older engine fails each flush loudly.

  • nx upgrade backfills legacy notes that RDR-192 hid from search (nexus-wbfpw.41). A note stored before nexus-b6enc has a catalog document but no manifest row, and the engine's live(c) predicate has hidden such a chunk since RDR-192 Phase 2; only the operator tenant had been censused and backfilled, so every other install kept invisible notes that the coming reaper would delete. The upgrade ladder has a new rung, rdr192-manifest-backfill: it censuses every non-quarantine collection, backfills each one holding a legacy-unmanifested chunk, and records completion in nexus.ladder_completions only after a fresh census reads zero, with a census summary in the record. It deletes nothing and is safe to re-run. A census answer without its bucket totals, an empty listing over a catalog that holds manifest rows, an unreachable engine and an old engine all defer rather than read as clean. A note the backfill cannot heal defers the rung: nx upgrade finishes its other steps and exits 0, names the collections, the skipped-document counts per class and the remedy (no verb heals it, and its text is hidden from nx store get, so re-put your own copy under the same title; nx t3 census-manifest-less now prints each legacy row's owner title), and is not even re-examined for 24 hours at the same package version (NX_RDR192_BACKFILL_RETRY=1 retries now). A backfill that errors is never remembered and is retried at the next session start; an unwritable config directory or a census outage during verification defers instead of failing; a clean tenant pays for one census. detect() never takes a census, so nx doctor and nx upgrade --dry-run stay cheap. nx t3 backfill-manifest --only-gapped no longer manifests a document with more matched chunks than its registered chunk count or two chunks at one position (a note with a repeated piece, fewer matched than registered, is still healed), and re-reads the manifest just before writing. The engine side is Rdr192BackfillGate, which the reaper (nexus-2x9xa) calls and which refuses to reap on a tenant without the record. See docs/migration-runbook.md.

  • Engines v0.1.133 to v0.1.142 overstated the census scope_chunk_total on a collection where one chash is named by several manifest rows; fixed in v0.1.143 (nexus-wbfpw.60, .62). Identical text shared by N documents was counted N times (code__1-15 read 8,913 for 5,641 chunks). The count is the denominator of nx t3 gc's 0.25 floor and of the chunks= detail the rdr192-manifest-backfill rung prints, so against those engines the floor ran up to about 58% looser than intended, and the reaper logged false reaper_census_scope_much_larger warnings. Chunks nx t3 gc moves go to quarantine, so a move made under the looser floor can be restored. The new engine counts chunks; buckets, items and the other totals are unchanged and no changeset moves.

  • Engine: Liquibase's history table is pinned to public, and a split history refuses the boot (nexus-q81g7). Liquibase puts DATABASECHANGELOG in current_schema(), and the default search path makes that the schema named after the migrating role once one exists. A role named nexus, t1 or staging booted once, created its own schema mid-walk, read an empty history there on the second boot and failed on CREATE TABLE nexus.memory. The migration session now pins search_path to public and logs its role, schema and path (schema_migration_session). If databasechangelog exists in another schema with more rows than public's, the boot stops before any changeset runs and names ALTER TABLE <schema>.databasechangelog SET SCHEMA public. nx doctor and the daemon read public.databasechangelog explicitly. A role with no like-named schema, such as production's nexus_admin, already resolved to public and is unchanged.

  • nx index repo does not bisect a 429 or a deadline-aborted flush (nexus-eoido). A failed multi-file flush was split and each half retried with a full budget, including on a 429 or an engine request-deadline abort, so one refused request became 1, 2, 4 and more requests that each re-embedded and re-billed. A throttle now fails the batch as one: its files are deferred to the next run's staleness check, the Retry-After goes to the shared rate brake, and the throttled files are reported apart from failed ones. A client-side call deadline defers without pacing the brake. Every other failure still bisects, a deadline abort once.

  • The local MiniLM model is provisioned once under a lock (nexus-ccre5). Several cold first uses made several downloads, and one could delete a peer's just-published files. Provisioning now takes an exclusive lock and rechecks inside it, so N cold callers make one download; transient HTTP errors (429, 5xx, transport) retry with backoff, a checksum mismatch still fails loud, and temporary archives older than a day are swept.