Repository navigation
v7.69.0
Pairs with engine-service-v0.1.145, which also carries v0.1.144. Neither engine adds a changeset. The engine now deletes an origin collection's quarantine rows with it and retags them on rename (nexus-wbfpw.71), lets the reaper pass a tenant that holds nothing and reports tenants_empty on its last pass (nexus-wbfpw.73), retries a manifest write that loses a deadlock (nexus-wbfpw.66) and logs reaper_pass_failed when a pass cannot list its tenants (nexus-wbfpw.67). All three wire changes are additive, so the engine can deploy before this client. One direction is not neutral: an older client's nx collection reindex against the new engine deletes the collection's quarantine rows, audited, as nx collection delete does (see the first collection-delete entry below). Local installs get the new engine with this release.
Fixed
-
Engine: the reaper no longer refuses a tenant that holds nothing (nexus-wbfpw.73). The backfill gate refused every tenant with no
rdr192-manifest-backfillrow innexus.ladder_completions, so a tenant with no chunks, no collections and no manifest rows (on productionconexus-edge,defaultandsmoke-2nx) was refused withBACKFILL_INCOMPLETEevery hourly pass, and only annx upgraderun with that tenant's own credential could record "nothing here". The gate now passes a tenant that holds no chunk row in any collection (aquarantine-*collection counts as holding chunks). That reads at least as strictly as the client rung's empty-listing branch, which skips quarantine in its census and reads the catalog's live document chunk count; the engine counts quarantine chunks, and with the manifest's foreign key tonexus.chunksa tenant with no chunk has no manifest row either. It writes no completion, so the client rung stays the one recorder. A tenant that gains content and still has no record is refused again, and a tenant holding only quarantine chunks is not empty. An empty tenant counts intenants_visitedand as neither refused nor errored (refused_totalandreaper.last_pass.tenants_refusedstop counting it), and logsevent=rdr192_backfill_gate_passed_empty_tenantat DEBUG (one line per empty tenant per hourly pass).reaper.last_passgains an additivetenants_emptycount, the visited tenants that held no chunk, andnx doctor's "alive but doing nothing" row now judges againsttenants_visited - tenants_empty: the default tenant is always visited and empty in cloud, so without it a pass where every real tenant is refused would read healthy. A pass that visits only empty tenants (a fresh local install) reads healthy by design; an engine that sends notenants_emptyis read as 0. No changeset.docs/operations/engine-reaper.mdand RDR-192 carry the definition. -
nx t3 gc --dry-runno longer says a real run "will REFUSE" over a document that is notindex_state='complete'when nothing is reapable (nexus-wbfpw.69). That check guards the move only; with nothing to move a real run goes straight to the quarantine expiry, so the dry run exited 0 while printing a refusal, which read as a broken exit code (productionrdr__1-20, 2026-10-03). The line now says the state blocks a move and not the expiry; when there are chunks to move, the refusal is still listed and the dry run still exits 1. -
Engine: deleting or renaming an origin collection no longer strands its quarantine rows, the ghost sweep keeps such an origin live, and the paths that moved or deleted a
quarantine-*collection's chunks with no audit row are closed (nexus-wbfpw.71 and the producer half of nexus-wbfpw.68; Sam, 2026-10-03, option (b)).nx collection delete <origin>now also deletes the origin's rows from every registered quarantine collection in the same transaction (rows taggedorigin_collection = <origin>, plus untagged rows inquarantine-<origin>, plus untagged rows in the sibling derived from the origin's catalog row when the origin is the only live collection sharing that row's owner, model, version and content type; another origin's rows in a shared sibling stay), writes onecollection_delete_quarantinegc_auditrow per sibling, unregisters a sibling it empties, and reports the count asquarantine_chunks(the CLI prints it, with the chunk count).nx collection delete quarantine-<name>is allowed, prints the rows it removed and writes onequarantine_collection_deleterow with the chashes (truncated at 5,000, count exact); it is the audited route for quarantined rows whose origin is gone. The 294 such rows in production (nexus-wbfpw.65) were already removed by hand on 2026-10-03 (gc_auditrows 10590 and 10591, operationmanual_expire_dead_origin_quarantine), so this verb is the route for any later case.nx collection reindexre-registers the same name, so it sends the newkeep_quarantinefield and the rows stay restorable. Only the new client sends that field: an OLD client'snx collection reindexagainst the NEW engine deletes the collection's quarantine rows (audited, onecollection_delete_quarantinerow per sibling), exactly asnx collection deletedoes. A canonical rename of an origin (/v1/catalog/collections/rename, and/v1/chash/rename_collectionwhen the source is no longer a live registered collection) retags its quarantine rows to the new name with onequarantine_retagrow per sibling; sibling names do not change, because the engine finds a sibling by the tag. The ghost sweep (nx catalog sweep-ghosts, the boot sweep) holds an origin whose chunks all sit in quarantine instead of deleting its registry row, counted underquarantine_held(the held names are logged at INFO,event=ghost_sweep_held_quarantined_origin). A held origin is a registered origin, so the engine's rows in its quarantine siblings go on the ordinary 14 day engine expiry; before this change a ghost-swept origin's rows were skipped for ever asorigin_not_registered. Rows the client moved (nx index repo,nx t3 gc, untagged) wait fornx t3 gc, which the floor still blocks for such a collection (nexus-wbfpw.74, nexus-wbfpw.75), so the origin can stay registered and empty;nx collection delete <origin>clears the hold and takes the rows, audited.POST /v1/vectors/store-delete, collection rename andPOST /v1/catalog/collections/rehomenow answer 400 for aquarantine-*collection (source or target) and namenx t3 quarantine restoreandnx t3 gc. Not closed: the RDR-162 cross-model COPY keeps the source's quarantine rows with the source (it stays live; they carry its model's vectors),store-putandupdate-metadataon aquarantine-*name are not guarded, and there is still no engine expiry for rows whose origin is gone. Wire (additive):/v1/catalog/collections/deleteacceptskeep_quarantine(absent or JSONfalseis the default, JSONtruekeeps the rows, any other present value answers 400 naming the field) and returnsquarantine_chunksfor aquarantine-*name too; the new 400s are on routes that already existed. This client works against an engine without the change (the count key reads 0). No changeset. -
nx t3 gcexpires from every quarantine collection that holds the collection's chunks, not only the one its catalog row names today (nexus-wbfpw.58, RDR-192 Phase 3 critique S3). The client names the sibling it moves chunks into from the catalog row, and catalog-044-3 rewroteowner_idon repo collections after chunks had been moved, so a chunk moved under the earlier name sat in a sibling the client no longer looked in. The engine's own expiry skips chunks the client moved (they carry noquarantined_bytag), so nothing expired them: storage only, the chunks stayed hidden andnx t3 quarantine restorestill reached them.nx t3 gcnow asks the engine which siblings hold the collection's chunks (the resolution the restore verb uses, by the chunks'origin_collectiontag) and expires from each, plus the row-derived name andquarantine-<name>; it prints oneClient expiry of <sibling> ...: N expired, M refusedline for each sibling it tried, with that sibling's own counts. The floor is judged per sibling: a stranded sibling with 100 or more eligible chunks that are more thanNX_GC_FLOOR_FRACTIONof its rows is refused and its line namesNX_GC_FORCE=1, so the way past it isNX_GC_FORCE=1 nx t3 gc. A failure on one sibling exits 1 and names what the earlier siblings already expired. A T3 handle with no expire route now printsClient expiry of <sibling> did NOT run: ...on stderr instead of returning silently.nx index repois different on purpose: it runs this prune on every index, so it uses only the two derived names (the row-derived name andquarantine-<name>), sends no engine query, and treats the second name as best-effort (a failure there is logged and skipped). The re-reference restore is indexer-only and uses the same two names. A chunk stranded under a third name therefore waits fornx t3 gc; nexus-wbfpw.64 moves the resolution into the engine's expire and restore-rereferenced routes and retires the client's query. -
A manifest write that loses a deadlock to a chunk write is retried instead of failing the request, and two engine counters read once per request (nexus-wbfpw.66). Every transaction that writes
catalog_document_chunksrows (write_many,append, manifest write and purge, the import routes, collection delete, rename and re-home,purge_trash) now runs under the engine's deadlock retry: a 40P01 re-runs the whole transaction, up to 4 attempts, and surfaces as before only when all of them deadlock. Two counters moved with it.raced_embeds_totalonGET /v1/statusnow counts the raced embeds of committed writes only, from both write paths; the combined-write path used to count inside its transaction, so a reading that drops across this engine cut is a change of definition and not necessarily of rate.ownerless_writes_would_refuse_totalin log-only mode no longer counts a request once per retry attempt when the in-transaction ownership recheck found it unowned; it counts it once. Engine only, no wire change. -
Engine: a reaper pass that cannot list its tenants logs
event=reaper_pass_failedat ERROR, not a zero-filledevent=reaper_run(nexus-wbfpw.67, conexus-lv6t). On engine-service-v0.1.143 that path logged WARNevent=reaper_run tenants=0 ... errors=1 wall_clock_cut=false error=<msg>, so an alert onreaper_pass_failedmissed it and a heartbeat onreaper_runread the failed pass as alive. The new line isevent=reaper_pass_failed error_class=<class> error=<msg> failed_passes_total=<n> stage=tenant_list(the same fields, in the same order, as the existing line for a Throwable that escapes a pass, which carries nostage=), and noreaper_runline is written for that pass.failed_passes_totalandlast_completed_pass_atin/v1/statusare unchanged; they already counted this path. -
Engine: a reaper refusal or failure on a later origin no longer reports zero for chunks earlier origins of the same sibling already expired (nexus-wbfpw.53). Each origin expires in its own transaction, but the lock-timeout, statement-timeout and generic-failure branches reset the counts, so
reaper_passandreaper_rununder-reported what had committed. The counts now survive the refusal, andreaper_expire_skipped,reaper_expire_refusedandreaper_expire_failedgainexpired_before_refusal=andexpiry_protected_before_refusal=(additive).docs/operations/engine-reaper.mdno longer says a timed-out expiry deleted nothing. -
The release battery and the e2e host scripts no longer call a bare
python3(nexus-u67ow). On a host whose firstpython3is 3.9, as on hellmini, the battery's cut mode aborted with a SyntaxError reading the artifacts manifest.tests/e2e/lib/python.shnow resolves one Python 3.10 or newer (3.11 wheretomllibis read), exports it asE2E_PYTHON, and refuses with exit 2 naming every candidate when none qualifies.NX_E2E_PYTHONpicks one explicitly. A lint intests/scripts/test_e2e_python_resolver.pyfails on a new barepython3call.