Skip to content

v7.69.0

Choose a tag to compare

@github-actions github-actions released this 03 Oct 15:18
· 2466 commits to main since this release
24938cc

Pairs with engine-service-v0.1.145, which also carries v0.1.144. Neither engine adds a changeset. The engine now deletes an origin collection's quarantine rows with it and retags them on rename (nexus-wbfpw.71), lets the reaper pass a tenant that holds nothing and reports tenants_empty on its last pass (nexus-wbfpw.73), retries a manifest write that loses a deadlock (nexus-wbfpw.66) and logs reaper_pass_failed when a pass cannot list its tenants (nexus-wbfpw.67). All three wire changes are additive, so the engine can deploy before this client. One direction is not neutral: an older client's nx collection reindex against the new engine deletes the collection's quarantine rows, audited, as nx collection delete does (see the first collection-delete entry below). Local installs get the new engine with this release.

Fixed

  • Engine: the reaper no longer refuses a tenant that holds nothing (nexus-wbfpw.73). The backfill gate refused every tenant with no rdr192-manifest-backfill row in nexus.ladder_completions, so a tenant with no chunks, no collections and no manifest rows (on production conexus-edge, default and smoke-2nx) was refused with BACKFILL_INCOMPLETE every hourly pass, and only an nx upgrade run with that tenant's own credential could record "nothing here". The gate now passes a tenant that holds no chunk row in any collection (a quarantine-* collection counts as holding chunks). That reads at least as strictly as the client rung's empty-listing branch, which skips quarantine in its census and reads the catalog's live document chunk count; the engine counts quarantine chunks, and with the manifest's foreign key to nexus.chunks a tenant with no chunk has no manifest row either. It writes no completion, so the client rung stays the one recorder. A tenant that gains content and still has no record is refused again, and a tenant holding only quarantine chunks is not empty. An empty tenant counts in tenants_visited and as neither refused nor errored (refused_total and reaper.last_pass.tenants_refused stop counting it), and logs event=rdr192_backfill_gate_passed_empty_tenant at DEBUG (one line per empty tenant per hourly pass). reaper.last_pass gains an additive tenants_empty count, the visited tenants that held no chunk, and nx doctor's "alive but doing nothing" row now judges against tenants_visited - tenants_empty: the default tenant is always visited and empty in cloud, so without it a pass where every real tenant is refused would read healthy. A pass that visits only empty tenants (a fresh local install) reads healthy by design; an engine that sends no tenants_empty is read as 0. No changeset. docs/operations/engine-reaper.md and RDR-192 carry the definition.

  • nx t3 gc --dry-run no longer says a real run "will REFUSE" over a document that is not index_state='complete' when nothing is reapable (nexus-wbfpw.69). That check guards the move only; with nothing to move a real run goes straight to the quarantine expiry, so the dry run exited 0 while printing a refusal, which read as a broken exit code (production rdr__1-20, 2026-10-03). The line now says the state blocks a move and not the expiry; when there are chunks to move, the refusal is still listed and the dry run still exits 1.

  • Engine: deleting or renaming an origin collection no longer strands its quarantine rows, the ghost sweep keeps such an origin live, and the paths that moved or deleted a quarantine-* collection's chunks with no audit row are closed (nexus-wbfpw.71 and the producer half of nexus-wbfpw.68; Sam, 2026-10-03, option (b)). nx collection delete <origin> now also deletes the origin's rows from every registered quarantine collection in the same transaction (rows tagged origin_collection = <origin>, plus untagged rows in quarantine-<origin>, plus untagged rows in the sibling derived from the origin's catalog row when the origin is the only live collection sharing that row's owner, model, version and content type; another origin's rows in a shared sibling stay), writes one collection_delete_quarantine gc_audit row per sibling, unregisters a sibling it empties, and reports the count as quarantine_chunks (the CLI prints it, with the chunk count). nx collection delete quarantine-<name> is allowed, prints the rows it removed and writes one quarantine_collection_delete row with the chashes (truncated at 5,000, count exact); it is the audited route for quarantined rows whose origin is gone. The 294 such rows in production (nexus-wbfpw.65) were already removed by hand on 2026-10-03 (gc_audit rows 10590 and 10591, operation manual_expire_dead_origin_quarantine), so this verb is the route for any later case. nx collection reindex re-registers the same name, so it sends the new keep_quarantine field and the rows stay restorable. Only the new client sends that field: an OLD client's nx collection reindex against the NEW engine deletes the collection's quarantine rows (audited, one collection_delete_quarantine row per sibling), exactly as nx collection delete does. A canonical rename of an origin (/v1/catalog/collections/rename, and /v1/chash/rename_collection when the source is no longer a live registered collection) retags its quarantine rows to the new name with one quarantine_retag row per sibling; sibling names do not change, because the engine finds a sibling by the tag. The ghost sweep (nx catalog sweep-ghosts, the boot sweep) holds an origin whose chunks all sit in quarantine instead of deleting its registry row, counted under quarantine_held (the held names are logged at INFO, event=ghost_sweep_held_quarantined_origin). A held origin is a registered origin, so the engine's rows in its quarantine siblings go on the ordinary 14 day engine expiry; before this change a ghost-swept origin's rows were skipped for ever as origin_not_registered. Rows the client moved (nx index repo, nx t3 gc, untagged) wait for nx t3 gc, which the floor still blocks for such a collection (nexus-wbfpw.74, nexus-wbfpw.75), so the origin can stay registered and empty; nx collection delete <origin> clears the hold and takes the rows, audited. POST /v1/vectors/store-delete, collection rename and POST /v1/catalog/collections/rehome now answer 400 for a quarantine-* collection (source or target) and name nx t3 quarantine restore and nx t3 gc. Not closed: the RDR-162 cross-model COPY keeps the source's quarantine rows with the source (it stays live; they carry its model's vectors), store-put and update-metadata on a quarantine-* name are not guarded, and there is still no engine expiry for rows whose origin is gone. Wire (additive): /v1/catalog/collections/delete accepts keep_quarantine (absent or JSON false is the default, JSON true keeps the rows, any other present value answers 400 naming the field) and returns quarantine_chunks for a quarantine-* name too; the new 400s are on routes that already existed. This client works against an engine without the change (the count key reads 0). No changeset.

  • nx t3 gc expires from every quarantine collection that holds the collection's chunks, not only the one its catalog row names today (nexus-wbfpw.58, RDR-192 Phase 3 critique S3). The client names the sibling it moves chunks into from the catalog row, and catalog-044-3 rewrote owner_id on repo collections after chunks had been moved, so a chunk moved under the earlier name sat in a sibling the client no longer looked in. The engine's own expiry skips chunks the client moved (they carry no quarantined_by tag), so nothing expired them: storage only, the chunks stayed hidden and nx t3 quarantine restore still reached them. nx t3 gc now asks the engine which siblings hold the collection's chunks (the resolution the restore verb uses, by the chunks' origin_collection tag) and expires from each, plus the row-derived name and quarantine-<name>; it prints one Client expiry of <sibling> ...: N expired, M refused line for each sibling it tried, with that sibling's own counts. The floor is judged per sibling: a stranded sibling with 100 or more eligible chunks that are more than NX_GC_FLOOR_FRACTION of its rows is refused and its line names NX_GC_FORCE=1, so the way past it is NX_GC_FORCE=1 nx t3 gc. A failure on one sibling exits 1 and names what the earlier siblings already expired. A T3 handle with no expire route now prints Client expiry of <sibling> did NOT run: ... on stderr instead of returning silently. nx index repo is different on purpose: it runs this prune on every index, so it uses only the two derived names (the row-derived name and quarantine-<name>), sends no engine query, and treats the second name as best-effort (a failure there is logged and skipped). The re-reference restore is indexer-only and uses the same two names. A chunk stranded under a third name therefore waits for nx t3 gc; nexus-wbfpw.64 moves the resolution into the engine's expire and restore-rereferenced routes and retires the client's query.

  • A manifest write that loses a deadlock to a chunk write is retried instead of failing the request, and two engine counters read once per request (nexus-wbfpw.66). Every transaction that writes catalog_document_chunks rows (write_many, append, manifest write and purge, the import routes, collection delete, rename and re-home, purge_trash) now runs under the engine's deadlock retry: a 40P01 re-runs the whole transaction, up to 4 attempts, and surfaces as before only when all of them deadlock. Two counters moved with it. raced_embeds_total on GET /v1/status now counts the raced embeds of committed writes only, from both write paths; the combined-write path used to count inside its transaction, so a reading that drops across this engine cut is a change of definition and not necessarily of rate. ownerless_writes_would_refuse_total in log-only mode no longer counts a request once per retry attempt when the in-transaction ownership recheck found it unowned; it counts it once. Engine only, no wire change.

  • Engine: a reaper pass that cannot list its tenants logs event=reaper_pass_failed at ERROR, not a zero-filled event=reaper_run (nexus-wbfpw.67, conexus-lv6t). On engine-service-v0.1.143 that path logged WARN event=reaper_run tenants=0 ... errors=1 wall_clock_cut=false error=<msg>, so an alert on reaper_pass_failed missed it and a heartbeat on reaper_run read the failed pass as alive. The new line is event=reaper_pass_failed error_class=<class> error=<msg> failed_passes_total=<n> stage=tenant_list (the same fields, in the same order, as the existing line for a Throwable that escapes a pass, which carries no stage=), and no reaper_run line is written for that pass. failed_passes_total and last_completed_pass_at in /v1/status are unchanged; they already counted this path.

  • Engine: a reaper refusal or failure on a later origin no longer reports zero for chunks earlier origins of the same sibling already expired (nexus-wbfpw.53). Each origin expires in its own transaction, but the lock-timeout, statement-timeout and generic-failure branches reset the counts, so reaper_pass and reaper_run under-reported what had committed. The counts now survive the refusal, and reaper_expire_skipped, reaper_expire_refused and reaper_expire_failed gain expired_before_refusal= and expiry_protected_before_refusal= (additive). docs/operations/engine-reaper.md no longer says a timed-out expiry deleted nothing.

  • The release battery and the e2e host scripts no longer call a bare python3 (nexus-u67ow). On a host whose first python3 is 3.9, as on hellmini, the battery's cut mode aborted with a SyntaxError reading the artifacts manifest. tests/e2e/lib/python.sh now resolves one Python 3.10 or newer (3.11 where tomllib is read), exports it as E2E_PYTHON, and refuses with exit 2 naming every candidate when none qualifies. NX_E2E_PYTHON picks one explicitly. A lint in tests/scripts/test_e2e_python_resolver.py fails on a new bare python3 call.