v7.16.2
A patch of production-outage and diagnostic fixes found by running the system
against itself. Engine identity unchanged at engine-service-v0.1.85.
Four fixes and one perf change. Three share a shape that is worth naming: a
short-lived git worktree under /private/tmp reaching into long-lived
production state, in a repo whose own guidance routes sessions into exactly
such directories.
Fixed
- The aspect-worker died permanently and silently when its launch directory
was deleted. It inherits the cwd of whatever spawned it -- and RDR-173
requires that parent to be an interactive session holdingclaude -p
credentials. When that directory goes away,os.getcwd()raises inside every
os.path.abspath(), soaspect_readers.uri_forthrows and every batch
fails. Observed on a production install: 16 consecutive batches, zero
successes, ~40 minutes, all at WARNING with nothing alerting. The daemon now
chdirs toconfig_dirbefore anything can resolve a path. The trigger --
removing a finished worktree -- was correct; depending on it was the defect.
(nexus-yg70j) - An edge/WAF 403 told the operator to check their token. Every
nx index reporun against production died on exactly one 403 from
POST /v1/vectors/upsert-chunksbetween 08-19 and 08-23: 151 occurrences,
one per run, in 151 of 397 runs, and the RDR corpus went un-reindexed for
four days. The cause was an AWS WAF rule matching a JVM root package prefix
in the request body -- RDR-195 quotes a JVM stack trace, so the document
documenting the bug could not be indexed because the bug report reads as an
exploit payload. The client reframed that as "checkNX_SERVICE_URLis
reachable andNX_SERVICE_TOKENis valid" while 477 upserts succeeded in the
same run on the same token. An edge-generated response is now named as one.
The discriminator is a POSITIVE test onserver: awselb/2.0, measured
against the live edge -- the two obvious alternatives were both inverted (the
application emits NOServerheader and answers in plain text, not JSON).
(nexus-1jtob) - An index failure did not say which file failed.
run_file_loopre-raised
failures[0][1], so the traceback surfaced at the phase boundary with no
indication of the rejected document -- the SUPPRESSED sibling failures were
logged withfile=, the one that ended the run was not. Both loop paths now
name it. (nexus-1jtob) - 403 was in the ETL retry set on an unmeasured premise. It sat there as a
"transient edge 403"; a sweep of 3,675,603 edge log records over four days
found zero 403s on that path. The two populations were backwards by path: on
the vector path 403s are frequent and deterministic (and correctly excluded),
while the ETL path retried unpassable WAF refusals 5-8x with escalating
brake against our own edge. ALB 5xx is genuinely transient and carries the
sameserverheader, so429/502/503/504still retry. (nexus-1jtob) - A dispatchable workflow gated the wrong tree.
local-service-gate-nightly
andguided-upgrade-mvvhard-pinnedref: developin their checkout, which
overridesactions/checkout's triggering-ref default unconditionally -- so a
workflow_dispatchagainst any branch silently gateddevelopand reported
a confident verdict about a tree nobody asked about. This is why the
local-service gate's fix appeared to fail: the run never executed it. A
repo-wide lint now flags any dispatchable workflow whose checkout ref is a
literal. (nexus-pc15o) - The nightly local-service gate had been red for eight nights with zero
failing tests. Three RDR-196 live-dispatch families landed marked
integrationonly while six identical siblings carriedlived_in, so 30
tests that cannot authenticate aclaudeCLI on a runner skipped there: 21
legitimate skips + 30 = 51 against a budget of 25. The budget was right and
was not raised. Post-fix skips are exactly 21. (nexus-pc15o) - Committing inside a
git worktreefired a full production index of the
duplicate tree. ~1876 files, projected 64-131 minutes, detached so the cost
never landed on the committing session's clock. Nine such runs in one day
from three sessions. The hook now skips linked worktrees, discriminating on
--git-dirvs--git-common-dir-- not a path prefix, since four of the
nine were under.claude/worktrees/inside the repo. (nexus-ws67k)
Changed
config.ymlis parsed once per mtime instead of on every access.
(nexus-m20mf)using-nx-skillsreplaced itsMUSTimperative with situational triggers.
The instruction was already maximal, and measured 1-in-6 compliance across
six sandboxed runs -- writing it harder is the one remedy known to have
failed. (nexus-bc292)
Known gaps
aspect_readers.uri_forstill resolves relative paths against the process
cwd. The daemon no longer dies, but the same storedsource_pathyields
different URIs depending on where the process stands -- thenexus-3e4s
CWD-anchoring class.catalog/types.pyalready implements the correct shape
(anchor on a passedrepo_root); threading that through is tracked on
nexus-yg70j.- RDR-198 (collapse the duplicated client transport) was DECLINED after
measurement: the "90% of steady-state cost" that motivated it was measured
against a mocked-I/O harness and is 0.07% in production.