Skip to content

M06

M06 - self-healing (human-triggered) + shadow eval
@andaro74 andaro74 tagged this 13 Aug 05:19 
Built at half scope. Running Claude Code headless in CI needs an identity
holding bedrock:InvokeModel, and no such identity exists that does not weaken
ARCHITECTURE invariant 1 -- declined, and the design recorded as next
(ADR-035). The classifier shipped, so Act 3 runs from a laptop.

pave shadow-eval shipped in full, varying the candidate through the gateway's
routing table rather than by letting a caller name a model (ADR-036).

Three failures worth more than the features:

- The first shadow run reported "safe to adopt" having compared the incumbent
  to itself. The header was built from a stack output and a boolean. Fixed by
  observing model_id from the response bodies.
- The second run's cost delta went down because failing cases skip the judge.
  Quality down and cost down read as a trade-off; it was one fact counted
  twice. Fixed by counting judge verdicts per arm.
- Act 3 exposed that the contract suite does not type-check union-typed
  parameters at all. The AI declined to repair the helper under a schema_drift
  verdict, which is why the finding survived to be written down (ADR-037).
Assets 2
Loading