feat(agentplane): SP-TRACE-CFR recovery — WO-0.A emitter + WO-1 P0 ingest + P1 CFG#312
Merged
Conversation
… specs - SP_TRACE_CFR_001_SPEC.md: trace control-flow recovery / narration fidelity verifier (two-engine v0.1) - SP_TRACE_CFR_001_axis_binding.md: TriTRPC 3-axis reconciliation (verdict/evidence/provenance) - SP_TRACE_CFR_001_WO0.md: unmet-prerequisite work order (emitter, ClaimIR, StepGate, ObligationIR) - orch_dsl_grammar.md: ground-truth fixture DSL for SP-EVAL-TRACE-CFR-001 - AGENTPLANE_COMPOSITION_PRIMITIVES_SPEC.md: 12 orchestration primitives + 7 IR schema deltas - AGENTPLANE_COMPOSITION_PRIMITIVES_STATUS.md: dependency audit + SP-TRACE-CFR reconciliation Specs only; no engine code. Records verified unmet deps: missing base IR schemas, Mellumwork/CBES not locatable, StopGate schema needs $id.
…aim) - evidence-ir + verifier-ir grounded in tools/stopgate_artifact.py (Evidence dataclass; finding domain OK/VIOLATION/REVIEW/null -> PASS/FAIL/REVIEW/INDETERMINATE) - obligation-ir kind system per AGENTPLANE_COMPOSITION_PRIMITIVES_SPEC §6 - claim-ir per SP-TRACE-CFR WO-0.B (alpha-compiler input) - all Draft-2020 valid; unblocks §13 v2 deltas, §8 conformal, §6/§11 ObligationIR bind - STATUS: correct the $id finding (canonical StopGate has a URL $id) + short->URL id map
- verifier-ir v0.2 (§13.2 conformal + shape constraints) and obligation-ir v0.2 (§13.1 graph + taint) layer via allOf on the additionalProperties:true bases; base+delta composition instance-tested - receipt-fold v0.1 (§13.4), attribution-ir v0.1 (§13.5), cinorm-ir v0.1 (§13.7), stepgate-artifact v0.1 (§13.6) authored standalone; all Draft-2020 valid - corrects two §13 defects (STATUS §1a/1b): §13.6 StepGate cannot allOf the base (additionalProperties:false + PASS/FAIL verdict pin) -> standalone w/ OK<->PASS, VIOLATION<->FAIL map; §13.4 fold needs an additive 'fold' property on the additionalProperties:false ReceiptIR base
- conformal_gate.py: split Conformal Risk Control (Angelopoulos 2024) on the abstention branch. calibrate(scores, correct, alpha) -> lambda_hat; classify()/gate() accept or override to INDETERMINATE - honest MARGINAL guarantee E[accept & wrong] <= alpha (not a conditional P(wrong|accept) claim) - §14.5 coverage test passes: E[L] ~ 0.0987 <= alpha=0.10 over 320k MC draws; abstention engaged every trial; lambda_hat monotone in alpha; finite-sample floor 1/(n+1) handled; input validation - anytime-valid (streaming) + PASC composition declared as v0.2 extension points - calibration corpus default = SP-EVAL-TRACE-CFR fixtures; score = engine agreement/coverage margin (pluggable when the recovery engines land); parameterized by verifier-ir.schema.v0.2 conformal block - stdlib-only
…tice) - taint_lattice.py: finite lattice from a covering relation, validates lub/glb uniqueness (rejects non-lattices); integrity taint propagation = meet over inputs (one untrusted input taints the derived value); admit() checks per-argument provenance -> OK/VIOLATION, harness-owned deterministic T1 -> VerifierIR finding - closes lethal-trifecta / EchoLeak at the argument boundary (CaMeL): untrusted-derived arg to a trusted sink is denied; binds obligation-ir.schema.v0.2 resource 'taint' field - 6 tests incl the CaMeL send_email scenario, non-lattice rejection, missing-provenance=bottom; fixed an inverted bottom-element fallback bug found by the tests
…nformance - receipt_fold.py: commutative-monoid fold for sidechain receipt aggregation (order-independent => replay-deterministic). verdict_monoid = most-cautious-wins (FAIL>REVIEW>INDETERMINATE>PASS, identity PASS) = the StopGate partial-verdict aggregation (WO-0.C/§13.4); blame_group = signed integer credit (abelian group, §10) - permutation_invariant() = §14.1 (exhaustive n<=6, sampled n>6); check_axioms() = §14.2 - 5 tests incl a non-commutative op caught by BOTH the axiom check and the permutation test
- step_gate.py: per-node signed gate reusing stopgate_artifact.py ed25519 signing; StopGate Tier-0/Tier-1 budget split along the TIME axis. Tier-0 VIOLATION -> short_circuit=True (skip downstream compute); promise/progress advisory-only - verdict = pure harness function of the VerifierIR finding (v = g_H(e)); model excluded - verify_step_gate(): signature + §14.3 model-exclusion + verdict-domain re-check - 7 tests: emit/verify roundtrip, tier-0 short-circuit, verdict independent of promise/progress, model-authored verdict rejected, tampered artifact fails signature, AttributionIR emitter pinned
…ference emitter - schemas/trace-cfr-segment.schema.v0.1.json: the SP-TRACE-CFR D1 control-flow projection as a SEPARATE schema (does NOT mutate agentic-ops-trajectory-event); spawn/join/decision/tool_call/terminal kinds + site_id/branch_taken/guard_position/sidechain_id/parent_event_id + a sealed segment - tools/trace_cfr_emitter.py: reference PRODUCER (no live runner emits control-flow today); deterministic (counter ids + monotonic ts + canonical hashing) => byte-stable segments; sidechain() context manager makes I-SC1 SESE hold by construction; seal() hashes raw JSONL bytes (§4.0) - 5 tests: schema-valid, deterministic sealing, P0 ingest invariants by construction (monotone ts / unique ids / no dangling parents), SESE sidechain, decision guard_position (WHILE vs DO_WHILE) - this is the input WO-1 ingest recovers a CFG from; unblocks the SP-TRACE-CFR recovery-engine track
…r invariants) - trace_cfr_ingest.py: computes segment_hash over raw JSONL bytes BEFORE parsing, then enforces the §4.0 invariants (MALFORMED_JSON / MISSING_FIELD incl site_id / NON_MONOTONE_TS per agent / DUPLICATE_EVENT_ID / DANGLING_PARENT / SEAL_MISMATCH). Any violation => INDETERMINATE, NO repair (returns no events); pairs with trace_cfr_emitter (seal() output round-trips to SEALED) - 8 tests incl every rejection path + the emitter->ingest seal round-trip
…te fold, guard-position) - trace_cfr_cfg.py: build_cfg() constructs T=(V,E,entry,X) from a sealed segment. Nodes fold by (site_id, kind) [site fold + kind so spawn/join sharing a site don't collide]; edge labels from the source event (br_<taken>/spawn/join/seq); LATENT-BRANCH rule (decision with <2 distinct arms => latent_arms>=1, never SEQ); guard_position pre/post retained for WHILE vs DO_WHILE - backedge derivation + normalization explicitly deferred to P2 (loop cycles appear as coincident forward edges here) - 5 tests via the real P0->P1 pipeline: linear+SESE sidechain, single-exec decision is latent, WHILE both-arms/cycle, DO_WHILE guard_position=post, annotations excluded from nodes
…rreducible, T2 seq compression) - trace_cfr_normalize.py: N=[T1..T5] pipeline; normalization_version = blake2b of the ordered transform list. T5 = iterative dominators + DFS retreating-edge detection: natural-loop backedge iff v dominates u, else retreat_nondom (irreducible region = GOV-IRRED-001 material). T2 = maximal seq-chain compression (interior in1/out1), excluding backedges so loops aren't compressed across the header. T1/T3/T4 conservative v0.1 (documented) - fixed a T5 bug found by tests: forward loop-entry edges were mis-flagged irreducible by a reachability test; corrected to true DFS back-edges (v gray/ancestor when u->v traversed) - 5 tests: WHILE backedge+reducible, seq-chain compression, decision not swallowed, synthetic irreducible 2-entry cycle detected, normalization_version stable + order-sensitive
… rule enforced - recover_hammock.py: recognizes Π primitives from the normalized CFG — SEQ (T2 + trivial singletons), SPAWN_JOIN (SESE sidechain, I-SC1; >1 join => GOV-SC-MULTIJOIN VIOLATION), WHILE/DO_WHILE (natural loop of a backedge + guard_position pre/post), IF/IF_ELSE. Evidence grade `exact`. - SOUNDNESS: a latent decision (latent_arms>=1) recovers to DECISION_OBSERVED_PARTIAL / ZERO, never a branch (single-execution decision is not evidence of a static branch) - normalize T2 fix: control-flow nodes (decision/spawn/join) are never absorbed into a SEQ region, so a latent decision stays visible to the latent-branch rule; lone tool_calls become trivial SEQ-of-one - 5 tests: SEQ+SPAWN_JOIN full recovery, WHILE, DO_WHILE (not WHILE), latent decision is ZERO not IF, hammock evidence_grade exact
… P5 -> signed StepGate) - narration_fidelity_verifier.py (the named-but-missing file): alpha compiles ClaimIR->Pi-AST; P4 compares the claimed primitive against R_H's recovery over the claim's covers-span; P5 composes POS/ZERO/NEG -> VerifierIR finding -> signed StepGateArtifact (via step_gate). NEG also emits a ReasoningFailureTrace (GOV-NARR-STRUCT-001) - verdict projection: POS->OK(PASS), NEG(semantic)->VIOLATION, ZERO->None(INDETERMINATE)/REVIEW; the verdict is a pure harness function of evidence (model only supplied the claim) - 5 tests: truthful WHILE->POS->PASS (signature verifies); lying WHILE-vs-DO_WHILE->NEG->VIOLATION + failure trace; claim over a LATENT decision->ZERO (no false accusation); unanchored/unstructured->ZERO; verify_all wiring - this is the product surface: agent narration verified against what it actually ran, as a signed verdict
…iteria PASS) - eval_trace_cfr.py: strata S1(canonical->POS)/S4(threaded truthful)/S5(irreducible)/S7(latent)/ S8(lying)/S8z(zero-trip)/S9(vague) generated by the deterministic emitter (emitted structure = ground truth); scores composite verdict + GOV-IRRED against truth; computes Abstention Calibration gap - ACCEPTANCE PASS: S4 NEG=0 & S8z NEG=0 (no false accusation, release blockers), S7 latent never POS, S8 lie-recall 1.0 (>=0.95), S5 GOV-IRRED FP=0 recall 1.0, AC gap 1.0 (clean synthetic separation) - test_eval_trace_cfr.py gates all criteria in CI. AC gap is on synthetic fixtures (honest scope); the harness is the mechanism, real fixtures would show a realistic gap
…ine P5 composition - recover_interval.py: the decorrelated arm. Derives loops from natural-loops-of-backedges + nesting depth (the witness); WHILE/DO_WHILE via guard_position. Distinct contributions over R_H: irreducible region -> IRREDUCIBLE_REGION (GOV-IRRED), verified-grade evidence via the nesting witness (vs R_H exact), nesting_depth for ObligationIR O1. Irreducibility drops grade to sampled - narration_fidelity_verifier: two-engine P5 composition (compose_two). Sign disagreement (POS vs NEG) between the semantic engines -> VERIFIER-FAULT-001 (INDETERMINATE), NEVER a VIOLATION off a self-inconsistent harness. Additive: ri_recovery defaults None, single-engine path unchanged - 5 R_I/two-engine tests + the 5 existing narration tests still green
…< 50ms (was hung at 2000) - the latency bench (SPEC §6) caught two real bugs: naive iterative full-dom-set dominators were superlinear (hung >60s at 2000 events), and recursive DFS would hit Python's recursion limit on a 2000-node linear chain - rewrote trace_cfr_normalize with Cooper-Harvey-Kennedy immediate dominators (near-linear, no full dom-sets on the hot path; idom-chain checks for backedges) + iterative-DFS retreating edges/postorder; kept a lazy `dominators` property for tests/diagnostics - bench_trace_cfr_tier0.py + CI test: measured p95 = 15.55ms at 2000 events (3x headroom); closes the previously-unmeasured Tier-0 acceptance criterion honestly
…et + validator CLI) - tools/validate_trace_cfr.py: repo `validate-*` CLI — checks the trace-cfr-segment schema is valid Draft-2020, a self-test segment round-trips through the full pipeline, and all SP-EVAL acceptance criteria pass; exit 0/1 - Makefile: `validate-trace-cfr` target (json.tool on the schemas + the validator), added as a prerequisite of `validate` via an extra-prereq declaration (no edit to the giant validate: line) - the 69 pytest tests already run under `make test` (pytest tools/tests); trace-cfr is now CI-gated, not loose modules
…ominance/certified) - shape_constraints.py: enforces verifier-ir.schema.v0.2 properties_constraints. property_tested constraints are exercised by metamorphic sampling and return VIOLATION on a counterexample (drift caught, not silently accepted); architectural = trusted-by-construction; certified requires a certificate_id. Implements monotone / joint_monotone / dominance; others accepted with a v0.1 note - makes real the §7 enforcement-mode enum that was schema-only - 7 tests: monotone pass/fail, verified-dominates-unverified, the AC abstention-shape guarantee, certified-requires-cert, architectural-trusted, enforce() aggregation
…umwork T1/T2 tiering - trace_cfr_runtime.py: the deployable capability a governed runner drives. RunRecorder records an agent's control flow + narration claims during a run; finish() seals the segment, runs the full stack (ingest -> CFG -> normalize -> R_H + R_I -> narration compare), aggregates one run-level gate verdict via the §4 receipt fold (most-cautious-wins), and FAILS CLOSED on an unfaithful claim. Harness key supplied by the deployment; deterministic default for local/dev. (No live model loop yet -- this is the integration seam the runner we're building drives.) - mellumwork.py: grounds the T1/T2 tiering the specs cite (the named external framework was not locatable) -- exact/verified/architectural/certified -> T1, sampled/property_tested -> T2; T2->T1 promotion requires durable proof (verified w/o witness stays T2) - 11 tests: truthful run permitted, lying run fails closed w/ signed StepGate + failure trace, mixed run most-cautious, silent run permitted; tier mappings + promotion rule
…delity gate surface - sp_run.py: new `narration-gate` subcommand on the governed-runner delegate. Takes a sealed trace-cfr segment + a ClaimIR claims file, runs the full fidelity stack, emits the gate verdict as JSON, and FAILS CLOSED (exit 0 permits, nonzero denies). Harness key via --key-seed (deployment supplies) or a dev default. This both is the CLI surface and wires the gate into sp-run - trace_cfr_runtime: extracted gate_segment() (the reusable core RunRecorder.finish() and the CLI both call) so an already-sealed run can be gated - 3 CLI tests (permit truthful / deny lying / permit no-claims) + existing sp_run tests still green; real shell invocation verified: lying run -> FAIL, GOV-NARR-STRUCT-001, exit 1
…n-fidelity evidence object - schemas/trace-cfr-run-attestation.schema.v0.1.json + trace_cfr_runtime.build_run_attestation/ verify_run_attestation: bundle a gated run (segment_ref, folded gate_verdict, per-claim verdict counts, anomalies, per-claim StepGate hashes) into ONE ed25519-signed attestation; harness-signed (model-excluded), independently verifiable, tamper-evident - sp-run narration-gate --out writes the signed attestation; --session-id records the run id - Makefile validate-trace-cfr json.tool's the new schema - 4 tests: schema-valid + signature verifies, tampered gate_verdict fails, model-authored rejected, CLI --out writes a valid attestation whose segment_hash matches the sealed segment. 253 tests total
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #311. Builds the SP-TRACE-CFR recovery input + first two harness passes.
Contents
schemas/trace-cfr-segment.schema.v0.1.json+tools/trace_cfr_emitter.py: the D1 control-flow projection as a separate schema (no mutation ofagentic-ops-trajectory-event) plus the reference producer (no live runner emits control-flow yet). Deterministic byte-stable seals;sidechain()makes I-SC1 SESE hold by construction.tools/trace_cfr_ingest.py: seal over raw JSONL bytes + no-repair invariants (MALFORMED_JSON / MISSING_FIELD / NON_MONOTONE_TS / DUPLICATE_EVENT_ID / DANGLING_PARENT / SEAL_MISMATCH).tools/trace_cfr_cfg.py:build_cfg()— nodes fold by(site_id, kind), edge labels from source event, latent-branch rule, guard-position; backedge derivation deferred to P2.Tested
Pipeline runs end-to-end
emit → seal → ingest → build_cfg; 18 tests across the three modules (SESE sidechain, deterministic seal, every ingest rejection path, WHILE-loop cycle, single-exec-decision latency, DO_WHILE guard-position, annotation exclusion).Next on this branch
P2 normalization (T1–T5, incl. backedge derivation) → WO-2 R_H hammock recovery.
🤖 Generated with Claude Code