Repository navigation
Agent execution review
Recorded 2026-10-11. Objective: reduce the number and cost of agent/CI repair rounds while preserving FabricO11y's correctness, resource and evidence gates. This report diagnoses observed execution failures and proposes a measurable Codex CLI improvement. The proposed agent comparison has not run.
The strongest observed cause is incomplete validation of the execution system around a patch. Several expensive runs began before dependency inputs, fixture admission, compiler selection and verifier isolation were checked together. Generating another implementation from the same incomplete context could repeat those failures. Better preparation should supply executable constraints and feedback, then give generation a bounded opportunity to explore alternatives.
Some rounds found real missing cases; some repaired defective harnesses; some corrected errors introduced by the agent. These categories require different actions. Fewer rounds is useful only when the same independent gates remain.
The integration review records commands, revisions, resource observations and failure preservation. The examples below are grounded in that evidence rather than an interpretation of hidden model reasoning.
| Cause | Observed example | Earlier discriminator |
|---|---|---|
| Fixture and product constraints were not reconciled | The browser fixture emitted 10 probes/s against an 8 requests/s private-console peer budget and received HTTP 429 | Compare fixture cadence with admission constants; run one real enrollment before the full browser campaign, retaining a separate overload control |
| Build placement was assumed | Delivery looked under repository target/release while Cargo built on the data drive |
Resolve the effective target and execute the actual binary's startup/readiness probe |
| Shared build state invalidated isolation | Mutation workers inherited an absolute shared Cargo target; logs showed a worker running another source tree's executable | Use worker-local targets and verify a known defect fails in that worker before starting the audit |
| Negative controls drifted from production | Binding mutations edited the frozen legacy reference, and a remote-configuration selector targeted a moved line | Require a unique production edit, show its diff, and run its named checker before the complete registry |
| Generated tests did not independently cover boundaries | Isolated mutations exposed mixed-number boundaries, promotion/binding cases, inclusive timestamps, equal-time identities and cursor pruning | Literal expected outcomes and an independent bounded relational reference, alongside unchanged Python oracles |
| The agent introduced an ordinary implementation error | A launcher refactor added a redundant borrow; Clippy failed | Run relevant lint before pushing the candidate. The fix removed the borrow without disabling the lint |
| The dependency freeze omitted a workspace | Fuzzing silently updated tracked fuzz/Cargo.lock; later checks exited zero on dirty inputs and final receipt validation rejected them |
Locked resolution of every workspace before expensive checks; a post-trial digest and final live-worktree check |
| Local tool selection was incomplete | One fuzz attempt used direct Cargo without the rustup proxy; another compiled for 318.670 s before finding no C++ compiler, despite an existing data-drive toolchain | Probe stable and nightly through the intended executable; validate CXX and a small compile before the full build |
| Invalid implementations could exhaust their test clients | An isolated cursor mutant made the client accumulate repeated pages: five-second timeout, 1.9G cgroup peak | Check strict progress over finite keys before the original complete-drain assertion; retain every original answer assertion |
| Interrupted work lost its feedback | Hosted mutation run 38156896105 timed out without an active audit transcript because the registry buffered until exit | Write partial streams while work runs and retain raw tool outcomes; require completion receipts separately |
The agent owns the redundant-borrow and local tool-selection mistakes. They are not evidence of an unavoidable network or hardware limitation.
The fuzz failure is especially instructive. Extended run 38151336752
ran from 15:27:20 to 16:34:00 UTC: 66 minutes 40 seconds, including setup and
checks. All ten check commands exited zero, but input identity was invalid.
Local cargo metadata --manifest-path fuzz/Cargo.toml --locked --offline
reproduced exit 101. The successor freezes the lock and rejects rewrites; the
final validator also rejects tracked or untracked live changes. See
lock repair,
fuzz guards, and
live-input validation.
Six earlier candidate heads from 2c1ceb2 through 1777e66 each launched three
workflows. The table follows the inspected October 10–11 integration window;
it is not a project-wide failure-rate estimate. The final successor is included
to distinguish the completed repair from those historical attempts.
| Head | Fast / focused | Extended outcome |
|---|---|---|
2c1ceb2 |
Browser 429 failure / success | Failed: interrupted mutation audits and delivery path failure |
f56de16 |
Success / success | Canceled; preserved as incomplete |
6aa7e4c |
Canceled / success | Canceled; preserved as incomplete |
4f300a9 |
Redundant-borrow lint failure / success | Canceled after known failure |
eb5b096 |
Success / success | Failed final input/receipt validation after all ten commands exited zero |
1777e66 |
Success / success | Mutation audit exceeded 30 minutes; remaining workflow canceled after the recorded failure |
ed4b95c |
Success / success | Success; all ten extended receipts independently validated before merge |
A source push supersedes CI work, and repeated cold builds amplify a small mistake. The last two failed families were detectable before the longest run. Passing a component does not mean a candidate satisfies the complete gate. Cancellation saves resources once a defect is known, but the canceled run still provides no completion evidence.
There was also duplicate scheduling. Head 61d2946 launched the same Rust/console
workflow for push
and pull request
three seconds apart. The candidate's Rust workflow now restricts push runs to
main, retaining the PR run and post-merge main check. This removes a demonstrated
duplicate without removing either required verification surface.
During this review the agent also repeated status polling and lengthy handoffs. Those are observed workflow costs; their token share has not been measured. Compact persisted state and a deterministic monitor should handle mechanical waiting, leaving inference for new evidence and consequential decisions.
The conversation's scope expanded from ingestion memory into query algorithms, upstream research, RCA, self-observation, authentication, PWA and packaging. That increased the number of interacting obligations. Continuous research needs a backlog; a usable product needs a finite acceptance candidate. Preserve both by admitting a bounded delivery queue and recording new research outside its freeze. Scope growth does not excuse stopping an already authorized finite task.
The inspected goal state is usageLimited, not complete. That is one concrete
reason automatic continuation cannot be assumed. It does not explain every
earlier stop. An active finite goal and a durable admitted queue are separate
requirements: a budget-limited goal cannot be treated as active by an instruction
to keep working. OpenAI documents thread-scoped goal state and budget-dependent
continuation in Using Goals in Codex.
The context hook reads up to 8,000 characters of CURRENT.md from the invoking
checkout. Here the original checkout and isolated candidate differed; a session
hook also reported truncation. That creates a risk of using stale status and
losing the actionable tail of the document. This is an observed mismatch and a
plausible contributing mechanism, not a proven token-cost or failure attribution.
The configured Bun documentation hook remains disabled. Passive telemetry hooks
do not return context or block completion; their current schema records lifecycle
and tool status, not per-turn token usage.
A separate read-only inspection of this thread's local CLI event log found 7,099 usage events with no observed cumulative-counter decrease or malformed JSON line. The latest usage snapshot was 2026-10-11 17:37:37 UTC, within a log spanning October 4–11. These are recorded cumulative counters, not an attribution of cost to the six integration heads above.
| Counter | Recorded tokens |
|---|---|
| Input | 808,172,514 |
| Cached input | 790,077,824 |
| Input minus cached input | 18,094,690 |
| Output | 2,817,826 |
| Reported reasoning output | 1,169,877 |
| Total input plus output | 810,990,340 |
Cached input is approximately 97.76% of reported input. Repeated cached context dominates that large total; it is not 808 million fresh input tokens. Do not add reasoning output again to the recorded input-plus-output total. Pricing, model changes and provider accounting prevent a dollar estimate from these counters alone. A usage event is also not necessarily a distinct inference.
The log contained 142 turn-context records: 87 selected xhigh, max or ultra,
30 selected medium, 13 selected high, two selected low, and ten lacked an
effort value. Models varied between GPT-6 Astra and GPT-6.1 Sol. This supports
testing a cheaper setting for routine work; it does not show that higher effort
caused the defects. Harder tasks are a confounder.
Only counters and selected model/effort fields were exported into the controlled
data-drive receipt agent-usage-observation.json. Prompts, reasoning contents,
credentials and command payloads were excluded. The streaming inspection exited
0 under a 2 GiB cgroup with no swap: 0.984 s service runtime, 0.888 CPU s,
173.3 MiB reported peak. Per-repair-round cost attribution remains to be done.
Keep stable constraints in AGENTS.md and task-specific procedures in skills. Expose the active worktree, revision, queue and next action through a small run record. Do not add another long global instruction essay. Official Codex best practices recommends practical repository guidance, explicit completion criteria and task-relevant context. That guidance supports this proposal; it does not establish an empirical improvement for Fabric.
Let generation produce a candidate from context, and let deterministic checks admit or reject it. Success requires environment readiness, fixture validity, implementation correctness, oracle sensitivity, frozen inputs and complete evidence. Formally:
P(success | context) = product of P(stage_i succeeds | prior stages, context)
This uses conditional probabilities; the stages are not assumed independent. More reasoning can improve a candidate while leaving a missing compiler or stale lock untouched. Preparation should raise the probability of satisfying the whole chain, especially its weakest known transitions.
For a preflight costing c, detecting a defect with probability p, and avoiding
later wasted work L, it is worthwhile when p × L > c. These probabilities
are uncalibrated hypotheses. The 66-minute lock failure gives an observed L for
one case, not a universal estimate.
Allocate exploration by expected useful information per token and execution cost. Generate genuinely different causal hypotheses, choose the smallest discriminating experiment, and converge when evidence distinguishes them. Repeated wording of the same assumption adds little diversity. Once a candidate is frozen, queue new ideas outside it unless evidence requires a repair.
- Admit a finite task. Record outcome, authorized scope, invariants, required commands, resource limits, cleanup and an explicit completion condition. Keep release qualification distinct from source integration.
- Check prerequisites before implementation. Inspect current main and the active worktree. Resolve every workspace lock, compiler/proxy path, native prerequisites, effective build/scratch roots and enforced cgroup. Probe actual fixture startup, trust material, cadence and owned cleanup. Reuse validated data-drive tools rather than installing duplicate toolchains.
- Explore only consequential uncertainty. Maintain a short assumption ledger: claim, observed evidence, competing explanation, discriminating check. For difficult bugs, consider two or three different mechanisms before editing.
- Repair one causal family. Classify failures as product, verifier, harness, environment or evidence defects. Preserve the failure and add the appropriate deterministic witness. Never weaken an oracle to match an implementation.
- Run checks in dependency and cost order. Cheap configuration/format/lock checks first; focused behavior and a representative negative control next; complete registered gates last. Use isolated caches where source variants require isolation. Ordinary positive checks retain their full domain.
- Freeze and finish. Record commit, tree, toolchain and input hashes; push one coherent candidate. Monitor matching CI without editing its source. Keep wiki observations separate. Validate all required receipts, merge only the tested candidate, and clean owned scratch. A partial pass is an internal checkpoint.
A compact run record should contain objective, scope, worktree, head,
tree, input_hashes, queue, active_runs, receipts, failed_evidence,
resource_limits, next_action, external_dependencies and done_when.
Resume from verified artifact identity rather than reconstructing the task from
conversation alone. Model prose is never the completion oracle.
Read-only inspection found codex-cli 0.160.0 and local defaults
gpt-6.1-sol, xhigh, default service tier. This reports configured defaults;
it does not prove every earlier turn used them. No global settings were changed.
Test medium reasoning for routine implementation and mechanical triage, retaining higher effort for security, algorithms and stubborn causal uncertainty. Keep the same model initially so a workflow comparison is not confounded by a model change. Supported effort varies by model/client; use documented overrides from the configuration reference. No temperature or sampling-seed control is assumed.
Capture codex exec --json events and a schema-constrained final record through
the existing resource launcher, using owned data-drive paths. Local exec --help
confirms --json, --output-schema, --output-last-message and per-run -c.
The official non-interactive guide
documents these interfaces and usage events. Collect actual emitted usage,
including cached inputs where present; do not infer billed cost from conversation
length. Raw streams may contain private material and require controlled storage.
Let a finite external controller own queues, deadlines, artifact identity and CI waiting. It should invoke the agent when new evidence requires a decision, and reject unsupported completion reports. Do not turn the controller into unlimited resampling. Delegation, when authorized, needs bounded assignments and artifact contracts; another model's agreement cannot substitute for executable evidence.
Two narrow verifier controls have already been applied during integration: the cursor mutant changed from timeout to a caught failure, and forced-stop output changed from a failing retention regression to a passing one. The clean successor's complete 286-mutation audit retained its seven equivalence reasons; seven prior paging timeouts became assertion failures. This demonstrates those specific mechanisms, not the broader agent-workflow hypothesis below.
The final hosted successor also completed the same 286-mutation audit: 242
caught, seven reasoned equivalents, 36 unviable and one timeout, in 334.097 s
recorded check time. All three matching workflows passed; downloaded receipts
validated before PR 31 was merged at 6a3b9c9 with the identical tested tree.
This closes the observed integration queue. Cache and execution differences
prevent attributing the entire timing change to one repair, and this result does
not establish the effectiveness of the proposed CLI protocol or qualify a release.
Use isolated, bounded fixtures derived from the retained cadence, shared-target, stale-selector, redundant-borrow and fuzz-lock counterexamples. Compare the current workflow with the protocol above on paired task variants. Keep model, toolchain, resource caps, required gates and initial cache state matched. Use at least three independent runs per task cell as a pilot; that is not enough by itself to claim statistical superiority. Separate known-regression replay from held-out variants. Keep reasoning effort fixed for the initial workflow comparison. Test medium versus xhigh separately afterward; combining both changes would obscure their individual effects. Expand into an interaction study only if the pilot warrants the additional inference cost.
Null hypothesis: the protocol does not reduce avoidable full-CI revisions or cost-to-verified-completion. Alternative: it reduces those costs while preserving correct answers and rejection of every registered negative control.
Record first-full-gate success, source revisions, canceled work, failure class, wall/CPU time, emitted token usage, active inference versus wait time, peak memory, disk use, cleanup and false completion claims. Report paired differences and uncertainty. Select a cheaper setting only if it maintains verification quality; few rounds achieved by skipping checks is a failed optimization.
The pilot remains proposed. Current repairs demonstrate that specific defects are caught, not that the whole agent protocol has already improved success rate. The captured cumulative counters are not yet mapped to individual repair rounds or billing, and hidden generation state is not a causal measurement. Observed execution evidence supports the diagnoses above; broader attribution requires the proposed paired comparison.
Sanitized research belongs here. Raw observations and receipts remain under
/run/media/kmosoti/data/FabricO11y/evidence/release-triage-2026-10-11.