|
| 1 | +--- |
| 2 | +name: agent-performance-audit |
| 3 | +description: "Audit local Codex agent histories for a repository, record separate Claude activity coverage, compare correction and efficiency trends, and generate privacy-minimized JSON plus self-contained HTML. Use for Theo-style agent audits, monthly agent-performance checks, or post-issue-batch workflow reviews." |
| 4 | +--- |
| 5 | + |
| 6 | +# Agent Performance Audit |
| 7 | + |
| 8 | +Produce comparable Codex behavioral audits without copying raw conversations |
| 9 | +into the report. Claude history contributes separate coverage counts, not the |
| 10 | +Codex behavioral metrics. Keep source histories local and read-only. |
| 11 | + |
| 12 | +## Run |
| 13 | + |
| 14 | +1. Resolve the repository, an exact lowercase marker matching its repository |
| 15 | + basename, and an inclusive date window. The marker is a sanity check; |
| 16 | + histories are scoped by the target's canonical checkout path or normalized |
| 17 | + remote identity. Use a monthly window or the period since the previous major |
| 18 | + issue batch. Codex turns are included by task-start date and retained whole |
| 19 | + across date boundaries. |
| 20 | +2. Use the task visualization directory for output. Identify the current audit |
| 21 | + task's session ID when available and pass `--exclude-session`; this validates |
| 22 | + that it contains a detectable audit turn but never drops the session's other |
| 23 | + turns. The script also detects audit turns by invocation/request markers. |
| 24 | +3. Run: |
| 25 | + |
| 26 | + ```bash |
| 27 | + python3 <skill-dir>/scripts/audit_agent_history.py \ |
| 28 | + --repo /absolute/repository/path \ |
| 29 | + --marker repository-name \ |
| 30 | + --since YYYY-MM-DD \ |
| 31 | + --until YYYY-MM-DD \ |
| 32 | + --output-dir /absolute/visualization/path |
| 33 | + ``` |
| 34 | + |
| 35 | +4. For a comparison, pass the previous generated JSON with `--baseline`. |
| 36 | +5. Inspect aggregate results. Read raw surrounding history only for a small |
| 37 | + number of representative causal cases; keep those notes short and abstract. |
| 38 | + Never paste full messages, commands, outputs, or hidden instructions. |
| 39 | +6. If causal notes materially improve the report, write a JSON array with |
| 40 | + `title`, `cause`, `control`, and `status`, then rerun with `--case-notes`. |
| 41 | +7. Deliver the generated HTML link. Keep JSON beside it as reproducible evidence. |
| 42 | + |
| 43 | +Run `python3 <skill-dir>/scripts/audit_agent_history.py --help` for all flags. |
| 44 | + |
| 45 | +## Privacy Contract |
| 46 | + |
| 47 | +- Emit aggregate counts, rates, tool categories, duration percentiles, |
| 48 | + category-only correction summaries, and concise causal notes only. |
| 49 | +- Exclude injected AGENTS/skill/plugin/browser/environment/automation messages, |
| 50 | + explicit audit sessions, session IDs, source paths, cwd values, raw tool |
| 51 | + inputs/outputs, full user messages, and result excerpts. |
| 52 | +- Redact URLs, emails, home paths, UUIDs, secret-shaped strings, and long hashes. |
| 53 | +- Keep the HTML self-contained, script-free, under 512 KB, and free of local |
| 54 | + filesystem paths or private links. |
| 55 | +- Do not upload or publish the report unless Bram explicitly requests it. |
| 56 | + |
| 57 | +## Interpretation |
| 58 | + |
| 59 | +- Correction detection favors precision and misses polite dissatisfaction. |
| 60 | +- Nonzero shell output is not automatically agent error; separate test/build |
| 61 | + failures, probes, invocation errors, missing paths/modules, and permissions. |
| 62 | + Batched shell wrappers without per-command result envelopes are reported as |
| 63 | + uncovered and excluded from per-shell-output denominators. |
| 64 | +- Token counts are model-accounted usage, not unique text or a billing estimate. |
| 65 | +- Compare schema-2 datasets produced by this script for the same repository and |
| 66 | + matching-duration windows. Schema 2 derives each turn from cumulative Codex |
| 67 | + token deltas. Do not compare it directly with older one-off or schema-1 data. |
| 68 | +- A local HTML report is diagnostic evidence, not a public performance claim. |
| 69 | + |
| 70 | +## Cadence |
| 71 | + |
| 72 | +Run monthly or after a major issue batch. Create a recurring automation only |
| 73 | +when Bram explicitly requests scheduling. |
0 commit comments