Design: make debrief portable across Agent Skills-compatible runtimes

[Wavez]

Context

debrief was originally designed as a Claude Code skill. The question raised here is how to
make it available to other Agent Skills-compatible runtimes — Codex in particular — while
preserving the richer Claude Code experience (nested skill invocation, memory, hooks,
CLAUDE.md, transcript access) where it's available.

The first draft of this discussion proposed a .codex-plugin/ directory, per-host reference
forks (references/claude.md, references/codex.md), and a references/host-capabilities.md
capability-negotiation layer. Before committing to that shape, I went and read the actual Agent
Skills specification, Claude Code's own plugin docs, OpenAI's Codex skills docs, and Anthropic's
own skill-authoring tooling — and tested the result against this repo directly. Several of the
original assumptions don't hold up. Below is what's actually true, followed by a smaller
concrete plan.

What the Agent Skills spec actually says

Primary sources: agentskills.io/specification,
github.com/anthropics/skills,
github.com/agentskills/agentskills.

A skill is a directory containing, at minimum, a SKILL.md file:

skill-name/
├── SKILL.md          # required: frontmatter + instructions
├── scripts/           # optional: executable code
├── references/        # optional: documentation
└── assets/             # optional: templates, resources

SKILL.md frontmatter fields, in full — there is nothing beyond this list:

Field	Required	Notes
name	Yes	≤64 chars, lowercase + hyphens, must match the parent directory name
description	Yes	≤1024 chars, what it does and when to use it
license	No	License name or pointer to a bundled license file
compatibility	No	≤500 chars, free-text environment requirements, e.g. "Designed for Claude Code (or similar products)"
metadata	No	Free-form string → string map for anything the spec doesn't define
allowed-tools	No	Experimental, space-separated pre-approved tools

Worth flagging: some of what came up in initial searches (SEO content on agensi.io,
thepromptindex.com, rzlt.io) claimed frontmatter fields like trigger, version, and
compatible_agents exist. They don't. The real spec has exactly the six fields above. I'm
calling this out explicitly because it's the kind of confidently-wrong claim that's easy to
build a design around without checking.

There is no capability-negotiation protocol anywhere in the spec. The compatibility field
is free text — nothing parses it, nothing negotiates against it. The only mechanism the
ecosystem actually provides for "behave differently depending on the host" is writing the
SKILL.md body defensively: conditional prose like "if X is available, do Y; otherwise say X
isn't available and skip." That's it. This directly answers Open Question 1 from the
original post: neither the skill nor the host negotiates capabilities via any schema, because no
such schema exists. And it's already how this skill is written in several places — section 0 and
section 4 of SKILL.md already say "if no memory system... is accessible in this environment,
say so and skip this check" rather than assuming Claude Code's memory system exists.

.claude-plugin/ is not part of the spec — it's Claude Code's own distribution system

This is the load-bearing correction. .claude-plugin/plugin.json and marketplace.json are
Claude Code's own plugin/marketplace mechanism (confirmed via
Claude Code's plugin docs) — how this repo currently
gets installed via /plugin install. They are not mentioned anywhere in the Agent Skills spec,
and Codex has no equivalent concept of a plugin manifest at all.

Concretely, that means:

• .codex-plugin/ doesn't correspond to anything Codex reads. There's no such convention.
• Codex discovers skills by scanning .agents/skills/ — repo-level (current directory → parent
  directories → repo root) and user-level (~/.agents/skills/), per
  OpenAI's own docs. No manifest, no wrapper
  directory, no plugin registration step. It just needs the SKILL.md folder to exist at one of
  those paths.
• The one genuinely real Codex-specific extension point is an optional
  skills/debrief/agents/openai.yaml file — purely cosmetic (display name, icon, whether the
  skill can be auto-invoked or only via $debrief). The skill works fully without it.

So skills/debrief/SKILL.md — right where it already lives, name: debrief matching its parent
directory debrief/ — already satisfies the spec as written. Nothing needs to move.

I tested this empirically, not just from docs

GitHub CLI ships a (preview) gh skill command that installs Agent Skills from any GitHub repo
into the correct host-specific directory, for 40+ agent hosts including Codex, purely via the
skills/*/SKILL.md discovery convention the spec defines. I ran a read-only check against this
repo, live, with zero changes made to it:

$ gh skill preview Wavez/debrief-skill debrief

It found and rendered skills/debrief/SKILL.md correctly on the first try. That means

gh skill install Wavez/debrief-skill debrief --agent codex --scope user

almost certainly already works today, exactly as-is. The "make debrief installable in Codex"
goal this discussion opened with is already met by the repo's current layout — what's missing
isn't structure, it's (a) one file's content, and (b) documentation telling people the install
command exists.

What "evals" means, for anyone reading this who (like me, an hour ago) wasn't sure

A skill is prose, not code — you can't unit-test it with normal assertions. Anthropic's own
skill-creator skill (shipped in anthropics/skills) defines the practice it uses internally:

1. Write a small number of realistic test prompts (Anthropic's own guidance: start with 2-3).
2. Describe what a good output looks like for each one.
3. Run the task twice — once with the skill active, once as a baseline without it (or against
   the previous version, when iterating).
4. Grade the outputs. Two grading modes exist: scripted assertions for skills with
   deterministic output (a file converter, a fixed workflow), or an LLM-as-judge rubric for
   skills with subjective output.
5. Anthropic's own tooling (eval-viewer/generate_review.py) renders a side-by-side HTML view of
   the before/after outputs so a human can look at them and give feedback, which then drives the
   next revision of the skill's wording.

Anthropic's guidance is explicit that subjective-output skills mostly want the rubric approach,
not hard assertions — and debrief is squarely in that bucket. It produces a judgment call (a
retrospective report), not a deterministic transform, so its evals should be a handful of
representative session-transcript scenarios graded by an LLM-judge rubric describing what a good
debrief report looks like for that scenario — not scripted pass/fail checks, and not the
heavier Docker/pass@5 benchmarking machinery built for teams needing CI-grade reliability
guarantees, which is more than this repo needs.

The original proposal's evals/cases/*.md + evals/rubric.md shape was already pointed the
right direction — it just needed the explanation of what goes in it and why.

The actual gap, and live evidence of what happens if you fix it the wrong way

Once .claude-plugin/ is understood as Claude Code's own packaging (not something to fork per
host) and SKILL.md itself is already mostly defensive, the real Claude-specific surface area in
this repo is almost entirely in one file:
skills/debrief/references/suggested-improvements.md.
It routes every suggested improvement to one of: project/local memory, repo CLAUDE.md, global
CLAUDE.md, or a settings.json hook — all Claude Code-specific mechanisms, with no fallback
language for a host that has none of them.

There's also a live, present-tense example in this repo right now of why forking per-host
reference docs (as the original proposal suggested with references/claude.md /
references/codex.md) is the wrong shape: an untracked AGENTS.md file already sits in this
repo, produced by a mechanical find-and-replace of CLAUDE.md. It's already wrong — it
references .Codex-plugin/plugin.json, a path that doesn't exist (the real plugin directory is
lowercase .claude-plugin/, and Codex doesn't use plugin manifests in the first place). That's
the exact "lowest-common-denominator fork drift" this discussion opened by saying it wanted to
avoid, caught in this repo before any of this design work even landed.

Revised plan

Smaller than the original proposal, because most of the original proposal's surface area turned
out not to correspond to anything either spec defines or Codex reads:

1. No .codex-plugin/ directory. Nothing reads it.
2. No references/host-capabilities.md capability-negotiation layer. No host implements
   negotiation; there's nothing to build against.
3. No references/claude.md / references/codex.md split. The Claude-specific surface area
   lives in one file. Fork it and you get exactly the AGENTS.md bug above, twice as fast.
   Instead:
4. Rewrite suggested-improvements.md with a conditional preamble: route to
   memory/hooks/CLAUDE.md when those exist; otherwise fall back to "whatever agent-instructions
   file this host reads" or say plainly that a destination isn't available on this host. Same
   pattern SKILL.md itself already uses successfully.
5. Fix or eliminate the stray AGENTS.md. Its content is almost entirely generic
   git/release-engineering process — nothing in it is actually Claude Code-specific except the
   one broken path. Cheapest fix: symlink AGENTS.md → CLAUDE.md so there is exactly one file
   to keep correct, not two to keep in sync by hand.
6. Optional, cosmetic: add skills/debrief/agents/openai.yaml for a nicer Codex display name
   and icon. Not required for the skill to work.
7. Document the install path in the README:
   gh skill install Wavez/debrief-skill debrief --agent codex (or --agent claude-code, or any
   of the ~40 other supported hosts). This is the actual "make debrief available in Codex"
   deliverable — no repo restructuring required, just this line plus step 4.
8. Add a small evals/ folder: 2-3 scenario files + one rubric, LLM-judge graded, matching
   Anthropic's own "start small" guidance rather than the original proposal's five-case,
   fully-scaffolded suite.

Answers to the original open questions

1. Should the portable skill negotiate capabilities itself, or should each host adapter provide
   an explicit capability profile? — Neither exists in the ecosystem to build against. Write
   SKILL.md prose defensively instead; that's the only mechanism that actually exists.
2. Should Claude remain the canonical "full-featured" adapter? — Yes, and this is a normal,
   spec-sanctioned pattern (the compatibility field's own example is literally
   "Designed for Claude Code (or similar products)"). No opposition needed.
3. Which concepts are important enough to preserve cross-host? — The four-part report format,
   the evidence requirement, and the no-auto-apply rule are already pure prose and already
   host-agnostic. Only the "where do I save this suggestion" destination logic
   (suggested-improvements.md) is Claude-specific, and it needs conditional prose in place, not
   a fork.
4. What's the smallest eval suite that verifies equivalent behavior across hosts? — 2-3
   scenario evals, LLM-rubric graded (per Anthropic's own starting-point guidance), not a
   cross-host equivalence suite — there's no mechanism to run the same skill identically across
   hosts to compare against, since each host's tool surface differs by design.

Next steps

Planning to open a PR against this repo implementing points 4–7 above (the evals/ folder in a
follow-up, since it benefits from a couple of real review cycles first). Flagging here before
starting in case anyone wants to weigh in on the AGENTS.md symlink-vs-rewrite call in point 5,
or has a different view on how much Codex-specific documentation belongs in this repo versus
just linking to gh skill install and letting people install it their own way.

[Wavez]

Implementation plan for the revised design above, written with the writing-plans skill and saved in-repo at docs/superpowers/plans/2026-08-16-agent-skills-portability.md. Bite-sized, TDD-style tasks (write the check, verify it fails/confirms the bug, do the change, verify, commit) — each task ends in its own Conventional Commit so this can land as a stack of small, individually reviewable commits rather than one big diff.

---

Agent Skills Portability Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make debrief install and run cleanly under Codex (and any other Agent
Skills-compatible host), without forking the skill or building a capability-negotiation layer
that nothing in the ecosystem actually uses.

Architecture: The skill's portable core (skills/debrief/SKILL.md) already satisfies the
Agent Skills spec as-is — no directory move needed. The only real gap is one reference file
(suggested-improvements.md) that assumes Claude Code-only destinations with no fallback. Close
that gap with conditional prose (not a per-host fork), add a Codex-cosmetic file, document the
existing cross-host install path, fix a stray broken file already sitting in the repo, and add a
small eval suite so future edits to SKILL.md can be checked for regressions.

Tech Stack: Markdown (SKILL.md, references), YAML frontmatter, a Codex agents/openai.yaml
descriptor, git symlinks, gh skill (GitHub CLI, v2.90+) for cross-host distribution.

Spec: GitHub Discussion #14 — the
research findings and 8-point plan already agreed there. Also: this repo's CLAUDE.md (commit
conventions, release automation) and the upstream Agent Skills
specification.

Global Constraints

• Every commit MUST follow Conventional Commits (<type>: <description>, lowercase, imperative)
  — enforced by .github/workflows/commitlint.yml (@commitlint/config-conventional).
• skills/debrief/SKILL.md and skills/debrief/references/*.md are the skill's actual runtime
  behavior, not documentation — edits to them are typed feat/fix/refactor per what changed,
  never docs. docs: is reserved for README/CHANGELOG-only edits.
• Do not hand-edit .claude-plugin/plugin.json's or .claude-plugin/marketplace.json's
  version fields — release-please owns those.
• Do not create a .codex-plugin/ directory, a references/host-capabilities.md
  capability-negotiation layer, or a references/claude.md / references/codex.md fork of the
  skill body. Discussion Design: make debrief portable across Agent Skills-compatible runtimes #14 establishes why: none of the first two correspond to anything a
  real host reads, and the fork pattern is what produced the broken AGENTS.md this plan fixes
  in Task 1.
• No destructive git operations without explicit confirmation (this plan's Task 1 removes an
  untracked file only — nothing committed is deleted anywhere in this plan).

---

Task 1: Fix the broken AGENTS.md

Files:

• Delete: AGENTS.md (currently untracked — a mechanical, buggy find/replace fork of
  CLAUDE.md)
• Create: AGENTS.md (symlink → CLAUDE.md)

Interfaces:

• Consumes: nothing (first task, no dependency on later tasks)

• Produces: a single source of truth for repo-contributor instructions, read as CLAUDE.md by
  Claude Code sessions and as AGENTS.md by Codex sessions in this repo

• Step 1: Confirm the current AGENTS.md is untracked and contains the known bug

Run:

git status --porcelain AGENTS.md
grep -n "Codex-plugin" AGENTS.md

Expected: first command prints ?? AGENTS.md (untracked); second command prints two lines
referencing .Codex-plugin/plugin.json and .Codex-plugin/marketplace.json — a directory that
does not exist anywhere in this repo (the real one is .claude-plugin/).

• Step 2: Remove the broken file and replace it with a symlink

rm AGENTS.md
ln -s CLAUDE.md AGENTS.md

• Step 3: Verify the symlink resolves correctly and the bug is gone

ls -la AGENTS.md
cat AGENTS.md | grep -n "Codex-plugin"
cat AGENTS.md | grep -n "claude-plugin"

Expected: ls -la shows AGENTS.md -> CLAUDE.md; the first grep returns nothing (no more
.Codex-plugin references); the second grep shows the correct .claude-plugin/ paths,
inherited from CLAUDE.md.

• Step 4: Commit

git add AGENTS.md
git commit -m "fix: replace broken AGENTS.md fork with a symlink to CLAUDE.md"

---

Task 2: Make suggested-improvements.md degrade gracefully on non-Claude hosts

Files:

• Modify: skills/debrief/references/suggested-improvements.md (full rewrite — every
  destination gets a host-agnostic fallback)

Interfaces:

• Consumes: nothing new (this file is already linked from skills/debrief/SKILL.md's "Suggested
  improvements" output-format section; that link stays valid, no signature/name changes)

• Produces: a destinations reference that a Codex (or any other host) session can follow without
  assuming Claude Code's memory system, CLAUDE.md, or settings.json hooks exist

• Step 1: Replace the full file content

Overwrite skills/debrief/references/suggested-improvements.md with:

# Suggested improvements: destinations

Read this once you have at least one suggested improvement to route — it defines the possible
destinations and how to choose between them. Name the specific destination for each item — don't
just say "edit CLAUDE.md".

**Host check first:** the destinations below assume Claude Code's memory system, `CLAUDE.md`,
and `settings.json` hooks. Not every host that runs this skill has those. Before routing
anything, confirm each mechanism actually exists in this session:
- No persistent memory system exposed? Skip the "project/local memory" and "global instructions
  file" destinations below and say so — don't invent a substitute.
- No `CLAUDE.md`, but the host reads its own repo-level instructions file (e.g. `AGENTS.md`)?
  Route there instead, under the same "repo-level, committed, shared" criteria as the "repo
  instructions file" bullet below.
- No lifecycle-hook mechanism (`settings.json` or equivalent)? Drop the hook destination
  entirely and fall back to documenting the regression in whichever text destination is
  available — a hook is an enforcement upgrade over text, not a required destination.

Destinations, where available:

- **Project/local memory** — personal to this repo, not committed. Claude Code-specific; skip if
  unavailable.
- **Repo instructions file** (`CLAUDE.md`, or the host's equivalent such as `AGENTS.md`) —
  committed, shared with anyone who opens this repo.
- **Global instructions file** (`~/.claude/CLAUDE.md`, or the host's equivalent) — applies
  across every repo and session, not just this one. Flag explicitly that it's global so the user
  knows the wider blast radius before agreeing. Skip if the host has no such global file.
- Extracting hand-written logic into a reusable script or skill asset.
- **A hook** (`settings.json`, if the host supports lifecycle hooks) — for a standing
  instruction that's already been through memory/instructions-file once and still regressed.
  Memory can only describe desired behavior; a hook can enforce it. Don't default to this for a
  first-time miss — text guidance is cheaper and usually sufficient. Reach for it specifically
  when the regression check (section 4) shows the same rule failing again despite already being
  documented. Scope the hook to match the regressed rule's own scope, not to whichever repo you
  happen to be in right now: a habit that came from *global* instructions needs a *global* hook
  (`~/.claude/settings.json`) — a project-only hook would leave every other repo's sessions
  still exposed to the same regression. A convention that came from the *repo* instructions file
  or project memory needs a *project* hook instead (`.claude/settings.json`, or
  `.claude/settings.local.json` if it shouldn't be shared with other contributors). Skip
  entirely if the host has no hook mechanism.

These destinations aren't mutually exclusive — a finding can belong in more than one. Ask
independently for each, don't stop at the first match:
- True regardless of which repo you're in (a general working-style preference)? → global
  instructions file, if one exists on this host.
- Worth a personal, detailed record for next time (the incident, the "why," the full context)?
  → project memory, if this host has one.
- A reusable fact, technique, or convention that anyone working in this codebase — including a
  fresh session with no memory of this one — would need to redo the same work? → the repo
  instructions file, even if it's *also* going to project memory in more detail.
- A rule that's already been documented (memory or the instructions file) and regressed anyway?
  → a hook, if the host supports one, in addition to keeping the existing documentation as
  context for *why* the hook exists — scope it to match where that documentation lives (global →
  global hook, repo/project → project hook), not to the current repo by default.

• Step 2: Verify no destination is stated as unconditionally available

grep -n "skip if" skills/debrief/references/suggested-improvements.md
grep -c "Claude Code" skills/debrief/references/suggested-improvements.md

Expected: the first command shows at least 4 matches (memory, global file, hook criteria, hook
skip clause); the second shows the file references "Claude Code" only where a mechanism is
explicitly called out as Claude-specific, not as an assumed default.

• Step 3: Commit

git add skills/debrief/references/suggested-improvements.md
git commit -m "feat: make suggested-improvements destinations degrade gracefully on non-Claude hosts"

---

Task 3: Add a cosmetic Codex descriptor

Files:

• Create: skills/debrief/agents/openai.yaml

Interfaces:

• Consumes: nothing (purely additive, optional per the Agent Skills spec — SKILL.md behavior
  is unaffected whether or not this file exists)

• Produces: Codex-specific display metadata; no other task depends on this file existing

• Step 1: Create the descriptor

mkdir -p skills/debrief/agents

Write skills/debrief/agents/openai.yaml:

display_name: Debrief
short_description: Reviews how this session went — process, not the deliverable it produced.
allow_implicit_invocation: true

• Step 2: Verify it's valid YAML and doesn't break skill discovery

python3 -c "import yaml, sys; yaml.safe_load(open('skills/debrief/agents/openai.yaml'))" && echo "valid yaml"
gh skill preview Wavez/debrief-skill debrief --allow-hidden-dirs 2>&1 | head -5

Expected: valid yaml prints with no error. The gh skill preview call targets the published
repo (unaffected by this uncommitted local file) and should still succeed exactly as it did
during discussion #14's research — this step's real purpose is confirming the local YAML parses
cleanly before it's committed and later observed live once pushed.

• Step 3: Commit

git add skills/debrief/agents/openai.yaml
git commit -m "feat: add Codex display metadata for the debrief skill"

---

Task 4: Document the cross-host install path in the README

Files:

• Modify: README.md:26-35 (the existing ## Install section)

Interfaces:

• Consumes: nothing

• Produces: user-facing install instructions for Codex and other gh skill-supported hosts,
  placed immediately after the existing Claude Code install block

• Step 1: Insert a new subsection after the existing install block

In README.md, after the existing block (currently lines 26–35: the ## Install heading
through the > [!TIP] auto-update line), insert:

### Other agent hosts (Codex, etc.)

`debrief` is a spec-compliant [Agent Skill](https://agentskills.io/specification) — the same
`skills/debrief/SKILL.md` works outside Claude Code. Install it into Codex (or any of the
~40 other hosts [GitHub CLI's `gh skill`](https://github.blog/changelog/2026-04-16-manage-agent-skills-with-github-cli/)
supports) with:

```
gh skill install Wavez/debrief-skill debrief --agent codex --scope user
```

Swap `--agent codex` for `--agent claude-code`, `--agent cursor`, etc. Preview the skill's
content first without installing anything:

```
gh skill preview Wavez/debrief-skill debrief
```

• Step 2: Verify the README renders correctly and the commands work as documented

grep -n "Other agent hosts" README.md
gh skill preview Wavez/debrief-skill debrief 2>&1 | head -5

Expected: the grep confirms the new heading landed; the gh skill preview call succeeds
(read-only, matches the verification already done in discussion #14).

• Step 3: Commit

git add README.md
git commit -m "docs: document cross-host install via gh skill"

---

Task 5: Add a small eval suite for SKILL.md regressions

Files:

• Create: evals/rubric.md
• Create: evals/cases/redundant-tool-calls.md
• Create: evals/cases/unverified-done-claim.md
• Create: evals/cases/clean-session.md

Interfaces:

• Consumes: the report format defined in skills/debrief/SKILL.md's "Output format" section
  (four headed sections: Overall assessment, Concrete issues, What went well, Suggested
  improvements; severity markers 🔴 High / 🟡 Medium / 🟢 Low)

• Produces: a fixed set of session scenarios + a shared grading rubric that any future edit to
  SKILL.md can be re-run against by hand (run /debrief against each scenario, grade the
  output against rubric.md) to check for regressions before merging

• Step 1: Write the shared rubric

Write evals/rubric.md:

# Debrief eval rubric

Use this to grade a `/debrief` report produced against one of the scenarios in `cases/`. Score
each dimension 0.0–1.0. A passing report scores at least 0.8 on every dimension.

## Dimensions

**1. Format compliance (0.0–1.0)**
Does the report contain exactly the four headed sections from `SKILL.md`'s "Output format" — in
order: Overall assessment, Concrete issues, What went well, 💡 Suggested improvements? Deduct for
missing, reordered, or renamed sections.

**2. Evidence requirement (0.0–1.0)**
Does every item under "Concrete issues" cite a specific, real moment from the scenario (a named
tool call, a specific quoted claim, a specific repeated action) rather than generic advice?
`SKILL.md` section 0 requires this explicitly — score 0.0 for any finding that could apply to
any session unchanged.

**3. Severity calibration (0.0–1.0)**
Do severity markers (`🔴`/`🟡`/`🟢`) match `SKILL.md`'s calibration guidance? A `🔴` must be
wasted work, wasted tokens, or a claim that risked being acted on while wrong — not routine
inefficiency. Score down for a `🔴` on something that's actually a `🟢`-level nitpick, or for a
real regression downgraded to look safe.

**4. No fabricated criticism (0.0–1.0)**
For scenarios with little or nothing wrong (see `clean-session.md`), does the report say so
plainly instead of manufacturing issues to fill out every section? `SKILL.md`'s Rules section
requires this. Score 0.0 if any "Concrete issues" item doesn't survive the evidence-requirement
check above.

## How to run

1. Feed the scenario in a `cases/*.md` file to a fresh Claude Code session as if it were the
   actual conversation history for that session (paste it as the setup, then invoke `/debrief`).
2. Grade the resulting report against all four dimensions above.
3. Record the scores and the specific evidence for any score below 0.8 — cite the sentence in
   the report that failed the dimension.

• Step 2: Write the first case — redundant tool calls

Write evals/cases/redundant-tool-calls.md:

# Case: redundant tool calls

## Scenario

The user asks the assistant to check whether `src/utils/parse.py` has a bug in its date-parsing
function. Over the session:

1. The assistant reads `src/utils/parse.py` in full.
2. The assistant reads `src/utils/parse.py` again three turns later, having made no edits to it
   and having received no indication the file changed, to "double check" the function signature
   before writing a fix.
3. The assistant writes the fix, then reads `src/utils/parse.py` a third time immediately after
   writing it, to confirm the edit landed — despite the Edit tool's own result already showing
   the new content.
4. The assistant runs the test suite once, it passes, and the assistant reports the fix is done.

No user corrections, no compaction, single file touched throughout.

## Expected debrief output

- **Concrete issues** must include a `🟡 Medium` (not `🔴` — this wasted tokens but the final
  claim was still correctly verified by the test run) finding citing the specific pattern: the
  same file read 3 times with no changes in between, naming at least the "double check before
  writing" re-read as the avoidable one (the post-Edit read is more defensible since it's
  confirming the actual write, though also flaggable).
- Must **not** invent an unrelated finding — nothing else went wrong in this scenario, so
  "What went well" should note the test-before-claiming-done step, and there should be no
  second `🔴`/`🟡` beyond the repeated-reads finding.

• Step 3: Write the second case — unverified done claim

Write evals/cases/unverified-done-claim.md:

# Case: unverified "done" claim

## Scenario

The user asks the assistant to fix a failing test in `tests/test_billing.py`. The assistant:

1. Reads the failing test and the implementation it covers.
2. Edits `src/billing.py` to change the rounding logic.
3. Tells the user "That should fix it — the rounding now matches the test's expectation," and
   stops, without running `pytest tests/test_billing.py`.
4. The user replies "I ran it, it's still failing — off by one cent on the second test case."
5. The assistant re-reads its own edit, finds a second rounding call it had missed, fixes it,
   runs the test suite this time, and confirms it passes.

## Expected debrief output

- **Concrete issues** must include a `🔴 High` finding: the assistant claimed the fix was
  complete without running the test suite, and the claim was wrong — this is exactly the
  "claim that risked being acted on while wrong" case `SKILL.md`'s calibration guidance
  reserves `🔴` for. The finding must cite the specific claim ("That should fix it...") and the
  fact that verification (`pytest`) was skipped before making it, not after.
- Must **not** soften this to `🟡` — the report should treat this as the single most severe
  finding in the session, per the calibration note that most clean sessions have zero or one
  `🔴`, reserved for genuine wrong claims that could have shipped.
- "What went well" may reasonably note that the assistant did verify before its *second* claim
  of completion.

• Step 4: Write the third case — clean session (calibration check)

Write evals/cases/clean-session.md:

# Case: clean session

## Scenario

The user asks the assistant to rename a config field `maxRetries` to `retryLimit` across the
codebase. The assistant:

1. Runs a single grep for `maxRetries` across the repo, finds it in 3 files.
2. Reads each of the 3 files once.
3. Edits all 3 in a row, batching the edits with no unnecessary back-and-forth.
4. Runs the existing test suite once; it passes.
5. Reports the rename is complete and cites the 3 files changed.

No errors, no repeated reads, no user corrections, four total turns.

## Expected debrief output

- "Overall assessment" should say plainly that this was a clean, efficient session — not stretch
  to find a problem.
- "Concrete issues" should be **empty or explicitly state there's nothing to report**, per
  `SKILL.md`'s Rules section ("don't manufacture criticism to fill out every section; if a
  section has nothing real, say so briefly and move on").
- Any report that invents a `🟢 Low` nitpick not grounded in a real inefficiency in this
  scenario (e.g. commenting on turn count, or claiming the grep "could have been narrower" with
  no evidence it caused any actual cost) **fails the "no fabricated criticism" rubric dimension**
  — this case exists specifically to catch that failure mode.

• Step 5: Verify all four files are internally consistent

ls evals/cases/
grep -l "🔴\|🟡\|🟢" evals/cases/*.md
grep -c "Expected debrief output" evals/cases/*.md

Expected: ls shows exactly 3 case files; the severity-marker grep matches all 3 (each case
states what markers it expects); the third command shows 1 for each file (one "Expected
debrief output" section per case, no duplication).

• Step 6: Commit

git add evals/
git commit -m "test: add a small eval suite for SKILL.md regressions"

---

Self-Review Notes

• Spec coverage: all 5 non-deferred items from discussion Design: make debrief portable across Agent Skills-compatible runtimes #14's revised plan are covered —
  fix AGENTS.md (Task 1), fix suggested-improvements.md (Task 2), add agents/openai.yaml
  (Task 3), document install (Task 4), add evals/ (Task 5). The discussion's point 8 already
  noted evals/ would land in this pass; nothing from the 8-point list was deferred beyond what
  discussion Design: make debrief portable across Agent Skills-compatible runtimes #14 itself deferred (the follow-up PR was scoped to points 4–8, all of which are
  tasks here).
• No placeholders: every step above writes complete, real file content — no "TBD" or
  "similar to Task N" shortcuts. Case files fully spell out scenario + expected output rather
  than gesturing at them.
• Naming consistency: evals/cases/*.md and evals/rubric.md match the paths discussion Design: make debrief portable across Agent Skills-compatible runtimes #14
  committed to; skills/debrief/agents/openai.yaml matches the path OpenAI's own docs specify
  for this optional file.