Skip to content

docs(skills): cue-eval-review — recorded-session review & replay-eval methodology - #87

Merged
xerhab merged 1 commit into
mainfrom
docs/cue-eval-skill
Jul 27, 2026
Merged

docs(skills): cue-eval-review — recorded-session review & replay-eval methodology#87
xerhab merged 1 commit into
mainfrom
docs/cue-eval-skill

Conversation

@xerhab

@xerhab xerhab commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

What

Adds a Claude skill at .claude/skills/cue-eval-review/SKILL.md documenting how cue quality is reviewed and evaluated, so future sessions follow the same methodology that produced the 2026-07 precision cycles (#80, #81, #86):

  • Data export — the docker exec Tenir-Postgres psql queries for segments and cues (host has no local psql).
  • Hand review — interleave production cues into the transcript by at_ms, bucket bad cues into named failure classes (cross-generation confabulation, false corrections, paraphrase duplicates with the measured similarity bands, stale-topic, mishears), review-only until asked to fix. Includes the NFKC-normalization gotcha for regex probes.
  • Replay eval — how to use scripts/cue_eval/ honestly: freeze the conversation set with controls, re-baseline on the current export, iterate subset→full, hand-read every iteration, match production decoding, treat stochastic effects as halved-not-fixed, and use the self-judge only comparatively.
  • Threshold calibration — simulate gates over real production cues, hand-classify the dropped pairs, record the measured bands next to the constant.
  • Ship & document — repo test/PR conventions plus appending an honest results section (including failed attempts and residuals) to the RESULTS doc.

Privacy note baked into the skill: deployment transcripts stay in analysis for the user and never get pasted into shipped code, prompts, tests, or docs.

Verification

Docs-only (no code paths touched). CLI flags and commands cited in the skill were checked against the current replay.py/judge.py argument parsers and the cues/segments table schemas in the deployment DB.

…y-eval methodology

Captures the workflow behind the 2026-07 cue precision cycles (PRs #80/#81/#86)
as a reusable Claude skill: DB export commands, the hand-review failure-class
taxonomy, the replay/judge harness rules that keep comparisons honest
(frozen set, re-baseline per export, subset-then-full, stochastic-effect
handling), threshold calibration on real data, and the ship/document steps.
@xerhab
xerhab merged commit ea860d6 into main Jul 27, 2026
@xerhab
xerhab deleted the docs/cue-eval-skill branch July 27, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant