Skip to content

v0.8.0 — Measure the toolkit, not the examples

Latest
Latest

Choose a tag to compare

View all tags
@ifrit98 ifrit98 released this 19 Oct 16:39
· 1 commit to main since this release
v0.8.0
8f62a24
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Measure the toolkit, not the examples.

  • explainer eval artifact: 30 held-out subjects (10 dev, 20 test) with fixed quizzes. A fresh agent builds each explainer under each condition (no guidance, the principles, the explain skill at the quick or published tier); a second answers the quiz from the renderings alone, quoting the passage behind each answer (unquotable answers score 0); a third grades blind.
  • The examples are frozen as reference implementations and regression fixtures (ROADMAP §15).
  • AGENTS.md for coding agents other than Claude Code: no @ imports, no .claude/ paths.
  • Docs: the author's eight understanding questions and the blind test's seven are described as two separate sets; "at least three dev subjects" everywhere.

What it found (dev baseline, 10 subjects)

  • The quiz is at its ceiling. Every condition scores 94–100%, with a strong reader and with Haiku, so it cannot rank the conditions.
  • The principles carry the measurable value. Appended to the system prompt, they cut length 40% (1,689 → 1,006 words), had the best mean effort rank in all three grading passes, and cost the least ($0.21 per build).
  • The quick tier is the shortest (606 words) but ranks below the principles and makes the most factual errors (4–9 across the passes).
  • The published tier costs 13 times as much as the principles ($2.81) and takes 11.8 minutes, for a lower rank and the same length. Its errors (1–2) overlap the principles' (1–4): no consistent measured gain.
  • This is a 10-subject dev run, and the effort ranking is a model's. It is noisy; a person's ranking has to confirm it before it steers a change.

The first run was invalid, for two reasons: builders ran inside this repo and loaded its CLAUDE.md in every condition, and the grader read Markdown with inline code stripped. Both are fixed; builds now run in a temp folder outside every project.

Details: What the artifact eval found.

Upgrade: plugin users get v0.8.0 on the next /plugin update; the explainer wrapper now pins @v0.8.0. explainer eval artifact needs the claude CLI.

Assets 2
Loading