hook-grader

An open-source eval that grades whether an ad hook will stop the scroll — calibrated against the judgment of a world-class creative strategist.

Most ad hooks die in the first second. This is the eval we use to catch the bad ones before they ship — and we open-sourced it so you can run it yourself, and understand exactly how it works.

It's also the simplest real example of an eval we could think of. If evals feel like a black box, read this repo — it'll click.

---

What an eval actually is (60-second primer)

An eval is just two things:

1. A judge — a rubric that reads something and grades it. Ours lives in JUDGE.md: give it a hook, it returns PASS or FAIL and the reason.
2. An answer key — examples a human graded by hand, so you can check whether the judge is any good. Ours lives in eval-set.json: 116 real ad hooks, each graded by Alysha, a world-class creative strategist.

A judge is only worth trusting if it agrees with the human. Ours agrees with Alysha's calls about 3 out of 4 times (~74%) — and you can verify that number yourself in one command (see below). When it disagrees, it errs strict: it's more likely to fail a hook Alysha would pass than the reverse. For a bad-hook detector, that's the right bias.

That's the whole trick. An eval is a test with a graded answer key. Once you see it here, you can build one for anything you care about.

What it's good at (and what it isn't)

Be clear-eyed about this — it's what makes the eval trustworthy:

• It's very good at catching what won't work. The rubric is a detailed taxonomy of the specific ways hooks fail — product-before-tension, unfinished thoughts, manufactured claims, calling out a demographic with no real stake. That's the reliable part.
• A PASS means a hook cleared the bar — not that it'll be a hit. No eval can promise a home run; that still takes real-world testing. What it can do is kill the dead-on-arrival hooks so you only spend on the ones worth testing.

Think of it as a bad-hook detector with a real floor for "good," not a talent scout for "great."

Use it with your own AI (about a minute)

The judge is just a prompt. To grade a hook right now:

1. Open JUDGE.md and copy the whole thing.
2. Paste it into Claude, ChatGPT, or your own agent.
3. Hand it your hooks.

That's it — you're now running the eval. (This is the fastest way to get what an eval is: you just became the harness.)

Run it at scale

To grade a batch with your own API key:

# one hook
node grade.mjs "Your dad's wallet is the reason your back hurts."

# a file, one hook per line
node grade.mjs --file my-hooks.txt

Set ANTHROPIC_API_KEY or OPENAI_API_KEY in your environment and it uses whichever it finds. Bring your own key, grade as many as you want.

Verify it yourself

Don't take the number on faith — reproduce it:

node eval-harness.mjs

This runs the judge against all 116 of Alysha's graded hooks and prints how often it agreed with her (~74% when we ran it with a Sonnet-class judge — the exact number depends on which model you point it at). That's the eval grading itself.

Who graded the answer key

Every verdict in eval-set.json is a real call by Alysha — a world-class creative strategist.

• X: @alyshaboehmm
• GitHub: alyshafrommotion

Study the set on its own and you'll learn as much as any hook-writing course: eval-set.md is the browsable version.

What's inside

File	What it is
JUDGE.md	The rubric — the judge itself. Paste it into any AI.
eval-set.json · eval-set.md	The answer key — 116 hooks graded by a world-class strategist.
METHODOLOGY.md	How the eval works: blind grading, consensus, how the answer key calibrates the judge.
grade.mjs	Grade your own hooks at scale (bring your own API key).
eval-harness.mjs	Reproduce the agreement score against Alysha's labels.

License

MIT — see LICENSE. Use it, fork it, build your own eval on top of it.

Built by the team behind Runneth.