Repository navigation
Model trials
What we have tried in place of Claude, what it cost and what came of it. The tools are on Model providers; this page is the record, newest first. Nothing here changed a default: Pass A is still Haiku, Pass B still Sonnet.
Every trial reuses a scored, reviewed stream's own windows, summary and prompt, so only the
model differs (clippa compare score). The columns below are clippa compare list's:
| Column | Meaning |
|---|---|
| approved | moments approved in review that the candidate picked |
| same | windows given the same verdict as the stream's own Haiku run |
| kept | windows the candidate kept (Haiku keeps about one in eight) |
| rej>kept | windows Haiku rejected that the candidate kept: the guard against other creators' audio |
| err | calls that failed or came back malformed |
The approved count is noisy. Haiku scoring the same three streams again with its own prompt
picked only 7 of the 18 moments it picked the first time, so 7 of 18 is where "as good as
Haiku" sits, and the picks nobody reviewed can't count either way. Read same, kept and
rej>kept alongside it.
score_a_v8 is a restructured v7: headed sections (role, input, speaker tags, selection,
the decision order, hard rejections, examples, output rules) and numbered output rules. It
keeps everything v7 decides on: the per-show gambling and third-party rules, reading aloud as
the show's own audio, the order payoff, context, quality, the four examples, crude is not a
reason to reject, and about one window in eight. A kept clip scores 5-10. The rule files are
copies of v7, so only the main prompt differs. Config: kino ran v8 for a week, then c404 moved to it too (see below); base stays v5.
| Haiku, prompt | approved (of 18) | same (of 194) | kept | rej>kept | labels right (of 66) | keep-or-not | Cost |
|---|---|---|---|---|---|---|---|
| v5 | 7 (a plain rerun) | 142 | 25 | 7 | 26 | 35 | $0.34 + $0.07 |
| v6 | 7 | 116 | 47 | 15 | 28 | 37 | $0.42 + $0.08 |
| v7 | 6 (2+3+1) | 112 | 92 | 37 | 39 (35 on a rerun) | 47 (42) | $0.55 + $0.09 |
| v8 | 8 (3+1+4) | 125 | 62 | 27 | 36 | 46 | $0.51 + $0.09 |
Cost is the three streams plus one labelled run. On the streams v8 picks the most approved moments of any prompt so far and keeps a third fewer windows than v7, with fewer of the ones Haiku's original run threw out. On the labels it is within v7's own rerun spread (35-39), it kept nothing against a hard rule (v7 kept 1), and its keeps are the most precise (88%). Its main miss: 13 of the 45 keeps called needs-context (v7: 8), so the context step still bites too hard. Neither gap is bigger than a rerun's, so v8 is a candidate, not a winner yet: the next test is a week of live streams on one show.
Kino ran on v8 from 11-09. On its three reviewed streams since (11-12, 11-14, 11-17) each was
rescored on v7 with compare score --prompt-version v7 (Haiku Pass A, Sonnet Pass B, about
$0.3 by the estimate, 218 windows). Of the moments you approved, v8 picked 21 (7+9+5) and v7
picked 9 (4+3+2). v7 added 13 picks of its own that nobody reviewed, so it can't be scored
either way: reviews only cover what v8 surfaced, so this shows how many approved clips v7
found, not how many good ones v8 missed. Where both picked the same moment the titles are
nearly the same. v8 was kept on Kino and Channel 404 moved to it on 11-18 (user). The base
[scoring] stays on v5.
On the same three Kino streams (Haiku Pass A, Sonnet Pass B, $0.67) v9 picked 9 of the 21
approved moments, as v7 did. A plain v8 rerun (compare score --repeat, $0.55) picked 10,
so the 21-vs-9 gap was review bias (only the original run's picks were reviewed) and rerun
noise, not v7 or v9 picking worse. v9's Pass A matched the original run on 79% of windows,
the v8 rerun on 74%. On the 66 labelled windows v9 got 36 labels right and 45 keep-or-not,
v8 37 and 46 (fresh runs the same day): v9 finds a few more keeps (64% recall against 60%)
less precisely (85% against 93%) and calls a keep needs-context less often (9 against 13).
Nothing here beats a rerun's spread, so v8 stays on kino and c404 (user, 11-20); v9 stays in
the package for a later trial. The lesson for the next prompt: compare a candidate with a
--repeat of the current one, never with the run that was reviewed.
score_a_v9 is v8 with two changes. why is the first field of the answer (from v9 on the
Pass A schema puts it first; models write in schema order), so the model reasons through
payoff, context and quality before it gives a verdict or a cut, rather than justifying one.
Claude and OpenAI keep schema order; a provider on JSON mode, or Gemini, may not, so try v9
on Claude. And the prompt says where the cuts may go: a cut may start in the lead-in when the
setup is said there, and ends by the window's last timestamp (the model sees nothing after
it). The rule files are copies of v8.
OpenRouter's plain deepseek/deepseek-v4.1-flash still points at the 2026-09-10 snapshot;
the newer one is only reached as deepseek/deepseek-v4.1-flash-1017, at $0.15 / $0.60 per 1M
tokens (the 09-10 snapshot is $0.045 / $0.50). Pinning a dated slug is the only way to know
which one ran.
Four tests, $1.98 in all. Not adopted: it picks as many approved moments as Haiku only by keeping five times as many windows, it misses most of what a hard rule should drop, and its reasoning makes it worse.
On three reviewed streams (compare score), the same three as the 11-05 runs, prompt v6,
Sonnet Pass B:
| Run | approved (of 18) | same (of 194) | kept | rej>kept | err | Cost | Time a stream |
|---|---|---|---|---|---|---|---|
| Haiku v5, a plain rerun | 7 | 142 | 25 | 7 | 0 | $0.34 | ~2 min |
| Haiku v6 | 7 | 116 | 47 | 15 | 0 | $0.42 | ~2 min |
| V4.1 Flash 10-17 v6 @none | 7 (3+2+2) | 69 | 125 | 57 | 0 | $0.43 | ~1.5 min |
| V4.1 Flash 10-17 v6 @medium | 2 (2+0+0) | 76 | 128 | 53 | 0 | $0.73 | ~6 min |
| V4.1 Flash 09-10 v6 @none | 4 (1+2+1) | 66 | 102 | 42 | 7 | $0.41 | ~1.5 min |
The 10-17 snapshot fixes the 09-10 one's errors. Reasoning on cost 1.7 times as much, took
four times as long and picked fewer approved moments. Reports, on kick-135553454,
kick-134467434 and kick-134812118:
<stream>/compare/score_openrouter-deepseek-deepseek-v4.1-flash-1017-<effort>-v6_a.md.
Against the labelled windows (models gold run, 66 windows from 12 streams, reasoning off).
Every row is a fresh run, so Haiku is graded here like the others:
| Model | labels right, v5 / v7 | keep-or-not, v5 / v7 | Cost a run |
|---|---|---|---|
| Haiku | 26 / 39 | 35 / 47 | $0.07-0.09 |
| DeepSeek V4 Pro | 46 / 48 | 48 / 51 | $0.22-0.29 |
| Qwen 3.8 Flash | 44 / 40 | 50 / 47 | $0.03 |
| V4.1 Flash 10-17 | 45 / 43 | 48 / 48 | $0.03-0.04 |
It finds 87% of the keeps but almost none of the rejects: 0 of 6 needs-context windows, 3-4 of 9 other creators' audio, and it kept 4 windows against a hard rule. Haiku's trouble is the opposite: it throws keeps away.
The built-in checks (models test, $0.016): JSON, vision, the house-style title and the
~9 h transcript all pass. The Pass A check fails on one thing only: all 12 verdicts were right,
but the clip it cut for a keeper was 28 s, under the 30 s minimum. That rules it out of
scoring.model_a and leaves the other 24 stages.
The decision-model lab's test (lab/run.py brief, all 1,341 windows, $0.73): the same 20
yes/no questions Jev answered, asked as one structured call. DeepSeek answers 0 or 1 nearly
every time instead of a probability. It said yes to "has a clip" on 74% of windows and flagged
casino gameplay on 5, where Haiku flagged 31. Its best question ranks windows at 0.65 AUC
against Haiku's keeps (Jev 0.77 on its best one, 0.83 combined), and combining its answers
does no better than chance (0.49). The lab ran from main's code with two small changes on
branch jev-lab: run.py --effort, and decider.py working without the Jev provider.
score_a_v6 adds an order of decisions (who carries the payoff, every reference explained,
then the score), says played clips and words read aloud are source, and aims at about one
window in eight. On the same 18 approved moments: Haiku v6 7, Qwen 3.8 Flash 6, GPT-6 Luna 5,
DeepSeek V4 Pro 4, V4.1 Flash 4, MiniMax 4, all at none. v6 loosened Haiku (12-19 windows
kept against v5's 7-11) without picking more, so the config stays on v5.
Twelve models, reasoning none and medium, as Pass A on kick-135553454 with Sonnet Pass B,
then the best on two more streams. Gemini 3.5 Flash Lite @none picked all 5 approved moments on
the first stream but 1 each on the other two. Argon agreed most with Haiku (36-38 of 60-67
verdicts) and kept fewest of its rejects, at about $0.60-0.80 a stream for Pass A. Every other
model kept 40-60 windows. Reasoning on cost more and did no better: Qwen 3.8 Flash @medium took
43 minutes a stream, and DeepSeek V4 Pro at high spent all 16,000 of Pass B's thinking tokens
before answering. All compare runs that week: $18.64.
TypeSafe's jev-1.14.0 answers typed yes/no questions for $0.042 per 1M input tokens, output
free. Ten side tests on branch jev-lab (rerun steps in lab/TESTS.md there) against 1,341
Pass A windows from 21 streams, Haiku's keeps, Pass B's picks and 7-day views. $0.89 for six
runs, 21.2M input tokens; Haiku's Pass A on the same windows was $1.28.
| Test | Result |
|---|---|
| Rank windows like Haiku (AUC vs Haiku's keeps) | best 0.83, with the show's full brief and 15 small questions combined in code; 0.72 without the brief |
| Stand in for Pass B (picks among keeps) | ~0.55, Haiku's own score 0.67 |
| Predict views (Spearman) | 0.04-0.28, Haiku's score 0.57 |
| Factual flags | casino gameplay 0.98, other creators' audio 0.86, needs context 0.70 |
| Reject the bottom 20% before Haiku | 99.6% agree with Haiku, no picks lost, but it saves about what Jev costs |
| Screen archive pieces for facts | best 0.72; keeping 95% of the facts still sends 89% of the words |
| Check finished Shorts before upload | every question near a coin flip (0.44-0.55) |
Good at concrete flags and at knowing when it is sure; no taste. Nothing of it went into the pipeline. What did: the free other-voices hold (Shorts with over 5 s of voices that aren't the show's cast arrive held, PR #146, see Speakers), which beat every Jev question on the one thing worth checking before upload.
clippa compare score <stream> --model openrouter:<vendor>/<model> --prompt-version v6 --effort none --dry-run
Use the three streams above so the tables line up, pin a dated slug, and run clippa models test first: it catches a model that can't follow the format for a few cents.
Home · Getting started · clippa doctor · These pages are built from docs/wiki/ in the repo, so edits made here are overwritten: how to edit them
Run it
- Overview
- Getting started
- Roles and services
- Configuration
- Pipeline
- Review
- Reviewing
- War room
- Clippa messenger
- Scheduling and uploads
- Costs
- Checkup
- Audit
- Backups
Install
- Install
- Install checklist
- Kitsu
- Discord app
- YouTube sign-in
- Windows
- macOS
- Transcribe on Modal
- NAS recorder
- NAS hub
- Claude on the NAS
- Phone access
- Model providers
- Adding a show
- YouTube channel
Features
- Pre-shows
- Fetching VODs
- Speakers
- Faces
- Covers
- Cast library
- B-roll
- Clip art
- Weekly issue
- X posts
- TikTok
- News watcher
- Character pieces
- Documentaries
Research
Working on Clippa