-
Notifications
You must be signed in to change notification settings - Fork 0
Braintrust Experiment Log
Experiment ID: main-1785257772
Link: https://www.braintrust.dev/app/DSHB_amfam_capstone_2026/p/AMFAM-Doc-Classification/experiments/main-1785257772
Dataset: fixed_size_sampled/images/ (1024×1024 padded PNGs)
Prompt: CLASSIFICATION_PROMPT from src/openrouter_classifier.py
Settings: max_tokens=500, temperature=0.1
image size: 1024×1024
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 70.00% (112/160 correct) |
| Prompt tokens (avg) | 1,768.89 |
| Prompt cached tokens (avg) | 872.80 |
| Completion tokens (avg) | 171.51 |
| Completion reasoning tokens (avg) | 156.02 |
| Total tokens (avg) | 1,940.41 |
| Duration (avg) | 106.74s |
| LLM duration (avg) | 10.08s |
| Time to first token (avg) | 10.06s |
| Errors | 0 |
- First run (main-1785255418) scored 0% due to
max_tokens=20— reasoning model used all tokens on hidden reasoning, leaving nothing for output. - Bumping to
max_tokens=500resolved the issue. - ~156 of ~171 completion tokens are hidden reasoning tokens (not visible in output).
- 48 misclassifications need investigation in Braintrust UI to identify confused class pairs.
- Images resized to 1024×1024 (vs original full-res) significantly reduces prompt tokens (1,769 vs 11,996 in original test).
Pricing: $0.30/M input tokens, $15.00/M output tokens
| Images | Prompt Tokens | Completion Tokens | Total Tokens | Estimated Cost |
|---|---|---|---|---|
| 800 | 1,415,112 | 137,208 | 1,552,320 | $2.48 |
| 25,000 | 44,222,250 | 4,287,750 | 48,510,000 | $64.74 |
| 320,000 | 566,044,800 | 54,883,200 | 620,928,000 | $823.57 |
vs. Original Kimi K3 estimate (full-res images, max_tokens=20): $1,257.98 for 320K — resizing to 1024×1024 saves ~$434.
Experiment ID: main-1785265188
Link: https://www.braintrust.dev/app/DSHB_amfam_capstone_2026/p/AMFAM-Doc-Classification/experiments/main-1785265188
Dataset: fixed_size_sampled/images/ (1024×1024 padded PNGs)
Prompt: CLASSIFICATION_PROMPT from src/openrouter_classifier.py
Settings: max_tokens=1024, temperature=0.1, reasoning.effort=medium
Image size: 1024×1024
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 74.38% (119/160 correct) |
| Prompt tokens (avg) | 1,637 |
| Prompt cached tokens (avg) | 8.17 |
| Completion tokens (avg) | 415.64 |
| Completion reasoning tokens (avg) | 413.82 |
| Total tokens (avg) | 2,052.64 |
| Duration (avg) | 48.10s |
| LLM duration (avg) | 4.60s |
| Time to first token (avg) | 4.58s |
| Errors | 0 |
- +4.38% accuracy over Kimi K3 (74.38% vs 70.00%) with 13 improvements and 6 regressions.
- Reasoning tokens are visible in Braintrust UI metadata for debugging misclassifications.
- 2.2x faster than Kimi K3 (48s vs 107s avg duration per image).
- More completion tokens (416 vs 172) due to visible reasoning chain, but cheaper output pricing.
- 41 misclassifications to investigate in Braintrust UI.
Pricing: $0.15/M input tokens, $0.60/M output tokens
| Images | Prompt Tokens | Completion Tokens | Total Tokens | Estimated Cost |
|---|---|---|---|---|
| 800 | 1,309,600 | 332,512 | 1,642,112 | $0.40 |
| 25,000 | 40,925,000 | 10,391,000 | 51,316,000 | $12.37 |
| 320,000 | 523,840,000 | 133,004,800 | 656,844,800 | $158.38 |
vs. Kimi K3 at 320K: $158 vs $824 — Gemini 2.5 Flash is ~5.2x cheaper with higher accuracy.
Experiment ID: main-1785270444
Dataset: 2550x3300_10perclass_160/images/ (2550×3300 padded PNGs, 300 DPI)
Prompt: CLASSIFICATION_PROMPT from src/openrouter_classifier.py
Settings: max_tokens=1024, temperature=0.1, reasoning.effort=medium
Image size: 2550×3300
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 73.75% (118/160 correct) |
| Prompt tokens (avg) | 3,701.00 |
| Prompt cached tokens (avg) | 0.00 |
| Completion tokens (avg) | 418.19 |
| Total tokens (avg) | 4,119.19 |
| Duration (avg) | 0.00s |
| Errors | 0 |
- Switching from 1024×1024 to full-res 2550×3300 doubled prompt tokens (1,637 → 3,701) but accuracy dropped slightly (74.38% → 73.75%), suggesting higher resolution does not help and may add noise.
- Cost per image roughly doubled vs 1024×1024 ($0.64 vs $0.40 for 800 images) with no accuracy gain — 1024×1024 is the better cost/performance tradeoff.
Pricing: $0.15/M input tokens, $0.6/M output tokens
| Images | Prompt Tokens | Completion Tokens | Total Tokens | Estimated Cost |
|---|---|---|---|---|
| 800 | 2,960,800 | 334,555 | 3,295,355 | $0.64 |
| 25,000 | 92,525,000 | 10,454,843 | 102,979,843 | $20.15 |
| 320,000 | 1,184,320,000 | 133,822,000 | 1,318,142,000 | $257.94 |
Experiment ID: main-1785272634
Dataset: 2550x3300_50perclass_800/images/ (2550×3300 padded PNGs, 300 DPI)
Prompt: CLASSIFICATION_PROMPT from src/openrouter_classifier.py (baseline, pre-disambiguation)
Settings: max_tokens=1024, temperature=0.1, reasoning.effort=medium
Image size: 2550×3300
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 72.88% (583/800 correct) |
| Prompt tokens (avg) | 3,701.00 |
| Prompt cached tokens (avg) | 0.00 |
| Completion tokens (avg) | 417.38 |
| Total tokens (avg) | 4,118.38 |
| Duration (avg) | 0.00s |
| Errors | 0 |
- Scaling from 160 → 800 images held steady at ~73% accuracy, confirming the 10-per-class sample is representative.
-
form(30%),presentation(36%), andspecification(46%) are the worst-performing classes —scientific_reportis massively over-predicted with 64 false positives across 7 classes. - Reasoning trace analysis revealed the model confuses fax cover sheets with memos, specs with scientific reports, and press releases with news articles — leading to the disambiguation prompt update.
- See
docs/50pic_cost_est_tst.mdfor full per-class accuracy breakdown and confused pairs analysis. - See
docs/confusion_matrix_main-1785272634.mdfor confusion matrix. - See
docs/misclassification_reasoning_main-1785270444.mdfor reasoning trace analysis.
Pricing: $0.15/M input tokens, $0.6/M output tokens
| Images | Prompt Tokens | Completion Tokens | Total Tokens | Estimated Cost |
|---|---|---|---|---|
| 800 | 2,960,800 | 333,902 | 3,294,701 | $0.64 |
| 25,000 | 92,525,000 | 10,434,437 | 102,959,437 | $20.14 |
| 320,000 | 1,184,320,000 | 133,560,800 | 1,317,880,799 | $257.78 |
Experiment ID: main-1785277280
Dataset: 2550×3300 padded PNGs, 300 DPI
Prompt: CLASSIFICATION_PROMPT from src/openrouter_classifier.py
Settings: max_tokens=1024, temperature=0.1, reasoning.effort=medium
Image size: 2550×3300
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 83.75% (134/160 correct) |
| Prompt tokens (avg) | 4,368.00 |
| Prompt cached tokens (avg) | 0.00 |
| Completion tokens (avg) | 412.39 |
| Total tokens (avg) | 4,780.39 |
| Duration (avg) | 0.00s |
| Errors | 0 |
Pricing: $0.15/M input tokens, $0.6/M output tokens
| Images | Prompt Tokens | Completion Tokens | Total Tokens | Estimated Cost |
|---|---|---|---|---|
| 800 | 3,494,400 | 329,915 | 3,824,315 | $0.72 |
| 25,000 | 109,200,000 | 10,309,843 | 119,509,843 | $22.57 |
| 320,000 | 1,397,760,000 | 131,966,000 | 1,529,726,000 | $288.84 |
- +10% accuracy jump (73.75% → 83.75%) from adding disambiguation rules to the prompt — largest single improvement across all experiments.
- Prompt tokens increased from 3,701 → 4,368 due to longer prompt with disambiguation rules, but the accuracy gain far outweighs the ~$0.08 cost increase per 800 images.
- Only 26 errors remain (down from 42). Top confusion is now
presentation → file_folder(4 errors) where cover/divider pages get misclassified. - See
docs/confusion_matrix_main-1785277280.mdanddocs/misclassification_reasoning_main-1785277280.mdfor detailed breakdown.
Prompt v10 is the full v9 ruleset plus new disambiguation rules covering every v9 miss AND every v8.5→v9 regression:
- questionnaire vs presentation (v9 misses: 2), questionnaire vs handwritten, resume vs scientific_report, scientific_report vs budget/email/form/presentation, memo vs letter, budget vs invoice, handwritten vs form, advertisement vs news_article.
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 100% (14/14 correct) |
All 13 prompt-related v9 misses now correct. The 14th case
(news_article→scientific_publication, test_imagesr_r_c_s_rcs96d00_...) was a
dataset mislabel — an American Journal of Epidemiology reprint (Vol. 119, No. 4,
1984) — ground truth flipped to scientific_publication; v10 predicts it
correctly.
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 97.5% (154/158 correct) |
Per-class (of ~10 per class; 2 rows without a stored attachment are skipped):
| Class | Correct/Total | Class | Correct/Total | |
|---|---|---|---|---|
| advertisement | 9/9 | news_article | 9/9 | |
| budget | 9/10 | presentation | 10/10 | |
| 9/9 | questionnaire | 10/10 | ||
| file_folder | 10/10 | resume | 10/10 | |
| form | 10/10 | scientific_publication | 11/11 | |
| handwritten | 10/10 | scientific_report | 10/10 | |
| invoice | 7/10 | specification | 10/10 | |
| letter | 10/10 | memo | 10/10 |
Remaining 4 misses (all invoice ↔ budget):
-
qia17d00— Tobacco Institute check + detachable "INVOICE" stub (state campaign contribution): predicted invoice, expected budget. -
dav40c00,wce83f00,ynj47c00— "ESTIMATE CHANGE ORDER" / "PRODUCTION ESTIMATE REPORT" / "NEWSPAPER ESTIMATE RECAP": predicted budget, expected invoice.
No memo↔invoice confusion in the final run (memo 10/10, letter 10/10). The residual error class is agency/vendor estimate-vs-bill (budget vs invoice).
- Experiments
qwen3.7-flash_v10_smokeandqwen3.7-flash_v10_reasoningon Braintrust.
All runs: fixed_size_sampled (160-image balanced sample, ~10 per class × 16),
reasoning enabled, exact_match tracked. 2 of 160 rows lack a stored attachment
and are skipped, so the scored total is 158.
| Experiment | Model | Prompt | Accuracy |
|---|---|---|---|
qwen3.7-flash_v1_reasoning |
qwen/qwen3.7-flash | v1 | 50.0% (11/22)¹ |
qwen3.7-flash_v1_reasoning-5718f5da |
qwen/qwen3.7-flash | v1 | 80.1% (125/156) |
qwen3.7-flash_v2_reasoning |
qwen/qwen3.7-flash | v2 | 78.5% (124/158) |
qwen3.7-flash_v3_reasoning-14553e3a |
qwen/qwen3.7-flash | v3 | 78.5% (124/158) |
qwen3.7-flash_v4_reasoning |
qwen/qwen3.7-flash | v4 | 83.5% (132/158) |
qwen3.7-flash_v5_reasoning |
qwen/qwen3.7-flash | v5 | 84.8% (134/158) |
qwen3.7-flash_v6_reasoning |
qwen/qwen3.7-flash | v6 | 83.9% (130/155) |
qwen3.7-flash_v7_reasoning |
qwen/qwen3.7-flash | v7 | 91.1% (144/158) |
qwen3.7-flash_v8_reasoning |
qwen/qwen3.7-flash | v8 | 91.8% (145/158) |
qwen3.7-flash_v8.5_reasoning |
qwen/qwen3.7-flash | v8.5 | 88.6% (140/158) |
qwen3.7-flash_v9_reasoning |
qwen/qwen3.7-flash | v9 | 91.1% (144/158) |
qwen3.7-flash_v10_smoke |
qwen/qwen3.7-flash | v10 | 100% (14/14)² |
qwen3.7-flash_v10_reasoning |
qwen/qwen3.7-flash | v10 | 97.5% (154/158) |
qwen3.7-flash_v10_reasoning_320 |
qwen/qwen3.7-flash | v10 | 85.3% (272/319)³ |
qwen3.7-flash_v10_smoke_full |
qwen/qwen3.7-flash | v10 | 82.9% (180/217)⁴ |
qwen3.7-flash_v11_reasoning_160 |
qwen/qwen3.7-flash | v11 | **98.7% (156/158)**⁵ |
qwen3.7-flash_v11_reasoning_320 |
qwen/qwen3.7-flash | v11 | 83.9% (266/317)⁶ |
qwen3.7-flash_v11_smoke_full |
qwen/qwen3.7-flash | v11 | 87.8% (209/238)⁷ |
gemini-2.5-flash_v3_reasoning |
google/gemini-2.5-flash | v3 | 77.4% (123/159) |
gemini-2.5-flash_v4_reasoning |
google/gemini-2.5-flash | v4 | 77.7% (122/157) |
¹ First v1 run aborted early (22 scored rows). ² Smoke test on the 14 v9-miss images. ³ 320-image fixed-size set (20/class), one row unscored. ⁴ Full v10 smoke run on the 239-row misclassification set; run aborted at 217 scored rows (82.9%). ⁵ Full v11 run on the 160-image set (2 rows lack a stored attachment). ⁶ 320-image fixed-size set (20/class), 3 rows unscored. ⁷ Full v11 smoke run on the 239-row misclassification set; 1 row unscored.
11/22 (50.0%). Aborted early — only 22 rows scored (budget 5/9, invoice 3/9, letter 1/1, presentation 2/3).
125/156 (80.1%). Weakest classes: presentation 3/9, invoice 4/10, questionnaire 6/10, budget 6/10, form 6/10. Strong: email, file_folder, handwritten, letter, memo all 10/10.
124/158 (78.5%). Weakest: invoice 3/10, presentation 4/10, budget/form/ questionnaire 6/10. file_folder, email, handwritten 10/10.
124/158 (78.5%). Weakest: questionnaire 4/10, presentation 4/10, invoice 4/10, budget 6/10. Handwritten drops to 9/10.
132/158 (83.5%). +5.0 over v3. resume jumps to 10/10; questionnaire 8/10, presentation 7/9, invoice 7/10. form still 6/10, budget 6/10.
134/158 (84.8%). form 9/10, budget collapses to 3/10. invoice 6/10.
130/155 (83.9%). budget 4/10, invoice 4/10, scientific_report 7/10. form 9/10.
144/158 (91.1%). form 10/10, presentation 10/10, questionnaire 10/10, resume 10/10. handwritten drops to 6/10, scientific_report 7/10. (v7 added presentation cover/slide rules.)
145/158 (91.8%). Best qwen result before v10. memo 8/10, handwritten 8/10, scientific_report 7/10. (v8 added agency estimate change order + scratchpad deliberation.)
140/158 (88.6%). Regression on file_folder (3/10) — the v8.5 folder-tab rewrite over-corrected cover pages. memo 10/10, form 10/10.
144/158 (91.1%). Recovered file_folder (10/10) and invoice (10/10); regressed questionnaire (7/10) and scientific_report (6/10). (v9 added folder-tab and budget-vs-form rules.)
154/158 (97.5%). See section above. Only 4 misses remain, all invoice ↔ budget
agency estimate/billing ambiguities.
272/319 (85.3%) on the fixed_size_sampled_320 set (20 images/class; one row
unscored). The 320-image set is far noisier than the curated 158 — the same
v10 prompt that reaches 97.5% on the archive set lands at 85.3% here. Errors
spread across 24 distinct confusion pairs; no single class dominates. The
biggest buckets are letter → memo (5) and scientific_report → form (5) —
mostly TO:/FROM:/SUBJECT internal memos and filled analytical/QA data sheets
that the v7+ cascade deliberately routes away from letter/scientific_report —
plus budget → form (4), specification → form (4), and invoice → budget
(3). All 11 budget ↔ invoice failures are covered by the same agency
estimate / money-record / form-footer patterns that v11 (and v11.5) target.
180/217 (82.9%) on the 239-row misclassification smoke set
(qwen_misclassification_smoke_v1_v11 — one image per v1-v11 miss, annotated
with the source prompt version). The run was aborted before the final ~22 rows
scored. 37 errors across just 5 confusion pairs:
-
invoice → budget(20): dominated by the three agency-estimate documents v11 already fixes (dav40c00ESTIMATE CHANGE ORDER,wce83f00PRODUCTION ESTIMATE REPORT,ynj47c00NEWSPAPER ESTIMATE RECAP — repeated across source versions) plusjow70f00(B&W payment VOUCHER for a grant). All are agency/vendor billing documents routed to the budget "planning" branch because v10's wording lacks the estimate-vs-bill signals. -
news_article → scientific_publication(9): all the same image (rcs96d00, an Am J Epidemiol reprint with its own running head + volume + page + copyright), which is genuinely a published reprint under check 6; the smoke annotation flags it as a miss but the reasoning is defensible. -
budget → form(6): aPOLITICAL CAMPAIGN CONTRIBUTION REQUEST(usa07d00, repeated across versions) and aPRICE VALUE ESTIMATEStable (acy93e00) — money-only records whose form-layout footer ("FORM M01A", field/approval blocks) pulled check 10 ahead of check 7. -
budget → invoice(1):qia17d00, a check stub whose stub columns are headed "INVOICE DATE/NO/AMOUNT" — already fixed by v11's explicit stub rule. -
form → presentation(1): one-off.
The smoke set confirms the v10→v11 estimate-vs-bill gap is the dominant budget/ invoice failure mode and shows the same budget→form money-record trap the 320 run exposes.
123/159 (77.4%). Weakest: invoice 3/10, questionnaire 4/10, form 5/10, presentation 6/10, budget 6/10, resume 6/10. email/file_folder/handwritten/ memo/scientific_publication 10/10.
122/157 (77.7%). Weakest: form 4/10, scientific_report 5/10, news_article 6/10, advertisement 6/10, budget 6/10. file_folder/email/handwritten 10/10.
qwen3.7-flash with the same prompt family outperforms gemini-2.5-flash by a wide
margin once the prompt reaches v7+ (91-97% vs 77%). qwen3.7-flash's explicit
forced reasoning (reasoning.enabled + effort high) produces a structured
scratchpad that reliably follows the v7+ step cascade; gemini's medium-effort
reasoning is less controllable and does not hit the same disambiguation rules.
v11 = v10 + an estimate-vs-bill disambiguation for check 7 (FINANCIAL DOCUMENT), restoring the agency-estimate invoice coverage that v10 narrowed away:
-
Root cause: v10 rewrote the invoice bullet to "billing document for
COMPLETED work ... listing ACTUAL billable charges" and dropped the
agency-estimate-change-order worked example that v8/v8.5/v9 taught. That made
the model route three agency estimate documents to budget:
dav40c00("ESTIMATE CHANGE ORDER", Walt Klein),wce83f00("PRODUCTION ESTIMATE REPORT AC14", Ted Bates),ynj47c00("NEWSPAPER ESTIMATE RECAP - PNO5", Leo Burnett). All three were correct in v9 (invoice 10/10, budget 9/10). -
Fix: invoice bullet restores the v9 wording (agency/vendor ESTIMATE
documents with unit prices/amounts/totals are invoices) and adds concrete
bill signals ("EST NO", "REVISES EST", "PRIOR ADJUSTMENTS", "EST AMT LESS
C/D", original-vs-present estimate columns, "BILLING TYPE PROGRESSIVE"). The
budget future-planning carve-out is narrowed to a pure planning recap with no
billing apparatus (the
tqi16e00"OUTDOOR ESTIMATE RECAP" bus-shelter case stays budget) and now explicitly keeps check stubs budget even when the stub columns are headed "INVOICE DATE/NO/AMOUNT" (theqia17d00case). A 4th worked example teaches the estimate-change-order→invoice case. -
Expected effect: v10's 154/158 → 157/158 (fixes the 3 invoice→budget
misses) while holding the
tqi16e00andqia17d00budget cases.
Three full v11 runs (reasoning enabled, effort=high) on Braintrust:
qwen3.7-flash_v11_reasoning_160, qwen3.7-flash_v11_reasoning_320, and
qwen3.7-flash_v11_smoke_full. Full reports (report markdown, per-class chart,
confusion matrix PNG/markdown, misclassification reasoning traces) live in
reports/.
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 98.7% (156/158) |
| Class | Correct/Total | Class | Correct/Total | |
|---|---|---|---|---|
| advertisement | 9/9 | news_article | 9/9 | |
| budget | 9/10 | presentation | 10/10 | |
| 9/9 | questionnaire | 10/10 | ||
| file_folder | 10/10 | resume | 10/10 | |
| form | 10/10 | scientific_publication | 11/11 | |
| handwritten | 10/10 | scientific_report | 10/10 | |
| invoice | 9/10 | specification | 10/10 | |
| letter | 10/10 | memo | 10/10 |
v11 improves v10's 154/158 → 156/158 (+1.3pp) and cuts the residual
invoice ↔ budget errors from 4 to 2. Only 2 misses remain, both in the
agency-estimate/billing space v11 targets:
-
jow70f00— invoice → budget: a "BROWN & WILL FARSON TOBACCO CORPORATION VOUCHER" for a $278,000 "Final contribution for grant" to The Franklin Institute. The model routes it to budget as an internal contribution/disbursement record ("money-only records ... whose whole content is an amount" → budget). This is the one voucher case v11 does NOT fix — the grant payment is genuinely ambiguous between a vendor bill and an internal disbursement. -
tqi16e00— budget → invoice: the "OUTDOOR ESTIMATE RECAP - ODO1(S)" bus-shelter planning recap that v11's carve-out explicitly keeps budget. The model still triggers the invoice rule on the "EST NO: 4155" line and the GROSS AMOUNT / GROSS CHANGE columns, even though it only plans future placements. The carve-out text needs to be stronger against an estimate number + comparison columns on an otherwise planning-only recap.
No other classes regressed — letter, memo, file_folder, presentation, questionnaire all hold at 10/10.
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 83.9% (266/317) |
Per-class (20/class; 3 news_article rows unscored):
| Class | Correct/Total | Class | Correct/Total | |
|---|---|---|---|---|
| advertisement | 18/20 | news_article | 15/17 | |
| budget | 13/20 | presentation | 14/20 | |
| 20/20 | questionnaire | 17/20 | ||
| file_folder | 18/20 | resume | 20/20 | |
| form | 14/20 | scientific_publication | 18/20 | |
| handwritten | 19/20 | scientific_report | 16/20 | |
| invoice | 15/20 | specification | 15/20 | |
| letter | 15/20 | memo | 19/20 |
v11 on the noisy 320 set is 83.9% vs v10's 85.3% (−1.4pp) — a small regression on the same set, so the estimate-vs-bill tightening helps the curated archive set but does not transfer to the noisier 320 sample. Errors spread across many confusion pairs; the largest buckets (report filters 3 errored + 2 empty-output rows, 315 analyzed):
-
letter → memo(5): TO:/FROM:/SUBJECT internal memos and correspondence the v7+ cascade routes away from letter. -
specification → form(5),scientific_report → form(3),budget → form(4),invoice → form(3),presentation → form(3): filled analytical/QA data sheets, labeled charts/tables, and money records whose form-layout footer (fields, approval blocks) pulls check 10 ahead of checks 7/8/9/13.formremains the biggest over-prediction sink on this set. -
invoice ↔ budget(4 combined),budget → invoice(2), plus singles — the residual agency-estimate/billing ambiguities.
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 87.8% (209/238) |
Per-class (one image per v1–v11 miss; 1 row unscored, email absent):
| Class | Correct/Total | Class | Correct/Total | |
|---|---|---|---|---|
| advertisement | 10/10 | news_article | 6/16 | |
| budget | 29/39 | presentation | 25/25 | |
| file_folder | 7/7 | questionnaire | 24/24 | |
| form | 16/17 | resume | 7/7 | |
| handwritten | 13/13 | scientific_publication | 6/6 | |
| invoice | 34/40 | scientific_report | 22/22 | |
| letter | 2/4 | specification | 1/1 | |
| memo | 7/7 | — |
v11 smoke = 87.8% (209/238) vs v10 smoke 82.9% (180/217, aborted) — the estimate-vs-bill fix clears most of the invoice→budget bucket (20 → 6), the dominant v10 failure. Remaining 29 errors across 6 pairs:
-
news_article → scientific_publication(10): all thercs96d00Am J Epidemiol reprint (repeated across source versions). A genuine published reprint under check 6 — flagged as a miss by the smoke annotations, but the prediction is defensible. -
budget → form(6): money-only records (usa07d00POLITICAL CAMPAIGN CONTRIBUTION REQUEST,acy93e00PRICE VALUE ESTIMATES) whose form-layout footer pulls check 10 ahead of check 7 — the same money-record trap the 320 run exposes. -
invoice → budget(6): residual agency/vendor billing documents (dav40c00,wce83f00,ynj47c00still flip for some source versions) plusjow70f00(the B&W grant VOUCHER, also v11's one 160-set miss). -
budget → invoice(4): thetqi16e00OUTDOOR ESTIMATE RECAP carve-out case (also v11's one 160-set miss), plus check-stub cases. -
letter → memo(2),form → presentation(1): one-off residuals.
Net: v11 is the best qwen result on the curated archive set (98.7%), fixes
most of the estimate-vs-bill smoke bucket, and the two remaining 160-set misses
(jow70f00, tqi16e00) are precisely the edge cases the next prompt version
should target.
v11.7 = v11.6 + a deliberately minimal 3-edit set (D, A, B — C and E skipped to reduce regression risk):
- Edit D — check 7 structural split between voucher/check-stub (budget) and vendor billing with amounts/totals (invoice).
- Edit A — check 8 adds: a labeled product/parameter rate-data chart (rates/units per product or parameter) is a specification.
- Edit B — check 10 adds a standalone-chart carve-out: a financial/money chart with no other document signals is budget; an unlabeled standalone chart/table is a presentation slide.
| Run | Accuracy |
|---|---|
160-image qwen3.7-flash_v11_7_reasoning_160
|
98.1% (156/159) |
56-row eval qwen3.7-flash_v11_7_eval
|
35.7% (20/56) |
v11.7 is the first 16-class prompt to exceed 35% on the eval set (v11.5 = 16/56, v11.6 = 17/56). Its 3 misses on the 160 set:
-
yvp54d00— form → budget: a "MILWAUKEE ADVERTISING CLUB" document requesting authorization of $690,000; the budget money-only rule catches the bare figure. -
cpt85d00— letter → memo: correspondence to "Mr. T. E. Sandefur"; the by-name + no-TO:/FROM: pattern routes it to memo. -
tqi16e00— budget → invoice: the OUTDOOR ESTIMATE RECAP bus-shelter planning recap (also a v11 miss).
v11.8 = v11.7 + two targeted fixes for the 160-set's remaining misses:
-
Fix 1 — budget money-only clause narrowed to bare amount-only requests
only; a document carrying an explicit "AUTHORIZATION REQUEST" for funds is a
form (fixes
yvp54d00). -
Fix 2 — memo-vs-letter: a by-name memo requires an explicit INTERNAL
TITLE/DIVISION; a bare honorific + name external addressee is a letter
(fixes
cpt85d00).
| Run | Accuracy |
|---|---|
160-image qwen3.7-flash_v11_8_reasoning_160
|
99.4% (157/158) |
56-row eval qwen3.7-flash_v11_8_eval
|
32.1% (18/56) |
160-set: 157/158 (99.4%) — best ever. All three v11.7 misses fixed:
-
yvp54d00form→budget → form ✓ (AUTHORIZATION REQUEST $690,000) -
cpt85d00letter→memo → letter ✓ ("Mr. T. E. Sandefur") -
tqi16e00budget→invoice → budget ✓ (OUTDOOR ESTIMATE RECAP)
Invoice held at 10/10. The only remaining miss is jed71e00
(form → presentation).
56-row eval: 18/56 — regression vs v11.7's 20/56, but the two recovered rows confirm both fixes transfer:
- ✓
rvl_cdip__form__0005.png— form → invoice → form (Fix 1) - ✓
rvl_cdip__specification__0019.png— spec → form → spec
The four eval regressions (all downstream of Edit B's carve-out and Fix 1):
- ✗
rvl_cdip__presentation__0001.png,rvl_cdip__presentation__0011.png— presentation → budget: titled, designed deck charts routed to budget by Edit B's "financial/money chart is budget" rule (check 10) ahead of check 9. - ✗
rvl_cdip__specification__0017.png— spec → form. - ✗
jed71e00— form → presentation (the same image that is the 160-set's one miss).
5 letter → memo eval misses are unchanged (gold-labeled literal TO:/FROM:
memo-header letters; pre-existing since v11.5 — Fix 2 only targeted bare-name
external addressees).
v11.9 = v11.8 + three edits that narrow Edit B so titled/designed deck charts no longer fall into budget:
- Edit 1 — check 10 carve-out narrowed: a product/parameter rate-data chart → specification; a research/measurement chart → scientific_report; a financial/money chart → budget only when it is a standalone data table used for money planning or tracking; a titled, designed deck chart stays presentation (check 9).
- Edit 2 — check 9 hardened: a titled/designed deck chart IS a presentation slide; don't route it to budget.
- Edit 3 — calibration line: deck charts → presentation, product/parameter charts → specification.
| Run | Accuracy |
|---|---|
56-row eval qwen3.7-flash_v11_9_eval
|
35.7% (20/56) |
Ties v11.7's best eval score and is +2 over v11.8's 18/56. The two presentation→budget regressions introduced by Edit B in v11.8 are recovered, plus two bonus fixes:
- ✓
rvl_cdip__presentation__0001.png— presentation → budget → presentation - ✓
rvl_cdip__presentation__0011.png— presentation → budget → presentation - ✓
test_imagesj_j_e_d_jed71e00...— form → presentation → form (this is also the one remaining 160-set miss in v11.8) - ✓
rvl_cdip__questionnaire__0016.png— questionnaire → handwritten → questionnaire
Two rows regressed against v11.8:
- ✗
rvl_cdip__form__0005.png— form → invoice (a Fix-1 success in v11.8, lost in v11.9) - ✗
rvl_cdip__news_article__0008.png— news_article → memo (new miss; was correct in v11.7 and v11.8)
Three rows remain misses but shifted prediction: advertisement__0015
(form→handwritten), presentation__0013 (memo→letter),
scientific_report__0016 (specification→form). The 5 letter → memo eval
misses are unchanged. v11.9 recovers the Edit B presentation regression while
holding v11.8's 160-set fix, so the next eval benchmark is the 480-image run
(queued) to confirm the generalization holds.
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 87.2% (279/320) |
Per-class (20/class):
| Class | Correct/Total | Class | Correct/Total | |
|---|---|---|---|---|
| advertisement | 18/20 | news_article | 18/20 | |
| budget | 16/20 | presentation | 16/20 | |
| 20/20 | questionnaire | 17/20 | ||
| file_folder | 17/20 | resume | 20/20 | |
| form | 17/20 | scientific_publication | 17/20 | |
| handwritten | 18/20 | scientific_report | 16/20 | |
| invoice | 16/20 | specification | 18/20 | |
| letter | 15/20 | memo | 20/20 |
v11.8 on the noisy 320 set is 87.2% vs v11's 83.9% (+3.3pp) — strong generalization of the 160-set improvement. Biggest per-class gains:
- budget 13/20 → 16/20 (+3)
- form 14/20 → 17/20 (+3)
- specification 15/20 → 18/20 (+3)
- presentation 14/20 → 16/20 (+2)
- invoice 15/20 → 16/20 (+1), memo 19/20 → 20/20 (+1)
Small losses: file_folder 18/20 → 17/20 (−1), handwritten 19/20 → 18/20 (−1), scientific_publication 18/20 → 17/20 (−1).
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 89.1% (424/476) |
Per-class (30/class; 4 rows unscored):
| Class | Correct/Total | Class | Correct/Total | |
|---|---|---|---|---|
| advertisement | 26/29 | news_article | 27/29 | |
| budget | 27/30 | presentation | 25/30 | |
| 28/29 | questionnaire | 28/30 | ||
| file_folder | 26/30 | resume | 29/32 | |
| form | 27/30 | scientific_publication | 27/30 | |
| handwritten | 27/30 | scientific_report | 24/30 | |
| invoice | 24/30 | specification | 27/30 | |
| letter | 23/30 | memo | 29/30 |
This is the first run on the 480 set (no v11 baseline). The 89.1% score holds up well given the larger, noisier sample. Biggest miss buckets:
- letter → memo (7): the persistent v11.5+ TO:/FROM: memo-header issue
- invoice → form (4), scientific_report → form (3): form-layout pull
- budget → invoice (3), invoice → budget (2): residual estimate/billing
- advertisement → presentation (3): labeled chart routing
- presentation → scientific_report (2), presentation → file_folder (2): deck-chart misrouting
- Built
fixed_size_sampled_v2: a fresh balanced 160-image slice fromjordyvl/rvl_cdip_100_examples_per_class, sampled with seed1738(10 images per class, 1024x1024 padded grayscale PNGs at 300 DPI). The slice has 160 unique filenames and zero pixel-hash overlap with bothfixed_size_sampledandfixed_size_sampled_480. - Built and uploaded
qwen_v115_v12_eval: the deduplicated union of failed rows from the v11.5, v11.6, v11.7, v11.7 reasoning, v12, and v12 reasoning evaluations. It contains 52 unique rows; each row retains the expected class, prior predictions, source versions, and capped reasoning traces. - Standing prompt requirement: future prompt changes must address the recurring
scientific_publication/scientific_reportconfusions, especially specialist periodical pages misread asnews_articleand research-lab records misread asform, while preserving v11.9's financial-chart and correspondence safeguards. - Added
PROMPT_V13as a focused derivative of v11.9. It broadensscientific_publicationto specialist science/medical/technical periodicals, routes identifiable research-lab records toscientific_reportrather than genericform, and keeps product documentation inspecification. It also narrows future-spend budget handling and preserves the explicit internal title/division requirement for by-name memos. - Queued
qwen3.7-flash_v11_9_reasoning_160_v2onfixed_size_sampled_v2, serialized behind the in-flight v11.8 320-image run. After it completes, the queue runsqwen3.7-flash_v13_reasoning_160onfixed_size_sampledandqwen3.7-flash_v13_hard_evalonqwen_v115_v12_eval; result logs arereports/eval_160_v2_v11_9.log,reports/eval_160_v13.log, andreports/eval_hard_v13.log.
Three additional v11.8 runs on the original 160-image fixed_size_sampled slice plus one
retroactive run on the 52-row qwen_v12_retroactive_eval slice. All runs use the v11.8 prompt;
reasoning is set to each model family's maximum effort (qwen high, kimi xhigh, gemini max).
| Run | Dataset | Temp | Model | Accuracy |
|---|---|---|---|---|
qwen3.7-flash_v11_8_reasoning_160_t0_3 |
fixed_size_sampled | 0.3 | qwen3.7-flash | 98.7% (157/159) |
qwen3.5-35b-a3b_v11_8_reasoning_160 |
fixed_size_sampled | 0.1 | qwen3.5-35b-a3b | 98.7% (155/157) |
qwen3.5-35b-a3b_v11_8_reasoning_v12retro |
qwen_v12_retroactive_eval | 0.1 | qwen3.5-35b-a3b | 30.8% (16/52) |
kimi-k2.6_v11_8_reasoning_160 |
fixed_size_sampled | 0.1 | kimi-k2.6 | aborted (network outage) |
gemini-2.5-flash-lite_v11_8_reasoning_160 |
fixed_size_sampled | 0.2 | gemini-2.5-flash-lite | 86.9% (139/160) |
Baseline comparison: v11.8 at temp 0.1 scored 157/158 (99.4%) on this slice. At temp 0.3 the
run scores 157/159 (98.7%) with one row failing to produce content (wat19d00, no usable
output after retries). Both remaining misses are pre-existing v11.7-era rows:
- ✗
jed71e00— form → presentation (the long-standing deck-chart miss; also v11.8's only 160-set miss at temp 0.1) - ✗
tqi16e00— budget → invoice (OUTDOOR ESTIMATE RECAP; was fixed by v11.7/v11.8 at temp 0.1, regressed back at 0.3)
So temperature 0.3 costs one regression on the budget/invoice boundary without recovering
jed71e00; net −0.7pp vs temp 0.1 on scored rows.
First run of the hybrid-reasoning Qwen3.5-35B-A3B on this slice: 155/157 (98.7%), three rows
failed to produce usable content (mvr50f00, iby31c00, umv76d00 — Qwen3.5 burns long
reasoning traces; retries grew max_tokens to 16k but still capped). Misses:
- ✗
jed71e00— form → presentation (same recurring miss) - ✗
noz90d00— form → advertisement (new single error for this model)
On the 52-row qwen_v12_retroactive_eval slice (all rows previously misclassified by v12),
qwen3.5-35b-a3b with v11.8 scores 16/52 (30.8%). Five rows errored with
finish_reason=length (rvl_cdip__form__0005.png, rvl_cdip__invoice__0006.png,
rvl_cdip__presentation__0011.png, rvl_cdip__questionnaire__0005.png,
rvl_cdip__scientific_report__0016.png). Notable: this slice is hard by construction (every
row is a known v12 miss), so the low absolute score is expected; the top confusion buckets are
letter → memo (5), scientific_report → form (4), budget/invoice and file_folder/presentation
pairs (2 each).
Full per-run artifacts: reports/report_*_v11_8_*.md, reports/confusion_matrix_*_v11_8_*.{md,png},
reports/misclassification_reasoning_*_v11_8_*.md, reports/per_class_accuracy_*_v11_8_*.png.
Completed at 139/160 (86.9%), temperature 0.2, reasoning.effort=max. Notable: it is the
only one of the v11.8 runs with zero failed/empty rows. The miss profile is different from the
qwen family — heavy → specification pull (memo 3, form 2, handwritten/letter/scientific_report
1 each) plus scientific_publication → scientific_report (2) and budget → invoice (2). The
jed71e00 (form → presentation) miss that every qwen run made was NOT missed by gemini.
Run was aborted mid-flight (~109/160 completed) due to a transient DNS/network outage against
api.braintrust.dev that crashed the Braintrust logging thread (no usable result row). The
manifest (reports/manifests/eval_v11_8_kimi.jsonl) preserves the completed rows if a rerun is
wanted; the experiment in Braintrust is partial and should not be used for comparisons.
Experiment ID: qwen3.7-flash_v0_reasoning_480-d4c97ff0
Dataset: 2550×3300 padded PNGs, 300 DPI
Prompt: CLASSIFICATION_PROMPT from src/openrouter_classifier.py
Settings: max_tokens=1024, temperature=0.1, reasoning.effort=medium
Image size: 2550×3300
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 69.17% (332/480 correct) |
| Prompt tokens (avg) | 1,262.68 |
| Prompt cached tokens (avg) | 0.00 |
| Completion tokens (avg) | 3,027.66 |
| Total tokens (avg) | 4,290.33 |
| Duration (avg) | 0.00s |
| Errors | 0 |
Pricing: $0.15/M input tokens, $0.6/M output tokens
| Images | Prompt Tokens | Completion Tokens | Total Tokens | Estimated Cost |
|---|---|---|---|---|
| 800 | 1,010,140 | 2,422,125 | 3,432,266 | $1.60 |
| 25,000 | 31,566,898 | 75,691,435 | 107,258,333 | $50.15 |
| 320,000 | 404,056,296 | 968,850,370 | 1,372,906,666 | $641.92 |
Experiment ID: qwen3.7-flash_v0_reasoning_800-f0b6b2e4
Dataset: rvl_cdip_800, 1024×1024 grayscale padded PNGs
Prompt: v0 from src/prompts.py
Settings: max_tokens=8192, temperature=0.1, reasoning.effort=high
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 66.12% (529/800 correct) |
| Prompt tokens (avg) | 741.1 |
| Prompt cached tokens (avg) | 0.0 |
| Completion tokens (avg) | 1,552.1 |
| Total tokens (avg) | 2,293.1 |
| Errors | 0 |
Pricing: /bin/zsh.03/M input, /bin/zsh.13/M output. Expected /bin/zsh.1792 (list price × measured tokens); actual billed /bin/zsh.1773 (+1.1%).
| Images | Expected Cost | Estimated Actual |
|---|---|---|
| 800 | /bin/zsh.18 | /bin/zsh.18 |
| 25,000 | .60 | .54 |
| 320,000 | .68 | .92 |
Experiment ID: qwen3.7-flash_v11.8_reasoning_800-f6f4648b
Dataset: rvl_cdip_800, 1024×1024 grayscale padded PNGs
Prompt: v11.8 from src/prompts.py
Settings: max_tokens=8192, temperature=0.1, reasoning.effort=high
| Metric | Value |
|---|---|
| Accuracy (exact_match) | 83.12% (665/800 correct) |
| Prompt tokens (avg) | 11,986.0 |
| Prompt cached tokens (avg) | 7,325.6 |
| Completion tokens (avg) | 1,906.5 |
| Total tokens (avg) | 13,892.5 |
| Errors | 0 |
Pricing: /bin/zsh.03/M input (/bin/zsh.003/M cached), /bin/zsh.13/M output. Heavy prompt caching (~61% of prompt tokens cached). Expected /bin/zsh.3276 (cache-adjusted); actual billed /bin/zsh.3427 (+4.6%).
| Images | Expected Cost | Estimated Actual |
|---|---|---|
| 800 | /bin/zsh.33 | /bin/zsh.34 |
| 25,000 | .24 | .71 |
| 320,000 | .04 | .08 |
Experiment: qwen3.7-flash_v11.8_reasoning_1600_balanced_1120 — 1120 images (70 per class × 16 classes)
Model: qwen/qwen3.7-flash
Prompt: v11.8
Dataset: rvl_cdip_1600
| Metric | Value |
|---|---|
| exact_match | 925/1120 (82.6%) |
| Failure rate | 0.0% |
| Near-miss | 72 (36.9% of misses) |
| Expected cost | $0.6815 |
| Actual cost | $0.4937 |