18 datasets × 3 runs: 7 of 9 LLM failures are one string (; RNA-Seq), and DeepSeek pro beats flash on every axis

[lhqing]

A design review off three benchmark runs of the same 18-case corpus at ac11b44 — no-LLM, DeepSeek
flash, DeepSeek pro. The question was "which model should be the default", and the answer turned out
to be the least interesting thing measured: 7 of the 9 non-green outcomes across both LLM runs
trace to a single string, and a second, smaller mechanism accounts for the other 2. Neither is a
model-quality problem. Both are places where code treats compatible information as contradictory.

Related: #184 (the library_strategy-as-chemistry defect, with a no-LLM reproduction), #61 (thread 1,
deeper metadata resolution), #185 / #186 (GSE234962's coin flip), #182 (_without_short_readings),
#50 (a deterministic GSM-title parser).

TL;DR

• deepseek-v4-pro should be the default. It wins on every measured axis including the ones
  flash is supposed to win — 2.6× faster wall clock, 3.5× fewer output tokens, higher field
  accuracy, half the questions, 1 failed document vs 6. The "cheap end of V4 is what a corpus-scale
  harness should reach for" claim in five docstrings is falsified by its own benchmark.
• One string, seven failures. library.chemistry = "RNA-Seq" — SRA's library_strategy, read off
  the ; <organism>; <strategy> suffix that GEO appends to every experiment title — appears in 7 of
  the 9 non-green outcomes. It is not wrong; it is a family term being coerced onto a leaf.
• The KB already has the hierarchy that would make it harmless. Spec.parent,
  node_kind: Literal["family","leaf"], children_decided_by (kb/schema.py:318-322) — and
  _match_tech returns str | None, so it structurally cannot say "this names a family".
• The other two failures are attribute drops: two sources describing one thing in slightly
  different words, compared with case-folded exact equality, and the attribute nulled. The eval
  harness grades that null as false_accept — "a confident wrong manifest" — which is the harness
  disagreeing with the resolver's own stated rationale.
• The benchmark is not measuring the code that ships. manifest fill reduces chemistry claims
  agreement-or-nothing; evals reduces them last-wins. Two callers, one stage, different answers.

Method

Three runs of eval run over the same 18-case benchmark corpus at ac11b44, same prompt version
(2026.7.4), differing only in extractor:

	no-LLM	deepseek-v4-flash	deepseek-v4-pro
cases correct	18 / 18	12 / 18	15 / 18
field accuracy	1.000	0.955	0.982
false accept	0.000	0.056	0.056
false refuse	0.000	0.000	0.000
questions asked	1	6	3
wall clock	89.8 s	369.3 s	140.5 s
LLM calls	0	157	141
output tokens	—	328,857	93,716
input tokens	—	483,766 (98.1% cached)	605,250 (96.8% cached)
documents failed	—	6 / 141	1 / 141
harvests complete	—	13 / 18	17 / 18

Caveat, and it is not a small one: both LLM runs are trials: 1, so stability is unmeasured
(reported 0.0 throughout, against 1.0 for the deterministic run). GSE234962 is known to flip
(#185/#186). Single-trial LLM grades are therefore claims about one sample, not about the extractor.
Everything below that rests on a mechanism traced into source is solid; anything resting on a grade
alone is one draw.

The default-model question, settled

Flash is not cheaper here in any currency this repo actually measures (there is no dollar pricing
anywhere in the codebase — cost is tokens and seconds). It burns 3.5× the output tokens, and a
large share of that output is discarded by construction: 38 field_not_permitted_for_doc
rejections on PRJNA1027859, 17 on GSE256266, 16 on GSE234962, where the model answers with fields
the prompt did not ask of that document's scope. Pro does this 0–2 times per case.

So: flip DEEPSEEK_DEFAULT_MODEL (harvest/providers.py:50), and retire the motivating claim in
providers.py:49, cli/eval.py:158, cli/harvest.py:138, cli/run.py:299, evals/run.py:488.

One live bug falls out: the pro run's stderr advises "re-run those cases with --model deepseek-v4-pro" while already running pro. The remedy string does not know what model ran.

Mechanism 1 — a family term coerced onto a leaf (7 of 9)

Every GEO experiment title carries a fixed suffix:

GSM9294935: MC38 Tumor, Pro VSV GFP treated control, 3 injections, 5 combined samples; Mus musculus; RNA-Seq

fields.py:143 asks experiment-scope documents for library.chemistry. That document is this
title. The model answers with the only technology-shaped token in it, and four independent, each
individually-defensible things line up (#184 traced the same chain):

1. The draft is unfalsifiable by construction. verify.py:130-134 already says so about itself:
   "Entailment is vacuous when value ⊆ quote." The value is copied out of the quote, so
   span_verified and entailment_ok both pass trivially.
2. The stated guarantee is never enforced. verify.py:129 claims the check's power "comes
   entirely from value being drawn from a controlled vocabulary: for library.chemistry the value
   is a KB id." Nothing validates that. "RNA-Seq" is not a KB id.
3. _match_tech (escalate.py:697-713) matches by symmetric substring. Pass 3 accepts
   needle in alias or alias in needle and returns the first hit in dict order. "rna-seq" is a
   substring of bulk-rnaseq-pe's alias "bulk RNA-seq".
4. That leaf has no barcode role, so _bulk_asserted_single_cell_observed (escalate.py:510)
   fires conflict-bulk-asserted-single-cell-observed (escalate.py:540) against correctly-resolved
   barcoded bytes → exit 4.

The check is inverted on the one field it was built for

In GSE256266 and PRJNA1027859 the model also proposed library.chemistry = "10x-3p-gex-v3.1" —
the correct KB id — and it was rejected not_entailed, while "RNA-Seq" was accepted with
entailment_ok: true. The tripwire rejects the right answer (prose spells 3' as 30) and admits
the wrong one (vacuity). That is worth saying plainly: on library.chemistry, entailment currently
has negative value.

It is not only spurious questions

GSE317744 is the sharpest case. Its expected.yaml pre-registers that 10x-5p-gex-v2 is
byte-identical to 3' v2 in every observable — distinguishable_by: [metadata, alignment], no byte
path at any rung — so the case exists to grade the metadata-hypothesis branch. Under flash,
"RNA-Seq" occupied that channel and the run resolved 10x-3p-gex-v2, i.e. the wrong chemistry,
and would have emitted soloStrand Forward for a Reverse library. So the same string produces both a
spurious refusal and, where metadata is the only evidence, a wrong answer.

The hierarchy is already there

#184 notes the loose substring direction "is not a one-line deletion" because family claims like
"10x 5'" depend on it. That blocker dissolves, because the loose match is a workaround for a
missing family node:

# kb/specs/10x-3p-gex/spec.yaml          version: family, no backend:   ← family node
# kb/specs/10x-3p-gex-v3/spec.yaml       parent: 10x-3p-gex             ← leaf
# kb/specs/bulk-rnaseq-pe/spec.yaml      LEAF, aliases: ["bulk RNA-seq", ...]
#                                        assay_ontology: ["EFO:0008896"]  # "RNA-Seq" (generic bulk)

bulk-rnaseq-pe is a leaf holding the generic EFO term, and there is no rna-seq family node
above it. Meanwhile confuse.same_family (confuse.py:203) already walks the tree, and
engine.py:180-182 deliberately hands escalate the full KB "so id/confusable lookups resolve for
unscored nodes". The machinery is in place and _match_tech is the one function that ignores it —
legibly, in its type: -> str | None cannot express "this is 10x-3p-gex, decide the leaf from
bytes". same_family is consulted downstream (escalate.py:417, :594) as a repair on a decision
that should never have been forced.

Proposed:

• Add a root rna-seq family node (node_kind: family, no backend, EFO:0008896, aliases
  RNA-Seq / RNA sequencing / transcriptome) parenting the rna-modality nodes. _match_tech then
  hits it at pass 2 (exact alias) and never reaches pass 3.
• Widen _match_tech to return a node, and change the conflict predicate from equality to
  subtree membership: an asserted node conflicts only when observed ∉ subtree(asserted).
  10x-3p-gex-v3 ∈ subtree(rna-seq) → consistent, no question. Prose claiming ATAC against 10x GEX
  bytes still blocks, which is the conflict worth having. A family term then narrows a candidate
  set — strictly more useful than today, where a hypothesis can only break an exact tie.
• Reject a library.chemistry draft that resolves to no KB node, in verify_drafts — where the
  docstring already assumes it happens.
• Independently: split the GEO title on its known grammar <GSM>: <description>; <organism>; <strategy> before it reaches the model. Those last two fields are structured columns SRA
  appends, not prose, and rendering them as prose is what manufactures the error. (This is F5 (design): deterministic GSM-title covariate parser #50's
  parser, one level up.)

Pass 3 also has a standing determinism hazard: it returns the first dict-order match, while
escalate.py:59-63 goes to real trouble to make candidate ordering deterministic "or
candidates[0].technology flips between runs of an unchanged input". The identical hazard is
unguarded here, which means adding a KB spec can silently change an unrelated dataset's manifest
hash. That matters a lot if the corpus is about to grow.

Mechanism 2 — two wordings of one fact, and the fact is discarded (2 of 9)

GSE310378, one deposit, one sentence:

• BioSample typed slot: treatment = "Pro VSV GFP"
• model, reading that same submission's experiment title: treatment = "VSV GFP treated control"

Both are contiguous word-runs of the same title. They overlap. Neither contains the other, so
_is_short_reading_of (records.py:600) fails both directions, _decide (records.py:528-533)
calls it an equal-authority disagreement, and stores null. PRJNA1027859 loses strain = "CQ757"
on one sample the same way. In both cases the no-LLM run gets the field right: adding the LLM
destroyed a fact the archive had already supplied.

That is verbatim the failure _without_short_readings was written to prevent (#182, GSE282765,
"Citrobacter rodentium infection" vs "Citrobacter rodentium"). It recurred in a new geometry days
later. That is the signal worth acting on: records.py:545-597 is a 50-line docstring guarding a
10-line predicate that handles exactly one near-miss shape, and there is no finite list of shapes.
The next ones are already visible — _norm_value (records.py:615-621) folds case and whitespace and
nothing else, so "wild-type" vs "wild type", "C57BL/6J" vs "C57BL/6", "5 days" vs "5d" are
all disagreements today.

The stated rationale does not survive contact with the harness

records.py:479 argues "a wrong value is permanent and a missing one is not." But the manifest is
write-once and content-hashed (R11 / ADR-0004), so a null is exactly as permanent — and the eval
harness grades the dropped treatment as false_accept: "a confident wrong manifest is the one
failure the corpus never recovers from." The harness and the resolver disagree about what null
means, and the harness is right: a manifest that silently omits a field it had evidence for, while
reporting that it decided, is a confident wrong manifest.

Proposed, cheapest first (R9 order):

1. Fold punctuation in _norm_value (-/_// → space before splitting). One line, preserves
   every safety property _is_short_reading_of relies on (whole words, contiguous).
2. Use the synonyms already in the repo. Every asked sample attribute carries NCBI's
   BioSampleAttribute.synonyms — "what submitters write instead" — used today to harmonize keys
   (io/attributes.py:121-127) and fed to the model as prompt text, but never to compare values.
   The one machine-readable vocabulary in the repo is applied to keys and withheld from values.
3. Generalize the containment predicate to overlap-within-one-source. Not "is the reading
   contained in the typed value" but "are the typed value and the reading both contiguous word-runs
   of the same source document, overlapping?" — if so, the submitter's typed string wins byte-for-
   byte. Same safety argument (nothing new reaches the manifest; you store exactly what a no-prose run
   stores), still one-directional. This closes GSE310378 with no model call.
4. Then build the arbitration verb, for genuine paraphrase with no textual overlap ("VSV-GFP" vs
   "vesicular stomatitis virus expressing GFP"). ArbitrationRequest / ArbitrationResponse are
   "modelled and unbuilt — there is no verb" (docs/agents/models.md:164), and AGENTS.md already
   grants the LLM exactly this as job (b): "arbitrate ambiguity code has already flagged." _decide
   is code having already flagged it. Building it completes the design rather than relaxing it —
   and it stays inside R2 because arbitration is a selection over a code-supplied closed set, not
   a generation: the response must be byte-equal to one of the inputs or abstain, and code checks
   that. The model cannot invent a value. Rungs 4–6 are empty; this is what belongs there.
5. Keep the variants either way. EvidencedStr already carries evidence; add variants so the
   manifest records treatment: "Pro VSV GFP" and that the experiment title wrote it "VSV GFP
   treated control". Strictly more information than either null or a silent pick, and it makes
   step 4 auditable after the fact.

The harness is not measuring the code that ships

Two callers reduce the same harvest output to a chemistry hypothesis, differently:

# cli/manifest.py:516-518  — the product. "Agreement or nothing."
values = {a.value for a in assertions if a.field == "library.chemistry"}
if len(values) != 1:
    return None

# evals/run.py:691  — the benchmark. Plain dict build in plan order: last document wins.
by_field = {a.field: a for a in accepted}

Record documents come last in plan order, so under the eval path one line of GSM title overwrites a
good 10x-3p-gex-v3 read off a supplementary table (docs/agents/eval-corpus.md:408-414 says this
outright). Under the product path that pair is {"RNA-Seq", "10x-3p-gex-v3"} → size 2 → None → no
hypothesis → no conflict.

Precisely what this does and does not explain: it does not account for pro's GSE229022 or
GSE256266, where "RNA-Seq" was the only accepted chemistry claim (the correct 10x-3p-gex-v3.1
had already been rejected not_entailed), so both reductions agree. It is the mechanism behind
GSE234962's documented flip. Structurally it makes the harness more question-prone than the thing we
ship. Before the corpus grows, that seam should be one function — otherwise we tune against an
instrument that fails differently from production.

Secondary findings

• over_ask hides that the answer was right. Every over_ask case in both runs had 100% correct
  fields — the manifest was correct and the run still exited 4. Grading that identically to a wrong
  answer means a question-gate regression and an extraction regression are indistinguishable. Suggest
  splitting the axis: decided-correct / decided-wrong / asked-but-would-have-been-right /
  asked-and-needed-to. The third bucket is currently the entire failure list.
• A valid batch under the wrong envelope is discarded whole. extract.py:341-347 raises
  ExtractUnavailable on a response whose top level is a bare JSON array, costing the entire
  document — even though every element would pass AssertionDraft.model_validate, and a partly
  invalid drafts array is already accepted with rejects recorded (extract.py:349-355). ADR-0009's
  "half a batch is worse than none" is about a silent half-parse; this is the same batch under a
  different envelope. Likely accounts for part of flash's 6 failed documents.
• _pretrimmed_blockers blocks on a signal it can already localize. The gate is
  obs.read_length.n_distinct == 1 (escalate.py:237), so one read a single base short in a
  2,000-read head blocks the dataset at exit 3, no appeal — while the Observation carries mode,
  min_len, max_len and percentiles that would tell a 0.3% ragged tail from a real offset shift.
  The over-length escape (escalate.py:239-240) shows the shape a principled exemption takes.
• The read budget is now justified for one statistic. probe/__init__.py:38-56 marks its own
  original justification FALSE ("A HEAD SLICE IS NOT A RANDOM SAMPLE", GSE305031's dark cycle:
  91.35% N at R1 cycle 2 in the first 2,000 reads, 0.00% in the last 2,000) and correctly moves the
  defence to onlist_hit_rate's denominator. But composition, segmentation, n_rate and
  consensus_match_rate are all still built from that head and have no equivalent coverage report.
  Nothing consumes n_rate as a warning today.
• Harvest is strictly one call per document (GSE256266: 13 docs → 13 calls; PRJNA1195922: 25 →
  25). GSE310378's three documents are 45, 209 and 213 characters and cost 9,382 input tokens — >95%
  prompt overhead, paid three times, three round trips. Batching same-scope documents is already safe
  by construction: every draft carries doc_sha256 and code greps the quote into that document, so
  cross-document contamination fails span verification automatically. The tripwire built for
  hallucination is also what makes batching sound.

Proposed order of work

1. Unify the two chemistry reductions (evals → _chemistry_hypothesis). It is the instrument; fix
   it before reading anything else off it.
2. Reject non-KB library.chemistry drafts in verify_drafts. Closes GSE229022 + GSE256266 and every
   latent instance of the title suffix.
3. Overlap-within-one-source containment in _without_short_readings. Closes GSE310378
   deterministically, no model call.
4. rna-seq family node + subtree-membership conflict + _match_tech returning a node. Not required
   for green after (2), but it is what makes the next hundred datasets work, and it removes the
   add-a-spec-changes-a-hash hazard.
5. Split the over_ask grade, and run the LLM cases at trials > 1 so stability is measured at all.
6. Then the arbitration verb, before the corpus grows.

Steps 2–4 are the concrete proposal for #184; steps 3–5 feed #61 thread 1.

[lhqing]

Closed out: both tiers green at three trials — and three things this review got wrong

Measured at 70ba9fd, after seven merged PRs (#191, #192, #193, #195, #197) and two ADRs.

no-LLM,  trials 1   18/18 correct   field accuracy 1.000   FA 0.000   FR 0.000   exit 0    90 s
pro,     trials 3   18/18 correct   field accuracy 1.000   FA 0.000   FR 0.000   exit 0   8m50s
                    stability 1.000 on every case — 54 case-draws, no flips

This is the corpus's first multi-draw measurement. Every number in this thread, and every number in
the issues that came out of it, was a single draw. Sixteen of the eighteen cases had never been
observed twice at the same commit.

---

1. The default-model finding stands. The mechanism findings stand. The case attribution never reproduced.

Both mechanisms this review traced into source were real, and both are fixed:

• library.chemistry = "RNA-Seq" — closed by ADR-0020, resolve: a chemistry string must NAME a chemistry, and a family term narrows (PR 2 of #184) #192
• the equal-authority attribute drop — closed by ADR-0021, resolve: the slot the submitter typed outranks a reading of that same deposit #193

But which cases each hit was a lottery, and this thread's per-case table did not survive re-measurement.
The very first re-run at ac11b44 — same commit, same model, same prompt — put the failures on a
different set of cases than reported here. GSE229022 and GSE256266, named here as pro failures,
both graded correct. PRJNA1027859, named here for the attribute drop, graded correct. Across
later runs GSE229022 went correct → over_ask → correct with no code change touching it.

The lesson is not that the analysis was sloppy — the mechanisms were traced correctly, into source.
It is that a single-trial per-case grade on this tier is one sample, and this thread read it as a
property of the extractor. Every issue cut from this review was therefore scoped to a mechanism,
never to a case list, and that is why the fixes hold at three trials.

2. "The benchmark is not measuring the code that ships" was right about the disease and wrong about the symptom

This thread identified the harness/product seam through one instance: manifest fill reducing
chemistry claims agreement-or-nothing while evals took them last-wins. That was fixed first, in
#191, deliberately ahead of any compiler change so the instrument would be trustworthy.

It was a measured no-op. Recomputing both reductions over the run's own accepted assertions: they
return the same hypothesis on all 18 cases, because no case ever produced two distinct chemistry
values. The specific claim in this thread — that the seam is the mechanism behind GSE234962's flip —
is unproven and was not reproduced.

The class was real, and the large instance was somewhere else entirely. evals/run.py called
resolve_dataset; manifest fill called resolve_runs, whose own docstring calls the former "the
bug" for a multi-file dataset. The benchmark packages are genuinely multi-run — GSE126954 holds 175
FASTQs, PRJNA1027859 holds 18 across 6 experiments — so 11 of 18 cases were graded through a code
path the product had abandoned. It passed only because those datasets are homogeneous, which is a
property of the corpus, not of the code. Closed by #197, which promoted the dataset-level reduction
into resolve.reduce_dataset so both front doors call one function.

So: right that the instrument diverged, right to fix it first — but the instance this thread found was
the small one, and it took the contract test written for the small one to expose the large one.

3. Three of this thread's proposals were rejected, for reasons that generalise

• The rna-seq family node. Not added. The KB hash is a run_id input, so a new spec re-keys
  every dataset — and it is unnecessary once the matcher is one-directional. ADR-0020 instead made
  resolve_chemistry accept alias ⊆ needle and refuse the reverse.
  Also: this thread's step 2 as written was a no-op. "RNA-Seq" did resolve to a KB node
  (bulk-rnaseq-pe, by substring), so "reject a draft that resolves to no node" would have rejected
  nothing. Two more strings had the same defect and are not in this thread: "Illumina" and
  "transcriptome" also resolved to leaves.
• Strict exact-alias matching, the obvious repair, is worse than the disease: it rejects
  "Chromium Next GEM Single-Cell 5' Reagent Kit v2" — GSE317744's own record — and would have
  closed the metadata channel in production while the corpus stayed green off its recipe hypothesis.
• Overlap-within-one-source for the attribute drop. Rejected: every string-comparison remedy
  (containment, overlap, synonym tables, punctuation folding) closes the paraphrase case and leaves
  the misfiling case open. GSE317744's colliding claim was "CCR9 KO" against a typed
  treatment = "MC38_3 weeks" — the model filed the genotype under treatment. Nothing that
  compares those two strings can help. ADR-0021's rebuttal table has the full argument.
• Splitting the GEO title on <GSM>: <desc>; <organism>; <strategy>. Rejected outright: it encodes
  one archive's display grammar, and seqforge also compiles in-house datasets with no accession.
  Replaced by an archive-neutral rule — a free-text span byte-equal to a typed attribute on the same
  record is a re-rendering, not a second source.

4. A green draw is not a fixed bug

Worth recording, because it nearly closed a live issue. After #192 the pro tier read 18/18, exit 0,
with the attribute-drop defect entirely unfixed — the model simply volunteered no colliding
treatment claim on that draw. resolve/records.py had not been touched by #191 or #192, so three
benchmark runs were three independent draws of one behaviour, and the defect fired on 3 of 6
case-draws. #193 shipped on the mechanism and the deterministic tests, not on the grade.

5. The secondary findings mostly shipped

Four of five, in #195: the bare-array envelope (a valid batch under the wrong envelope no longer costs
a whole document), the _pretrimmed_blockers gate (now a floor on the share of reads, not unanimity),
head-slice coverage (every head-derived statistic now reports what it was measured over), and
same-ask document batching. The over_ask split was declined — after ADR-0020 the tier has no
over_ask cases, so its motivating instances no longer exist.

Batching is worth a number. Per trial, against the pre-#195 run:

requests       141  ->   47    (0.34x)
input tokens   651k -> 320k    (0.49x)
output tokens  106k ->  55k    (0.52x)

Three times fewer requests and roughly half the tokens, for the same 141 documents.

6. What is still not measured

• Flash was never re-run. This thread's "pro wins on every axis" remains a single-draw claim about
  a model that has not been measured since. The default did flip to -pro (eval: one reduction for the hypothesis, and name the model that ran (PR 1 of #184) #191), and the falsified
  "cheap end of V4" claim was retired from five docstrings — but on evidence that was never repeated.
• EXTRACT_PROMPT_VERSION moved 2026.7.4 → 2026.7.5, so no token or grade number in this thread
  is directly comparable with one taken after #194 and #190 section A: the ceiling bounds spend, and five deferred mechanisms #195.
• 5 of 423 planned documents failed extraction in the acceptance run, leaving three cases
  (GSE234962, GSE256266, PRJNA1027859) grading over slightly less than their whole prose. No
  missing and no hallucinated fields in any of them and assertions_unchecked is 0, so the rates
  are honest — but they are narrower than they look, which is exactly what the coverage line added in
  eval: a skipped harvest stage is not a clean one #187 exists to say out loud.

---

Nothing carried over. #184, #189, #190, #194 and #196 are closed; the items that were argued but
unexercised were recorded as decided-not-doing rather than left to rot in a backlog, with the case
that would justify each one named. The two open issues (#153, #61) predate this review.