Subagent claims can ship with citations that don't support them — a concrete case and a cheap fix

[KurohaLab]

Version: 0.1.5-rc.1

What happened

I asked DSH to research the current camera market and write up an overview. One line in the
delivered report read:

Hasselblad XCD 2.8-4/35-100E (2025) — the company's first zoom, equivalent 28-76mm...

with a citation to a reputable review site attached to it.

I happen to know this field, and the claim looked wrong to me: Hasselblad had zoom lenses
well before that one. So I asked the agent to go back and check the source. It did:

• The cited review never says "first zoom". Its only mention of that lens is
  "the new 35-100mm f/2.8-4 zoom lens".
• The manufacturer's own press release never says it either — and in fact compares the new
  lens against an earlier zoom by name, which by itself disproves "first".
• Grepping the entire review for "first" returns only unrelated hits
  ("first-class build", "continuous AF for the first time").

So the superlative was fabricated by a subagent, and a genuine citation was attached to it.

Why this is worse than an ordinary hallucination

An unsourced wrong claim is easy to distrust. A wrong claim wearing a plausible citation
looks more trustworthy than the same claim bare. As the person reading the report, I had
no reason to suspect that line — it had a source, and the source was real.

It also compounds with depth: a subagent produced the claim, the parent agent folded it into
the deliverable, and neither layer re-checked the source.

Suggested fixes

1. Require quotes, not just URLs, in subagent results. If structured output supported
   {claim, source_url, quote}, a fabricated superlative would be far harder: the subagent
   would have to emit a quote that does not exist, rather than merely attach a URL that does.
2. Ship a citation-check helper. A tool taking a URL plus keywords and reporting whether
   they occur on the page. This is fetch + grep, trivial to implement, and would have caught
   this case in a single call.
3. Prompt-level: flag superlatives as high-risk. "First / most / only / never" deserve a
   mandatory "does the source literally say this?" step before the claim is allowed into a
   deliverable.

Cost of verification here was about two seconds of grep. Cost of skipping it was a false
claim in a finished report — caught only because I happened to know the domain well enough
to be suspicious. In a field I don't know, I would have had no way to tell.

Note on attribution

I understand part of this is model behaviour — both the fabrication itself and the
instinct to attach a real citation to it come from the model. I'm filing it here because
every fix I can think of is harness-level, and because I'm not aware of a dedicated
channel for the model itself. If useful, please forward the model-side half to the right
team.

The discussion is drafted by ds and reviewed by me.