-
Notifications
You must be signed in to change notification settings - Fork 0
Literature Review Bakeoff Findings
Part 5a of the Literature Review: Ornith-1.0-9B, the Searcher over fetching root cause, and the extended comparative survey.
Five live runs total against deepreinforce-ai/Ornith-1.0-9B (dense, Qwen3.5 based, MIT licensed):
two real DeepDelve bugs found and fixed, one Ollama serving layer gap isolated and confirmed, model
verdict left genuinely open.
A direct tool call smoke test was the cleanest of any candidate tested this project, short bounded
reasoning and correctly nested delegation calls. The first full benchmark hit a QuotaAbortException
(one specialist re delegating to its own sub analyzers until the shared quota emptied), never
reaching the completion check phase. A second run with quota bumped produced a real, accepted,
grounded report, but a near empty one: every dispatched Analyzer self rejected 19 genuinely on topic
fetched sources as non citable, trivially grounded (zero citations, nothing to be wrong about) but
delivering zero actual answer, the mirror image of the "grades its own homework" concern often raised
about this model family.
The root cause of the stuck loop failures was a real chat template defect, checked directly against this model's own GitHub issues: two open issues describe the identical symptom independently across unrelated harnesses (VSCode/Continue, Claude Code), and a Hugging Face discussion names the mechanism, the stock chat template injects an empty think block before tool calls, poisoning the model into associating empty thoughts with looping tool invocation. A community patched chat template (already under investigation the same day for a separate Qwen3-4B LoRA) got applied directly to this candidate's GGUF metadata and re imported; live retested, it now correctly split a two part request into two separate delegation entries, a real, measurable improvement.
Two genuine DeepDelve architecture bugs surfaced along the way, both model independent, both fixed and regression tested. First, a completion check scoped delegation tracking to the current process only, so a resumed run correctly told not to re delegate got a "your only next tool call must be delegate" directive anyway, derailing it into a reflection loop until forced quota abort. Fixed by also treating resume carried over fetched URLs as proof delegation happened in any prior session. Second, several Builder fixable grounding checks told the reader to "delegate a Searcher," worded for the Planner, but Builder has no delegation tool at all; a correction cycle got stuck narrating that delegation was required before it could continue, a reasonably honest reaction to a genuinely impossible instruction. Fixed with a clarification appended to Builder's dispatch branch explaining it cannot delegate and should drop or rewrite the claim instead.
Even after those fixes, a targeted "drop 3 flagged citations, keep everything else" correction cycle
kept failing, since Builder only had a full rewrite tool, meaning every correction meant regenerating
the whole document from cold context, whack a mole fixing old citations while introducing new ones
each time. A new edit_workspace_file targeted replacement tool was added to both Builder and
FindingsWriter's tool lists. Live tested once: the model didn't spontaneously reach for it even on a
textbook targeted fix case, inconclusive on a single sample, left open for a future session.
A serving layer finding, precisely isolated across four direct API tests: thinking suppression works
cleanly on Ollama's native endpoint in every case tested, including with tools present, but leaks a
short reasoning text into a reasoning field specifically over the OpenAI compat endpoint when tools
are present. Since DeepDelve is built entirely on that OpenAI compat client, every model ever tested
through Ollama has been subject to this endpoint level leak whenever it calls a tool with thinking
nominally disabled, a real, previously unattributed, narrower cousin of the already accepted Qwen3
think mode passthrough bug, confirmed here not to be inherent to the model or template since the
native endpoint proves full suppression is achievable.
Two Reddit threads read directly the same day independently corroborated the looping pattern across multiple unrelated users and harnesses for both the 9B and 35B variants, and one blunt outside opinion, "there's no good model at 9B," independently matched this project's own "no more small models" scoping decision made the same day from a source with no knowledge of it.
Note
Verdict as of 2026-07-28: left inconclusive, deliberately, per the Model Evaluation Standard, since every failure mode hit had an independent, non model explanation attached (a DeepDelve bug fixed twice mid session, a serving layer gap, a missing editing capability). What was positively confirmed: the model's initial cold start synthesis from real evidence was the strongest of any candidate tested this session. A clean re test three weeks later finally settled this, see below, not left open indefinitely.
A follow up run through the new native Ollama backend surfaced a recurring aclose() asyncio warning,
traced to the ollama Python package's own internal generator handling, a pre existing upstream
quirk, not DeepDelve code, confirmed via Python's own asyncio docs and a direct source read. Traced
through the raw session log to rule out actual data loss: none found. One real, separate issue
surfaced in the same run, unrelated to the backend: a write call failed Pydantic validation on a
missing filename, a genuine model generation slip that self corrected on the next call. A real,
separate quality gap also surfaced: the final report dropped genuinely on topic, well cited findings
in favor of an irrelevant citation, and the existing coverage check didn't catch it since the
aggregate citation ratio still cleared its threshold even though the wrong content was dropped, a
real, pre existing blind spot (measures how much evidence survives, not which) worth its own tracked
item. Verdict on the backend change itself: no data loss, no correctness regression found.
Resuming the native backend retest immediately hit a new failure: web_search calls looping with
corrupted, wrongly nested arguments, repeated 15+ times identically, never self correcting.
Two hypotheses got ruled out before the real cause was found. Multi turn context pollution was disproven directly: the very first call was already malformed and all 24 retries were byte identical, a deterministic single shot issue, not something needing a long conversation. A tool_call_id collision (every call in one conversation getting the identical id, since Ollama's own Python package silently drops the real per call id on parse) was confirmed as a real, still open upstream limitation, independently corroborated by an unrelated project's own Ollama client hitting the identical gap, but ruled out as the cause here since DeepDelve's dispatch pattern is strictly sequential, never actually ambiguous under that pattern.
The confirmed root cause: an open upstream Ollama bug where Qwen3.6 family models intermittently drift off their own XML tool call format, and this project's Ornith tag declared no explicit parser or renderer, plausibly why the drift produced silently corrupted arguments instead of a clean error elsewhere. A synthetic probe didn't reliably reproduce it; a faithful repro harness built from DeepDelve's own real client construction, prompts, and tool list did reproduce it on the very first turn. The fix: an explicit parser and renderer declaration in a rebuilt Modelfile, live verified with zero consecutive function call errors across 10+ real search calls versus failing by call 5 every time on the old tag, and confirmed again in a full ~47 minute benchmark run with 0 of 18 web search failures, a complete elimination of this bug class. Ornith's own overall verdict stayed open for an unrelated reason: the run hit its wall clock limit mid retry chain while still correcting a genuine, separate content quality issue, 6 honest write/review/fix rounds happened, it simply ran out of budget, a convergence speed question, not a new architecture bug.
Every confound raised above was fixed by this point: the chat template defect, the native backend
tool call corruption, and the two DeepDelve architecture bugs. A genuinely clean re test used the
fully patched tag with enable_thinking: false, confirmed via direct curl to be the correct clean
setting for this model on the native endpoint even with tools present, the opposite of what a
Qwen3 candidate tested the same week needed, a reminder to verify this per model rather than assume
one setting transfers.
Research quality was again the strongest of any candidate tested at this size, 17 sources, 22
findings, no tool call corruption, no infinite loop. But at the delegation quota exhaustion point,
the Planner narrated its wrap up as chat prose instead of ever calling the write tool, exhausting
the full writer retry budget with findings.md never accepted at all. Same narrate instead of call
failure class already seen across most sub 14B candidates tried, and independently corroborated for
this specific model family by the two Reddit threads cited above as a harness agnostic trait, not
something a serving layer or template fix reaches. This closes the long standing inconclusive
status for good: genuinely the strongest research quality candidate tested at this size, the
ceiling isn't research capability, it's converting research into a written artifact under real
task completion pressure.
Found while smoke testing a per facet Builder dispatch fix, with two small, purpose built single fact prompts that both timed out before ever reaching the completion check pipeline. Traced from the run state and session log, not guessed: the Planner's own plan was correctly scoped from the start, a clean 2 task plan matching the existing "dispatch a single Searcher" rule for simple factual queries. The actual budget sink was one level down: a single WebSearcher dispatch fetched 11 distinct URLs for one uncontested fact, despite explicit prompt instructions saying one authoritative source is sufficient and to stop early. That exhausted the task's context budget, triggering the existing one shot cutoff and a "wrapped up early" marker, itself the intended graceful degradation. The cascade came next: the Planner correctly read the cutoff marker as needing a follow up task, but dispatched an unforced second verification task for the OTHER, already clean facet in the same batch too, not distinguishing "this one needs a follow up" from "let me re verify everything."
This is a named, documented failure class, not a one off quirk: one paper names exactly this shape as a termination failure, ambiguous success conditions causing models to continue verification passes past the point of diminishing returns; another argues agents need a trained, adaptive stopping depth rather than a prompted one, since prompt only "stop when you have enough" instructions are documented as unreliable for this class of model.
langchain-ai/open_deep_research, the most directly comparable real open source project (same
Planner/Supervisor to Researcher shape), solves both halves of this exact problem with hard,
code enforced numeric counters, never a prompt only instruction: a per researcher step tool call
count ceiling, and a hard cap on the Supervisor's own reflect and follow up loop, both enforced in
code on every iteration, never left to the model's judgment.
Implemented and live tested 2026-08-01, mixed result reported honestly. A per task fetch cap and
softened cutoff wording fixed the exact traced incident: rerunning the same two prompts, neither task
ran away fetching sources, and no context budget cutoff fired in either run. But both runs still
timed out anyway. In both cases the model issued multiple delegate_tasks rounds (up to 6 for a
2 task query) with zero cutoff markers anywhere to have triggered it, proving the unforced
replanning is a more general tendency of this model to keep replanning regardless of any specific
tool result signal, not solely a reaction to the cutoff wording. This confirmed the softened marker
was necessary but not sufficient, and the same day a hard cap on the Planner's own top level replan
rounds was scoped and implemented, mirroring open_deep_research's own precedent directly: a new
_planner_delegate_over_cap enforced inside the delegation closure, a new
settings.max_planner_delegate_rounds (default 4).
Continuing the earlier survey's primary source reading methodology, prompted by the day's own work
after cloning open_deep_research to source the replan cap precedent above.
Uses a LangGraph supervisor/researcher graph, not DeepDelve's
typed hierarchy, with hard enforced iteration caps confirmed by reading the actual enforcement code,
the direct precedent for this session's fix. A genuinely new, not yet applied finding: its own
"Scaling Rules" directly address DeepDelve's still open task naming/facet collapse problem (a
Lisbon versus Mexico City comparison query that silently only ever covered one city), instructing
the model to delegate one sub agent per named comparison element with fully standalone instructions,
an axis DeepDelve's own single facet per slot rule doesn't cover. A live fabrication test running
DeepDelve's own gpt-oss model through this project's unmodified code on a simple factual query
found the headline fact correct (likely from training data, not the search it ran) but two of four
cited URLs dead on direct fetch and one supporting claim outright fabricated with a fake supporting
citation. This project has no citation exists verification and a bare self reported completion
signal with no structural check behind it, exactly the class of defect DeepDelve's grounding pipeline
exists to catch, and would have caught here.
Is the most directly comparable project by design goal (local model first), but not multi agent at all, one linear loop that never attempts multi facet decomposition, a structurally different way of avoiding the facet collapse bug class: it never tries to decompose in the first place. Its most consequential divergence: it defaults to raw JSON mode over native tool calling, the opposite bet from DeepDelve's own tool calling required stance, real evidence the team building the most local model focused reference implementation in this space didn't trust native tool calling enough to default to it. No citation to claim verification exists.
Re confirmed under its real repo name: its control flow is entirely host code driven recursion, not model driven, the model only ever fills fixed schema slots at fixed call sites and is never asked "should I search more?" This sidesteps the whole over fetching/over replanning problem class by construction, since a model with no iteration control authority has nothing to over exercise, a genuinely different fix family from either a hard cap or a smarter prompt, worth naming as a third option, though adopting it would mean giving up the Planner's ability to genuinely adapt its plan.
Is by far the largest, most production hardened codebase surveyed
(~9,900 lines), with a real swappable retriever abstraction across 19 provider modules, genuinely
more mature than DeepDelve's own hardcoded single search backend. Its SourceCurator ranks and
filters sources by credibility before they reach the writer, a real idea (separating "is this source
good" from "what does it say") but still pure LLM self judgment with nothing structural backing it.
Its own breadth/depth recursive mode has a documented fix, referenced by its own maintainers' issue
tracker, for stopping descent when every branch at a level fails rather than recursing forever, real
confirmation from a more mature project in the same lineage that dzhng's silent per branch failure
gap is a real production issue, not hypothetical. Its defensive multi strategy JSON repair chain,
cascading through several fallback parsers before giving up, is a battle tested hardening pattern
worth remembering for any future DeepDelve code path that has to parse free text by hand.
Is the one project in the whole survey that isn't a variant of "decompose into search queries, iterate, synthesize." Its real contribution is persona diversity driven facet discovery: it pulls real Wikipedia table of contents structure as grounding for generating several distinct "editor" personas, then runs a simulated multi turn dialogue per persona, each driving its own independent research thread before everything merges into one outline. This is a structurally different fix family for DeepDelve's own facet collapse problem than a prompt rule, generate diverse perspectives on the query first so each naturally drives its own thread, rather than hoping one top down planner's task naming discipline holds. Not scoped for implementation, a larger architectural change than a prompt fix, but the most novel, best grounded idea in this survey for that specific open problem. It has no verification layer of DeepDelve's kind at all, its quality mechanism is entirely upstream (facet coverage via personas), not downstream citation checking.
Nine projects now read directly across two sessions. The earlier survey's
qualifications (different projects solve different problems, sophistication isn't the same as
validation, no survey is exhaustive) hold and are reinforced, not weakened: GPT Researcher is a
materially larger, more mature codebase than anything in the earlier set and still has no
verification layer comparable to DeepDelve's, for the same reason, it optimizes retriever breadth and
production polish, not fabrication defense. The live gpt-oss fabrication test on open_deep_research
is the most concrete evidence yet, on this exact local model, for why that verification layer earns
its complexity: an unmodified, well regarded reference implementation produced a fabricated fact and
two dead citations on the simplest possible case, with nothing in its own code that would have caught
it. Two concrete ideas are now on record as candidate future work, deliberately not implemented this
session: open_deep_research's per comparison subject task naming rule, a small prompt level fix,
and STORM's persona diversity facet discovery, a larger architectural one, both targeting the same
open bug from different angles and cost levels.
History
Model Research
Reviews & Audits
Reference