-
Notifications
You must be signed in to change notification settings - Fork 0
Literature Review Bakeoff Findings 2
Part 5b of the Literature Review: the Lisbon/Mexico convergence chase, the 2026-08-17 seven fix arc, and AgentFloor.
The standing Lisbon versus Mexico City benchmark got rerun repeatedly to confirm a genuine end to end pass. It never fully converged, but each rerun cleared one real bottleneck and exposed a new one underneath, four distinct, independently real fixes in one investigative arc, not one bug with four symptoms.
findings.md was counting failed extractions as real, must-cite evidence. Most sources held only "No key findings extracted" narration, the Analyzer's own honest failure, but both grounding checks treated every URL as real evidence the report must cite, an unsatisfiable demand the Builder could only resolve by re-failing the check or fabricating a claim. Fixed with a two-signal null summary detector (short text plus a "nothing found" phrase match) excluding those URLs from the "must cite" set.
Two of those "null" sources turned out not to be genuine Analyzer failures at all, one was a CAPTCHA challenge page, one a literally empty fetch. Tavily's own extract API returned real content for both when tested directly. Added Tavily extract as a third rung in the existing stub retry chain. Two real bugs surfaced while wiring it in: the stub marker regex only covered paywall/404 phrasing, not CAPTCHA pages, and the extract call was sending the post-redirect URL instead of the original, so a bot-walled page's redirect chain just fetched the same challenge page again. Live verified against both exact failing URLs: one went from a 723-character captcha stub to 7,910 real characters, the other from 0 bytes to over 19,000.
With both fixed, the run still never converged. FindingsWriter itself was dropping 3 of 4 facets while writing findings.md, despite a complete, well under budget evidence blob in one dispatch, the same evidence-crowding pattern already fixed for Builder, one layer upstream. Fixed by giving FindingsWriter its own per-facet dispatch mirroring the existing Builder one, with a filter so each facet's dispatch only sees its own evidence. This retest also caught a live wording drift in fix 1's own detector, the model's failure narration had shifted to a phrasing the original regex missed, letting most entries slip through unfiltered; broadened the pattern.
Even after that, the run still never reached the completion check pipeline at all, the whole budget was spent in research. Traced from the raw persisted session transcript: grep_workspace_file/read_workspace_file share one global quota with no per-dispatch ceiling, so one Analyzer burning through a long irrelevant document starved every sibling Analyzer dispatched afterward, hitting a quota-reached wall almost immediately on perfectly readable pages, the exact "one task starves every sibling" shape the existing delegation and fetch caps already prevent elsewhere. Fixed with a new per-dispatch combined read/grep cap for Analyzer roles specifically, left uncapped for Builder/FindingsWriter/PeerReviewer since their read pattern (reviewing an already-written artifact) is legitimately different.
Live tested with all four fixes together: the run still timed out, but for the first time produced a real report with zero null-summary entries and both cities represented in findings.md. But Builder's own first draft still covered only one city, and the run was killed by the timeout the instant Builder finished, before the completion check pipeline got a second turn to fix it via the per-facet dispatch built for exactly that case.
A --resume-run on the interrupted run finished cleanly in 24 minutes this time, but produced a byte-for-byte identical report, still missing the second city, despite findings.md having clean sources for it the whole time. Root cause, traced directly: the resumed Planner was told in prose not to reopen broad research, and ignored it, redelegating new research for facets that were already well covered. Those new tasks got flagged as fabricated and dominated 6 of 8 completion check attempts, meaning the one check that would have actually caught the dropped facet never got a turn at all, since the whole pipeline runs one tier (task-level checks) to exhaustion before ever touching the other tier (grounding/coverage checks). A structural problem the Planner itself created in an unrelated tier permanently blocked the fix that would have worked.
The user's own framing: "we're going for the more structured approach, we cannot just do the smaller, we need to cut this from the root." Fixed structurally: on resume, if the required report artifact already exists on disk, the Planner's delegation round counter gets pre-set to its cap, so its very first delegation call this run is already rejected before it runs. A resumed run with an existing report becomes fix-only, the Planner structurally cannot reopen research. Live tested: the Planner correctly never delegated, finishing in 67 seconds instead of 24 minutes, but the report was still byte-for-byte identical for a different, unaddressed reason, the task verification flags themselves carry over on resume, so the same starved-check problem fired again from a pre-existing trigger instead of a self-created one.
The deeper cause: the completion check pipeline only ever evaluates its grounding/coverage checks when every task-level check has returned nothing, a hard two-tier gate, not just list-position priority. As long as any task-level problem keeps recurring, the coverage check that would catch a dropped facet never runs at all. Fixed by reusing the existing starvation-yield mechanism (already protecting one sibling check) to give the coverage check a real turn even while a task-level problem is still active, deliberately allowed to become the run's terminal reported blocker if it's the real problem, unlike the sibling it borrows the mechanism from.
Live tested: resuming the same run a third time, for the first time across every attempt this session, the report covered all four facets. Not a clean pass, and the run's own verdict says so honestly, several ordinary per-claim grounding issues remained flagged and unresolved, a separately surfaced, correctly reported concern, not evidence the structural fix failed. The bug this whole chain targeted, a whole facet silently vanishing with nothing able to see or fix it, is confirmed fixed.
Why two specific tasks never produced a real source: a dead end error message, not a quality problem
Investigating why two tasks kept producing fabricated sources across every retry: both hit the identical pattern, a search surfaced a promising URL, a sub-agent got dispatched to read it, and a real, sound existing check rejected the dispatch because the URL had already been fetched by a different task earlier in the run. The rejection message's own suggested fix, fetch it yourself first, was flatly wrong in this exact situation, since a fresh fetch of an already-fetched URL is guaranteed to hit the dedup wall again. The model had no way out, burned its entire delegation quota retrying the identical rejected shape, then gave up and narrated a summary straight from the search snippet, correctly caught as fabricated by the existing check. The source itself may have been perfectly fine, the model just could never get a working dispatch through to read it. Fixed by having the rejection message look up whether a sibling task already fetched the URL and, if so, name the real saved filename directly and explicitly say not to retry fetching. Implemented and unit tested, not yet live confirmed against a fresh occurrence of the exact collision.
Testing whether the external eval timeout (1800s) was shorter than the agent's own internal wall clock budget (2700s) found that every "timeout" seen that day was actually the external harness killing the process 15 minutes before the agent's own graceful stop logic ever got a chance to run. Rerunning with a longer external timeout let the process finish on its own, but it scored zero: the final report was an auto-recovered narration banner, the Planner never wrote a real one. Root cause, read directly from the attempt sequence: check_missing_artifact (the check that dispatches Builder to actually write the report) never fired once, because a task-verification problem kept winning first-match on 6 of 7 attempts, the same starvation shape as the two-tier fix above but this time within one tier, for a different pair of checks. The Planner spent its entire delegation budget re-verifying one persistently flagged task before ever reaching the point where Builder could be dispatched at all. Diagnosed and documented only, not yet fixed, per an explicit "document it and stop for today" direction; two candidate fix shapes were named (extend the same starvation-yield protection within this tier, or reconsider whether an unrelated problem resolving mid-streak should reset this check's own escalation counter at all) without picking one.
Same "chase convergence, layer by layer" shape as above, four fixes there and seven here across six live runs of the same standing benchmark. This section is the literature cross-check done after the fixes, to confirm they land in an active research area rather than inventing novel terminology for known phenomena, plus a real evaluation-methodology gap this whole pattern exposed. Every paper cited below was read in full, primary source, not a search summary, several after an earlier pass in this same review had wrongly cited them from summaries or an undercounted page read, caught and corrected on self-audit or by the user directly.
(a writer role ending its turn with nothing, no marker, unlike the two previously tracked cutoff-marker mechanisms) matches a documented, named 2026 production failure class: roughly 45 to 48 percent of agent failures reportedly close with a confident but empty or false completion claim rather than continued work. This lines up with DeepDelve's own measured 25 to 42 percent empty-summary rate across two live runs, two independent measurement methods landing in the same range, strengthening confidence this is real and general, not a serving quirk. Not yet fixed, the literature only names and measures it, no clean structural fix is offered.
(Tsui, "Self-Correction Bench," COLM 2026, arXiv:2507.02778, read in full). A correction to an earlier draft of this review: a widely repeated "two independent papers" framing for a 64.5% figure was wrong, a downloaded and grepped second paper never actually reports that number at all, it's a pure methodology critique with no measurement of its own. The real, single source: testing 14 open-source non-reasoning models found a 64.5% average self-correction blind spot, isolated by injecting the identical error either into the model's own prior turn or the user's prompt, a model that fixes the external version but not the identical internal one has the knowledge but fails to activate it, a genuine activation failure, not a competence gap. The blind spot isn't solved by frontier proprietary models either, Claude 3.5 Haiku and Sonnet 4 show 41 to 52 percent blind spots, lower than the open-source average but far from zero. A per-model breakdown (read on a later pass) shows an enormous spread directly relevant to DeepDelve's own choices: several Qwen3 sizes score a near-total blind spot (0.004 to 0.108) tested specifically in non-thinking mode, the same operating regime DeepDelve runs in by default, while one other model scores near-perfect. Robustness checks (temperature, compute budget, cross-judge agreement) all hold. A directly actionable, training-free intervention: appending the single word "Wait" after a model's own erroneous output, with no fine tuning, reduces the blind spot by 89% on average and nearly closes the gap to that same model's own full reasoning-mode variant. Not yet tried in DeepDelve's own retry instructions, a concrete, cheap follow-up worth scoping. Directly relevant to a same-day FindingsWriter loop: a byte-identical rejected findings.md snapshot repeating twice, 11 minutes apart, is a cleaner, more extreme instance of the same blind spot, since the deterministic fallback path removes model variance entirely, meaning DeepDelve's own retry architecture can hand a model literally identical input and expect a different result, something no amount of self-correction capability could ever fix on its own; the retry loop itself needs a fix, detecting an unchanged (problem, content) pair and escalating to a genuinely different strategy rather than a third identical retry.
A same-day dedup fix for repeated identical read calls, scoped narrowly from the session's own transcript before any literature check, turned out to match a named, established 2026 mitigation pattern applied more broadly: hashing repeated (tool, args, error) tuples and halting a stuck agent after 2 to 3 repeats. DeepDelve's own fix is a narrower special case, it discounts the quota cost of an exact repeat rather than halting outright, since the specific tool involved is idempotent. Worth naming explicitly for any future generalization to a non-idempotent tool: use the repeated (tool, args) key as a signal feeding the existing hard-abort threshold, not just a quota exemption.
Every fix in this whole arc was validated the same way, implement, run the benchmark once, confirm the target symptom didn't recur. This is exactly the evaluation gap current agent-reliability literature has converged on naming: a single pass rate conflates "can the agent solve this at all" with "does it solve it every time," and a benchmark reporting only one of the two hides the other story. A directly on-topic paper (Khanal, Tao, Zhou, "Beyond pass@1," arXiv:2603.29231, read in full across all 23 pages after an earlier pass wrongly stopped at page 6 on a bad page-count read) measures exactly this for a benchmark domain that includes "Agentic Web Research," the same task shape as DeepDelve's own pipeline. Its own motivating number: one frontier model scores 61% on a single attempt but only 25% when required to succeed across 8 repeats, a single best-effort attempt looking 2.4x better than the metric that actually matters for an unattended system. Its own methodology uses k=3 repeats, external precedent for a "k=3 is a realistic floor" recommendation. A genuinely important finding for DeepDelve's own "frontier models were a disaster" observation: the two models with the best very-long-horizon reliability also have the highest meltdown rates, since more capable, more ambitious multi-step strategies create more chances to both succeed and spiral, weaker models instead emit stable but shallow tool-call sequences that never spike but also never finish, not "frontier models are less reliable" so much as capability and ambition creating more opportunities for both outcomes. A memory scaffold finding: across every model tested, an augmented memory scaffold never improved long-horizon reliability relative to a plain react loop, neutral for some, actively worse for others, especially mid-capability models, "capable enough to use the scratchpad but not capable enough to absorb its overhead efficiently." A critical scope limitation stated by the paper's own authors: it evaluates only 10 open-source models, explicitly not GPT-4o/Claude/Gemini-class proprietary frontier models, so none of its "frontier" findings say anything about proprietary APIs one way or the other.
Implemented the same day: a reliability summary mode in the eval harness reporting both the "succeeds at all" and "succeeds every time" rates across repeated runs of the same query, the equivalent rigor bar for engine fixes that the project's own Model Evaluation Standard already sets for model comparisons. Deliberately not yet used to draw any k≥3 conclusion about that day's own fixes, since the pipeline was still actively changing underneath them.
(Mustahsan et al., "Stochasticity in Agentic Evaluations," arXiv:2512.06710, AAAI 2026, read in full including all appendices). Its Intraclass Correlation metric decomposes an evaluation's variance into real task difficulty versus trial-to-trial randomness for the same task. On the benchmark tier closest to DeepDelve's own open-ended shape, one strong model shows 70% of its observed variance is pure randomness, "single-run results are essentially unreliable," and even the best model tested only reaches moderate, not good, reliability there. Its own convergence analysis finds reliability estimates stabilize around 8 to 16 trials for structured tasks but around 32 for the hardest, open-ended tier, meaning the earlier "k=3 is a realistic floor" recommendation was borrowed from a different paper's own methodology choice for a differently shaped benchmark, not a claim that k=3 achieves a converged estimate for a task this hard. The paper's own closest real-world analog to DeepDelve, a deep research agent evaluation, used only 8 trials for cost reasons and explicitly says further research is needed for generalizable conclusions from that count, meaning the project's own k=3 default should be read as a cost-driven practical floor, not evidence of statistical convergence.
Two papers read in full to answer this directly. One (Su, Wu, University of Hong Kong, arXiv:2602.08272, a MARL sample-complexity paper) had been wrongly dismissed on an earlier 3-of-32-page read as training-theory-only and not load bearing, caught directly by the user. Finishing the read found a general, mechanism-level principle validated on real GSM8K data: decomposing into genuinely independent subtasks scales cost down, but decomposing into dependent subtasks introduces error propagation with a worst-case penalty that grows with the number of agents, and this gap widens as agent count grows. This maps precisely onto where that day's own bugs concentrated: DeepDelve's per-facet research dispatch is the genuinely independent, good case this theory predicts should benefit from decomposition, but the consolidation stage, one FindingsWriter dispatch integrating every facet, one Builder dispatch integrating the whole file, is exactly the dependent case, and every single bug in the whole seven-fix arc occurred at that consolidation junction, not during the independent per-facet dispatch, a real structural match, not a coincidence.
The second ("The Illusion of Multi-Agent Advantage," Jwalapuram et al., Salesforce Research et al., arXiv:2606.13003, read in full including all appendices) finds that automatically generated, dynamically routed multi-agent frameworks consistently underperform simple single-agent self-consistency despite costing far more, collapsing into simple ensembling most of the time. But this specifically targets systems where an LLM or controller decides the coordination structure itself at inference time, something DeepDelve doesn't do at all, its roles and dispatch shape are fixed and hand-designed for every query. The paper's own closest analog to that shape, a deterministic, code-driven pipeline with explicit role decomposition, is the one architecture in the whole paper that wins decisively, and the paper's own discussion states the principle directly: multi-agent coordination excels specifically when architectures are engineered to exploit parallelizable sub-problems or context protection, exactly what DeepDelve's Searcher/Analyzer split is built to do. A real caveat found on a fuller appendix read: the paper's own stated scope limitation is that it evaluates reasoning-heavy, closed-context tasks, not tool-heavy workflows where the bottleneck is external tool-call latency or protocol adherence, precisely DeepDelve's own actual bottleneck category, so "DeepDelve matches the winning pattern" is the best-supported reading but is an extrapolation across benchmark types, not a direct result.
So the task-division architecture itself isn't the evidenced problem. But the same paper's diagnostic methodology exposes a different, real risk: it identifies architectural bloat not from decomposition itself but from complexity added without verified causal contribution, mechanisms that cost real overhead but have near-zero measured influence on the outcome. This is the missing check for DeepDelve's own completion-check pipeline specifically, not the Planner/Searcher/Analyzer split: every mechanism added to that pipeline across many sessions has been validated only by "did the one specific symptom it was built for stop recurring in one live rerun," never by a controlled ablation (with versus without the mechanism, k≥3 trials each) the way MAST's own causal intervention evidence sets the bar for. A concrete, not-yet-implemented recommendation: once the reliability harness has enough data, run controlled ablations of a few of the more elaborate completion-check mechanisms to find out which are genuinely load-bearing versus expensive but unproven.
A first real result the next day confirmed two of them are load-bearing, using an adaptive-trial protocol (one run per condition first, escalating to more only when an early disagreement needs resolving). Disabling force_whole_rebuild dropped the mean score from a 0.75 baseline to 0.25 across 3 runs, both failing runs hitting the same underlying coordination failure the mechanism exists to break, a completion-check problem repeating 3+ times with the Planner only ever told to "acknowledge the gap" rather than genuinely change strategy. Disabling the no-progress guard dropped the mean to 0.125 across 2 runs, both timing out for two different specific reasons, a findings.md rebuild-reject loop in one and simple retry-budget exhaustion in the other, the guard's absence generically letting a run burn its whole time budget on unproductive retries regardless of which specific check triggers it. Two independently confirmed real contributions, a genuine result for the audit this section called for.
AgentFloor: How Far Up the Tool Use Ladder Can Small Open-Weight Models Go? (Karmakar and Chatterjee, arXiv:2605.00334, read in full, all 15 pages). A deterministic 30-task, six-tier capability ladder (from instruction-following up through long-horizon planning under persistent constraints), evaluated against 16 open-weight models and GPT-5, over 16,000 scored runs, native tool calling only.
Directly relevant to this project's own same-day disqualification of several candidates: sub-5B models clear 80-90%+ on the easiest tiers, but at the harder tiers, branching, multi-source synthesis, long-horizon planning, exactly where DeepDelve's real workload sits, no model in the entire corpus, including GPT-5, clears even a 60% reliability bar zero-shot.
Tip
This reframes the whole model search: the problem was never that any one local model is unusually weak. No current model handles this task class reliably zero-shot, real external validation that DeepDelve's heavy completion-check and retry architecture is a correct response to a genuine, externally measured capability gap, not overengineering.
Even more strikingly, "narrate/resign instead of execute" is GPT-5's own dominant failure mode on the hardest tier too, the model engages, sometimes partially executes, then stops without calling the required tool, structurally the same shape seen across nearly every disqualified DeepDelve candidate, just at a lower rate at the frontier, a general property of long-horizon agentic tasks that scale reduces but doesn't eliminate.
A directly actionable finding: a plan-then-execute-then-submit phase decomposition system prompt, the obvious-seeming fix for early resignation, regressed every single model tested, up to 33 points on one candidate, since the model complies faithfully with "plan before executing" and then emits a prose answer without ever entering the execute phase, the intervention meant to reduce resignation produces more of it. Checked against DeepDelve's own prompts: the Builder and FindingsWriter instructions contain a lighter-weight but structurally similar "deliberate before acting" block (<Show Your Thinking>) immediately before the exact tool call candidates have repeatedly narrated instead of calling. Not proven as a cause at the time, the paper's own harmful intervention was a much heavier, formally separated phase structure, but a plausible contributing factor worth a cheap, testable A/B.
That test was run the same session: a faithful isolated harness replayed FINDINGS_WRITER_INSTRUCTIONS against the real evidence base and write_workspace_file schema from a model already live-disqualified for narrating instead of writing, 9 reps with the block present versus stripped. Result: 9 of 9 real tool calls in both conditions, no difference at all. The block is not the cause, at least not in an isolated single-turn dispatch, a genuine negative result, not left open. This means the live disqualification failure, the same model and evidence narrating instead of writing during the actual multi-attempt benchmark run, comes from something specific to a full run's multi-turn or retry dynamics, accumulated context, quota pressure across several completion-check attempts, or conversation-length effects, not from this one prompt block in isolation. No code change made; the block stays as-is. If this failure mode gets root-caused later, the real variable to look at is what differs between an isolated first-shot dispatch (which converges cleanly) and attempt N of a real run (which doesn't), not the prompt content.
The paper's own design recommendation, route routine actions to small models and reserve frontier models for the narrow class that truly needs deep planning, matches DeepDelve's own role split in principle, but doesn't reopen the heterogeneous-tiering door closed earlier on VRAM contention grounds, that's a hardware fact independent of which small model is paired, not something this paper's findings change; it validates the direction was architecturally sound, just blocked here specifically by VRAM.
History
Model Research
Reviews & Audits
Reference