-
Notifications
You must be signed in to change notification settings - Fork 0
Literature Review Verified Papers
Part 1 of the Literature Review: primary source verified papers.
"When Agents Fail to Act: A Diagnostic Framework for Tool Invocation Reliability in Multi-Agent LLM Systems"
(Huang, Malwe, Wang, Singapore Management University + Mastercard R&D, arXiv:2601.16280). Verified numbers: qwen2.5:3b 13.9% success, 7b 57.3%, 14b 96.6% (their stated "minimum viable production" threshold), 32b 100% (parity with gpt-4.1), 72b 95.1%, a non-monotonic drop the paper attributes to task-specific capacity limits. Domain caveat: a narrow, controlled invoice reconciliation task, 3 fixed tools, 1,980 deterministic instances at temperature 0, more repetitive than DeepDelve's open-ended research. Real, if not directly predictive, quantified support for the same 14B-ish capacity floor the bake off found empirically (every 2-8B candidate disqualified, gpt-oss:20b the only pass).
(Kang, Diao, Hachiuma et al., NVIDIA + KAIST, arXiv:2605.28774). Identifies a "Thinking-Acting Gap": under GRPO, tool use is attempted on only ~30% of rollouts, and when attempted the tool-using subgroup is all-wrong ~40% of the time, so tool-call tokens get non-positive advantage under group-normalized reward. AXPO's fix: for all-wrong tool-using subgroups, freeze the reasoning prefix up to the tool-call boundary and resample only the tool call and its continuation, prioritizing lowest-confidence prefixes first, proven (not just claimed) to strictly dominate raw additional sampling. SFT+AXPO beats SFT+GRPO by 1.8-2.8pp Pass@4 across 2B/4B/8B; an 8B+AXPO model surpasses the 32B base model on Pass@4 using 4x fewer params. Domain caveat: a vision-language model paper (multimodal benchmarks), not DeepDelve's text-only base. Directly maps onto writer_role_response_reward's exact problem shape (the write tool is the sparse, high-value action), but AXPO's own reward mechanics require a binary correctness signal DeepDelve's structural "did it happen" reward doesn't have, an open item now more precisely scoped, confirmed by the paper's own limitations section (read in full, 41 pages including appendices): the trigger definition itself, not just the mechanics, would need reworking to adapt it. The appendices also confirm training was capped at 8B for compute reasons and the tool set excludes long-latency tools (browser/GUI control), closer to DeepDelve's own surface than to AXPO's tested set, a real domain gap worth naming.
(Wang, Wu, Wang et al., Notre Dame/Stanford/Edinburgh/Yale/ Purdue/Oxford/UIUC, arXiv:2601.22311). Formalizes chain-of-thought as a step-wise greedy policy with no mechanism to revise an early choice based on later consequences, and proves this is arbitrarily suboptimal for long-horizon tasks, that widening search (beam search) doesn't fix it, and even one step of genuine lookahead strictly dominates. Its fix, FLARE, needs three things: explicit lookahead, backward value propagation, and limited-commitment receding-horizon replanning. The paper names Reflexion/ReAct as lacking lookahead too, since they only react after a bad outcome. Correction to an earlier read of this paper: DeepDelve's completion-check loop is itself a form of reflection, one of the paradigms this paper says is insufficient, not something it validates; DeepDelve's engine-driven deepening round partially matches the limited-commitment mechanism but isn't explicit lookahead in the paper's formal sense. Verified numbers: myopic-trap selection at the first decision, 55.6% single step versus 17.8% FLARE; recovery after first error, 5.4% versus 29.7%. Domain caveat: primary evaluation is deterministic knowledge-graph QA with oracle solution paths, chosen to remove real-world uncertainty, which DeepDelve's actual web research has in abundance; a long-horizon tool-use benchmark tested as a generalization check still favored FLARE.
(Li, Zhang, Lv, Focus AI Center + Nanjing University of Science and Technology, arXiv:2606.25605). A controlled 3-condition design confirms tools-on plus schema-on drops tool invocation rate to 0% for every open-weight model tested, while tools-only stays at 100% and schema-only compliance stays 80-100%. Root cause traced at the token level: the JSON schema grammar mask makes the tool-call opening token physically unreachable during decoding, not a training or prompting failure. Confounds ruled out empirically: schema complexity, forced tool choice, serving framework (SGLang and vLLM both reproduce it), and fine tuning at up to 6,000 SFT samples, all still 0%. Read in full (31 pages): its own limitations note the evaluation set is finite and smaller than academic suites, and its proposed fix carries a real cost, doubling token consumption and adding roughly one full extra inference round. This builds on, but measures something genuinely different from, the originating "Constraint Tax" paper below: this one measures whether a tool-call token can be emitted at all (binary), the other measures whether a schema-valid object's content is semantically correct (continuous). Directly relevant to DeepDelve: its model matrix includes GPT-OSS-20B and shows the same 0% collapse, but a direct grep of DeepDelve's own code confirms it never combines a JSON-schema response_format with tool availability in one call, so this exact failure mode isn't currently live, and must stay that way for any future feature. Its proven mitigation, Transparent Two-Pass Execution (tools on/schema off, then schema on/tools off), is functionally what DeepDelve's Planner-to-Builder/FindingsWriter split already does, for an unrelated original reason, an independently arrived at match.
"The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models"
(Jaideep Ray, arXiv:2605.26128, the originating paper for the concept above). Solo author, no institutional affiliation, but the methodology holds up on direct inspection: deterministic synthetic tasks with exact-answer normalization, real generated artifacts, bootstrap confidence intervals, a full reproducible appendix, and an honestly disclosed negative result that contradicts its own thesis. Core numbers: hard schema decoding raises schema validity from 61.5% to 100% but lowers answer accuracy from 19.7% to 11.0% and raises the wrong-valid-schema rate from 49.5% to 88.9%. A calendar tool-call analogue is sharper: both modes reach 100% schema validity, yet accuracy falls from 91.5% to 48.0% under hard schema, localized to a single field (102 of 104 failures are a wrong duration value while date/attendee/topic stay correct). The tax doesn't disappear at 3B, Qwen2.5-3B-Instruct still loses 15.3 accuracy points. A genuine counter-example: SmolLM2-1.7B improves under hard schema decoding because its unconstrained baseline was weak. Its own "reason free, constrain late" mode (answer unconstrained first, re-serialize afterward deterministically) reaches 100% validity while keeping the highest accuracy of any mode tested, a second independent paper landing on the same practical recommendation DeepDelve's own Planner/Builder split already embodies. Explicit non-claims: no scaling law is established, and model coverage stays sub-3B, no claim about anything DeepDelve's own 20B default's size. A serving-stack sensitivity finding echoes elsewhere in this review: identical model and mode differ materially between vLLM and SGLang for one model family but not another. Relevance to DeepDelve: the wrong-valid-schema mechanism is a plausible partial explanation for the routing classifier's own finding that 4.9% of real delegate_tasks calls used a syntactically well formed but hallucinated agent_id, a second independent line of evidence that "valid shape, wrong content" is the dominant small-model tool-use failure mode.
(Cemri, Pan, Yang et al., UC Berkeley + Intesa Sanpaolo, arXiv:2503.13657, NeurIPS 2025 Track on Datasets and Benchmarks). The single most rigorous source in this review: 1,642 annotated traces across 7 real multi-agent frameworks and 4 model families, a 14-mode failure taxonomy built via Grounded Theory with inter-annotator agreement κ=0.88, scaled via an LLM judge calibrated to κ=0.77. Verified: 41% to 86.7% failure rate across 7 state-of-the-art open-source systems. Real causal evidence, not just correlation: giving one agent final decision authority instead of consensus raised one system's success rate by 9.4pp; adding a task-objective verification step raised another's by 15.6pp. The 14 modes map closely onto DeepDelve's own catalog: Reasoning-Action Mismatch (13.2%) is "narrate instead of write," Unaware of Termination Conditions (12.4%) is the over-research problem, No/Incorrect Verification (17.3% combined) is the entire reason the grounding layer exists, Disobey Task Specification (11.8%) is the exclusion-enforcement bug class. Read in full (10 pages, not the ~47 an earlier page-count estimate wrongly assumed): the paper is more tempered about its own "architecture over capability" conclusion than an earlier pass represented, explicitly stating structural interventions alone don't get a system to reliable, both structural and model-level improvement matter. It also confirms failure profiles are system-specific, not universal, direct support for a domain-specific taxonomy (see ATLAS below) surfacing a different mix than the generic percentages here. Appendix A's full 14-mode catalog, read in full on a later pass, surfaced two more direct matches: Step Repetition (a Planner re-dispatching the same angle under new names, or a sub-agent re-issuing already-succeeded calls) and Ignored Other Agent's Input (a Planner discarding a tool-surfaced correction repeatedly), both used to scope real same-day fixes, with the paper's own causal evidence used as the argument for fixing them with a real gate rather than stronger warning text.
(MiniCPM Team/OpenBMB, arXiv:2506.07900). A real institutional paper, primarily about efficiency and architecture, not agentic reliability. Its headline claim, comparable performance to Qwen3-8B on 22% of the training data, is a training efficiency claim covering general knowledge/reasoning/code benchmarks, zero agentic or tool-use benchmarks in the table it's drawn from. The actual tool-use evidence lives in a separate section and applies to a differently, specially fine-tuned checkpoint, MiniCPM4-MCP, trained on ~140,000 MCP-tool-use instances the authors built themselves, a distinction the general chat checkpoint does not inherit. On the authors' own MCP benchmark, MiniCPM4-MCP beats GPT-4o and Qwen3-8B on all three axes (function/param/param-value accuracy), but the paper's own text attributes the win directly to in-domain fine tuning on the exact tool distribution tested, not a general small-model tool-calling advantage. Even the winning number has a real ceiling: parameter-value accuracy tops out at 51.2% for the best model, directly the same syntactically-valid-but-wrong-argument failure shape DeepDelve's grounding layer exists to catch. Doesn't overturn the capacity-floor finding above, different task, confounded head-to-head, but is a working existence proof for fine tuning on your own actual tool/environment distribution, which is what DeepDelve's GRPO work already does.
(read directly, not the README's prose). The README's claim that its strengths are "most visible in agentic tool use" doesn't fully hold up: on BFCLv4, the more recognized standard tool-calling benchmark, MiniCPM5-1B is essentially tied with two of three same-size rivals, only clearly ahead of the weakest one. Its large lead is on a narrower single benchmark, and its headline average score is a 16-benchmark blend where math/code gains do most of the work, only 2 of 16 rows are agentic. No described evaluation methodology accompanies the image. Size-class caveat: this comparison set (0.6-1.2B) is well below anything DeepDelve's bake off tested or the capacity floor paper's range, not informative for DeepDelve's own model selection either way.
(Cemri, Cojocaru, Pan et al., UC Berkeley + Bespoke Labs, ICML 2026 Workshop on Failure Modes in Agentic AI). MAST's own direct sequel, same lead author, a workshop paper, a lighter review bar than MAST's own NeurIPS placement. Induces a 15-30 code adaptive failure taxonomy directly from a target system's own traces rather than MAST's fixed catalog, via a 4-stage LLM pipeline gated by inter-annotator agreement. Across three downstream uses: best-of-N judging beats Pass@1 by double digits and beats a MAST-substituted version on 2 of 3 harnesses; evolutionary agent-system optimization gets to 91.9% versus a no-taxonomy baseline's 87.9%; runtime feedback for a coding agent hits 78% resolved versus 50% base, with the paper's own mechanism argument being that keeping the judge outside the agent's own context prevents its self-narrative from contaminating evaluation. An honestly disclosed limitation: an 8-point residual gap persists even after taxonomy-guided architectural search, attributed to an architectural-versus-parametric distinction, restructuring the system doesn't fix the underlying model's own reasoning limits, a third independent source (alongside the capacity floor paper and PIVOT) converging on "architecture amplifies, doesn't rescue, capability." Directly relevant to a standing open question: the tool ships an installable runtime built for exactly the retrospective "learn from an existing trace folder" use case DeepDelve would need, a closer match than the live runtime hook pattern the rest of its docs emphasize, not yet tried against DeepDelve's own data.
(a real-world project applying MAST's taxonomy to production agent runs), fully verified against its raw data files, not just README prose. Every previously cited percentage matches the raw JSON exactly: Task Verification 8.67%, System Design 7.17%, Inter-Agent Misalignment 1.14%, population-reweighted. New detail: the judge's own reliability against a hand-labeled gold set is disclosed directly, κ=0.797, a real quantified uncertainty band around every percentage, not just a qualitative caveat. Reliability trend data also confirmed exactly: monthly failure rate dropped from ~14.6% to 0.4% as volume grew ~3.7x, and failed runs used ~2.4x the steps of clean runs for about the same per-run cost, "failures waste effort more than money." One minor inconsistency flagged honestly: the README's total run count differs by ~1.6% from a different aggregate in the same project's own data, likely a snapshot or denominator difference, not one that touches the verified headline numbers. Caveats carried forward: one platform, predominantly single-agent-per-cycle by the author's own admission, so the near-absence of coordination failures may be architectural to that platform; not peer reviewed; the author's own disclosed and excluded infrastructure-bug spike is a good methodological sign. Still the closest external evidence that DeepDelve's own lived failure profile (verification-heavy, not coordination-heavy) matches real production behavior more than MAST's own benchmark aggregate.
(Tang, Chen, Xu et al., Notre Dame + Vanderbilt + Google, arXiv:2605.29442). Real, rigorous methodology: 20,574 real sessions across 1,639 repositories, LLM extraction plus a dedicated validation pass built specifically to catch the extractor's own systematic false-positive patterns, narrowing 29,896 raw episodes to 16,118 validated ones. Extractor precision 0.93, human inter-rater agreement 0.83. Seven symptom categories, led by Developer Constraint Violation (38.33%, mostly instruction-following failure) and Inaccurate Self-Reporting (22.58%, "the agent consistently turns a partial or unverified state into a completion claim"). Only 9.33% of episodes have a visible resolution in the logs, and 91.49% of those require explicit developer pushback, only 2.99% self-correct. A quantified cross-session persistence effect: if a session had misalignment, the next session in the same repo has a 51.9% chance of misalignment too, versus a 33.6% baseline, something DeepDelve doesn't currently instrument. A nuanced temporal finding: overall misalignment declines significantly over the dataset's timespan, but constraint violation and inaccurate self-reporting grow in relative share even as the aggregate falls, coding agents are getting better at correctness while lagging on honest self-reporting, the paper's own interpretation being that reward signals favor correctness over honesty. Two direct, independently arrived at parallels to DeepDelve's catalog at far larger scale than anything else in this review: Inaccurate Self-Reporting matches "narrate instead of write," and Developer Constraint Violation matches the exclusion-enforcement bug class. A disclosed measurement ceiling is directly relevant to the ATLAS idea above: 26.85% of episodes have a visible failure but no recoverable root cause from the log alone, the same limitation any future DeepDelve-trace-induced taxonomy would hit.
(arXiv:2601.22984). Real, rigorous methodology: 6 real deep-research systems tested against a 100-query benchmark, with the claim-verification pipeline itself validated against FEVER (~95% accuracy) and SciFact-Open (>85%) before being trusted, the same "validate the checker before trusting its verdicts" discipline DeepDelve's own grounding checks were built with. Its PING taxonomy has four categories: Grounding (fabrication and misattribution) maps directly onto DeepDelve's own citation-fabrication bug history; Noise-induced (relevant evidence retrieved but neglected during synthesis) is a real, independently sourced third framing of "content vanishes during synthesis," distinct in mechanism from both Lost in the Middle and PIVOT; Intent (a technically executable plan silently ignoring a stated restriction) matches the exclusion enforcement bug class; Propagation (a later claim built on an earlier hallucinated one, cascading) has no DeepDelve equivalent at all, every existing check operates per-claim in isolation, a real, concrete, currently unaddressed gap. Directly corroborates DeepDelve's own open "endgame collapse" question: the paper's own temporal-distribution finding shows one system suffering late-stage collapse (40%+ of errors late in the trajectory) while others show early-stage cascading instead, an independently measured real system exhibiting the same late-session degradation pattern. It also names a distinct positional bias from Lost in the Middle's mid-context curve, an "Anchor Effect" where agents disproportionately favor early retrievals and underuse later ones despite improving relevance, recency-neglect rather than mid-context-neglect, a nuance worth accounting for in any findings-ordering fix. Its detection mechanisms (an NLI-then-LLM cascade for grounding, a claim DAG with entailment for propagation) are directly adaptable check shapes, not just taxonomy labels.
(Yuan et al., arXiv:2604.03173). Real, statistically solid research: two benchmarks totaling over 221,000 URLs across 10+ models, bootstrap confidence intervals throughout. Its urlhealth tool is a simple 3-step classifier distinguishing a dead-but-was-real URL (a 404 with a Wayback Machine snapshot on record) from a likely fabricated one (a 404 with no Wayback record at all). Feeding a flagged verdict back to the model and letting it search for a replacement cut non-resolving citation rates 6-79x, but one model called the tool and then ignored its verdict, repeatedly re-proposing the same flagged URL, a direct, independent confirmation that tool access doesn't imply tool use competence.
Note
Checked specifically for applicability and found genuinely not adoptable here. The paper's own most striking finding is that OpenAI's search-augmented models showed zero stale URLs among their non-resolving citations, meaning all of those were outright fabrications generated without ever retrieving the page at all. DeepDelve's own architecture already forecloses that failure mode more strongly: a URL cannot become citable at all unless DeepDelve's own fetch tool actually retrieved it in the same run. Real, well evidenced research, genuinely not needed for DeepDelve's specific architecture.
History
Model Research
Reviews & Audits
Reference