Research Small Model Reliability

Why DeepDelve Is Hard on Small Models — Research Report

2026-08-27. Written after a 3-candidate bake-off this session (nemotron-3-nano:4b, gemma4:12b, InternScience/Agents-A1-4B) all failed against the same complex Colombia B2B benchmark item. Full run-by-run detail lives in session_status/2026-08-26.md / CURRENT.md; this report answers why, whether DeepDelve's own design contributes, and what to try next.

Executive summary

DeepDelve's workload sits at the hardest tier the agentic-benchmark literature has identified: long-horizon, multi-step planning with persistent constraint-tracking and self-correction under pressure — a regime where even frontier models don't yet reach confident-deployment reliability, per a primary-source read of AgentFloor (arXiv:2605.00334, read in full, all 15 pages, directly — not summarized). That reframes this session's 3/3 failures: they are not proof these specific models are broadly incapable, they are consistent with a known, published capability ceiling that is somewhat independent of raw parameter count. Separately, and more actionably, this investigation found one concrete, well-evidenced DeepDelve-side bug: a QuotaAbortException (the "trapped in loop, quota exceeded for think_tool" abort that killed one of tonight's runs) skips the exact salvage/ completion-check path a near-identical malformed-tool-call failure was explicitly fixed to fall through to — meaning a real research run's fetched sources are currently thrown away on this specific abort path with no recovery attempt at all. That's a small, low-risk fix. A prompt-length reduction, by contrast, is NOT well-supported as a fix — the same primary literature found a "natural" structured-decomposition prompt simplification degraded performance across every tested model, and effects were model-specific, not universal. For candidates: the AgentFloor paper's own corpus found gemma4:26b the strongest open-weight model overall (already hardware-disqualified here), and BFCL v4 multi-turn results point to two purpose-built function-calling small models — xLAM-2-8b-fc-r and EGPO-4B — worth testing, though both need verification before assuming ROCm/ollama fit the way this session's candidates did.

---

Question 1: What about DeepDelve's own design makes this hard for smaller models?

Prompt weight

Per-role system-prompt sizes measured directly from src/prompts.py this session:

Role	Tokens (approx)
PLANNER_INSTRUCTIONS	~2,736
WEB_SEARCHER_INSTRUCTIONS	~2,132
ACADEMIC_SEARCHER_INSTRUCTIONS	~2,183
FINDINGS_WRITER_INSTRUCTIONS	~1,612
PEER_REVIEWER_INSTRUCTIONS	~1,195
DATA_ANALYZER_INSTRUCTIONS	~1,141
BUILDER_INSTRUCTIONS	~1,057
DOCUMENT_ANALYZER_INSTRUCTIONS	~944
SUBAGENT_DELEGATION_INSTRUCTIONS (appended to every role)	~342

The Planner's own system prompt alone is ~3,078 tokens once the delegation block is appended — about 19% of a 16K-token context budget before any tool schema, user query, or conversation history. That's real weight, but not obviously disqualifying on its own: this session's disqualified candidates were all run at their FULL native context (up to 262,144 tokens), not the 16K floor, so raw context exhaustion was never the observed failure mode in any of the 9 original + 9 retest runs — every failure was behavioral (fabrication, stuck loops, zero tool calls) or wall-clock speed, not "ran out of room." Verdict: prompt length is a plausible contributing factor to instruction-following degradation on weaker models, but the evidence from tonight's runs does not show it as the proximate cause of any specific observed failure. See the AgentFloor finding below for why a length-only story is likely incomplete anyway.

Retry-loop/verdict variety

src/engine/completion_checks.py currently defines 27 distinct problem/verdict types (grep -oP 'Verdict\(\s*\n?\s*"\K[a-z_]+' — full list: claim_unsupported, cross_source_contradiction, duplicate_report_sections, excluded_topic_present, findings_underuses_evidence, findings_ungrounded, missing_artifact, missing_findings, nli_unsupported, non_url_citation, not_delegated, not_grounded, propagated_ungrounded, quote_paraphrased, regulation_unsupported, report_style_violation, report_underuses_evidence, report_underuses_findings, specific_figure_unsupported, stale_findings, stub_source, task_verification_flagged, thin_coverage, topical_mismatch, uncited_claims, uneven_task_investment, untracked_delegation). Each carries its own differently-worded corrective SYSTEM WARNING. Layered on top: force_whole_rebuild's full-rewrite escalation, _with_wait_prefix's "Wait." self-reflection cue, and a distinct-problem attempt-bonus mechanism (+1 extra attempt per genuinely different consecutive problem, capped at 4). This is a genuinely large and structurally sophisticated state machine for a model to track across a long run. Tonight's Agents-A1-4B run 1 worked through 9 completion-check attempts hitting 5+ distinct problem types in sequence (missing_findings → missing_artifact → untracked_delegation → stub_source → non_url_citation → report_underuses_findings) and still didn't converge in time — not because it got confused by the variety (each individual correction was followed correctly, in sequence), but because there was simply a lot of REAL, sequential work to do. This argues the verdict-variety itself is not obviously the bottleneck for a model that's otherwise behaving — but see the task_verification_flagged/force_whole_rebuild interaction below, where a specific one of these 27 verdict types repeatedly defeated the strongest available escalation.

Multi-agent nesting depth

8 named sub-agent roles (DocumentAnalyzer, DataAnalyzer, WebSearcher, AcademicSearcher, PeerReviewer, Builder, FindingsWriter) plus the Planner itself — a real query can nest Planner → WebSearcher/AcademicSearcher → DocumentAnalyzer/DataAnalyzer (a Tier-2/3 delegation chain), with the Write→Review→Fix loop (Builder/FindingsWriter + PeerReviewer) layered independently on top, each dispatch a fresh-context LLM call. This is architecturally similar to what the MAST taxonomy (Cemri et al., NeurIPS 2025 — already cited in this project's own src/deepdelve.egg-info/PKG-INFO and src/tools/core.py/src/engine/orchestrator.py code comments, not independently re-verified in full this session, treated as a secondary citation here) documents as a coordination-failure risk surface distinct from single-agent capability: FM-1.3 "step repetition" and FM-2.5 (referenced directly in src/tools/core.py:58's no-progress guard and src/engine/orchestrator.py:1828's rename-reject-escalation logic) are BOTH failure modes this project has already built structural defenses against, meaning this project's own history already independently rediscovered that multi-agent nesting produces exactly the loop/ repetition failure shapes tonight's candidates hit.

Cross-reference to tonight's 3 observed failure shapes

Observed failure	Best-fit category
Zero real tool calls the entire run (nemotron-3-nano run 2)	Instruction-following collapse under system-prompt load, OR a variant of MAST's "premature termination"
Task-level fabrication surviving force_whole_rebuild (gemma4:12b run 1, both Agents-A1-4B runs)	MAST's failure-to-verify / fabrication-under-pressure class; also the exact shape check_task_verification_flagged's own docstring documents as previously observed live on gpt-oss AND Ornith-1.0-9B, 2026-07-31 — i.e. NOT unique to small models
think_tool spam loop, DeepDelve's own safety net aborted it	MAST FM-1.3 "step repetition" — the exact failure class src/tools/core.py's tool-failure-streak guard was purpose-built to catch, and did catch here (see Question 2)
Ollama XML tool-call parser crash (nemotron-3-nano run 3)	Not a model-capability failure at all — a serving-layer bug, per this project's own Model Evaluation Standard point 3

Notably, the fabrication-under-pressure failure (row 2) is explicitly documented in this project's own code as having already happened to gpt-oss:20b itself — the current, larger baseline model. That is the single strongest piece of evidence that this session's disqualified candidates are not uniquely bad; they hit the same wall the incumbent has also hit, just perhaps more often per unit of time given less headroom to recover.

---

Question 2: Can DeepDelve be changed to reduce this risk?

Finding A (concrete, high-confidence): the QuotaAbortException path skips salvage entirely

File: src/engine/tui.py, two call sites — run_cli (~line 2676-2678) and run_agent (~line 1586-1589). Both catch QuotaAbortException and do:

has_requests = False
break

This is a real bug, not a hypothesis. run_completion_check — the function that owns EVERY salvage/quarantine/narration-recovery path this project has (including _salvage_narrated_report, the exact mechanism that rescued Agents-A1-4B's genuinely good run 1 report tonight) — is called from inside the while has_requests: loop, gated on if not has_requests: (confirmed directly in run_cli, line ~2757 onward). break exits the loop immediately, on the SAME iteration, before that check is ever reached. Compare this to the malformed-tool-call-retry-exhaustion branch a few lines below in the exact same function, which has an explicit code comment specifically warning against this shape: "Deliberately does NOT continue here (that would skip straight past the while has_requests: check with has_requests already False, exiting the loop WITHOUT ever calling run_completion_check — meaning the 'finishing with whatever exists' message below would be a lie)." The QuotaAbortException handler makes exactly the mistake that comment warns against. Confirmed live tonight: the think_tool-loop- aborted run had fetched 12 real sources and made zero attempt to write findings.md/final_report.md from them — "Report: NOT WRITTEN" — with total data loss, when a salvage attempt (or at minimum the standard "unresolved issue, treat as unconfirmed" notification) was structurally available and simply never reached.

Proposed fix: replace the break in both handlers with the same fall-through pattern the malformed-tool-call branch already uses — set has_requests = False and let the current loop iteration finish naturally (falling into the if not has_requests: block) instead of jumping out. This doesn't fix the underlying loop-detection trigger itself (that's working as designed — it correctly caught the spam), it just stops throwing away real, already-fetched research when it fires. This benefits EVERY model, not just smaller ones, but a model prone to think_tool spam loops (as tonight's Agents-A1-4B run 3 was) benefits disproportionately, since it hits this path more often.

Finding B (concrete, medium-confidence): task_verification_flagged's escalation ladder has no cheaper fallback

check_task_verification_flagged (src/engine/completion_checks.py:210) already has a documented 3-strike ladder (redo → acknowledge-and-stop → force_whole_rebuild's one-shot full rewrite), and tonight confirmed that even the FULL rewrite doesn't always resolve a persistently-fabricating task (gemma4:12b run 1: 3 flags, full rebuild, still failed; Agents-A1-4B run 2: 6 flags across 8 attempts, full rebuild fired, still failed). A model-capability-agnostic ladder currently treats every model identically regardless of how much budget it's already burned unsuccessfully. A config-gated earlier exit (e.g. settings.task_verification_early_gap_threshold, defaulting to the current 3 strikes, settable lower) would let a known-weaker candidate accept an acknowledged gap sooner rather than burn its whole attempt budget on a task that historical evidence (its own whole_approach_retry_used_for ledger) shows it isn't going to fix. This is a genuinely small diff (one new config key threaded into the existing _consecutive_occurrences comparison already in that function) but the evidence for its benefit is weaker than Finding A — it trades completeness for survivability, and might just produce more incomplete-but-honest reports rather than more complete ones.

Finding C (evidence AGAINST a "lite prompt" fix): prompt-length reduction is not well-supported

The AgentFloor paper (arXiv:2605.00334, read directly, pp.1-3) found — in a controlled ablation across 16 open-weight models — that "a seemingly natural structured decomposition prompt degrades performance across all tested models," and that interventions which helped one model were "typically null on others." This is a direct, primary-source counter-argument to the intuition that simplifying DeepDelve's large role-instruction blocks would reliably help smaller models: the literature's own controlled test of a similar-in-spirit intervention (restructuring/decomposing prompt guidance) made things WORSE broadly, not better. Recommendation: do not prototype a "lite" prompt variant on the assumption that shorter/simpler instructions are safely net-positive — if attempted at all, it needs the same live A/B rigor this project already applies to model candidates (Model Evaluation Standard-style, isolated variable, k≥3 runs), not a one-shot assumption.

Finding D (weaker than first written — corrected 2026-08-27 after a full read of the paper)

Correction: this finding originally claimed AgentFloor's "step-budget ×2" ablation was convergent evidence that budget/step-count is "one of the few interventions with a broadly positive effect." Having now read the full paper (previously only pp.1-3 were read; see Sources), that overstates it. The 9%→44% figure (Fig. 1's annotation) is a single-task effect: doubling GPT-5's max_steps on tier D/E moved only task E1 (1/9→4/9 runs, +33pp [0,+67]) — the tier-E aggregate barely moved at all (+4pp [-7,+16], CI crossing zero), and the paper's own trace inspection attributes it to E1 specifically having "a genuinely tight" declared step budget, not a general step-budget-starvation pattern. The paper groups this alongside the explicit-submission prompt as one of three model-specific lifts (Section 5.2), explicitly contrasted with the structured-decomposition prompt, which was the sweep's one uniform effect (negative, for every model tested). AgentFloor's own conclusion is "interventions that help one model are typically null on others" — it does not support budget increases as a broadly generalizing lever.

This does NOT undermine this session's own max_completion_check_attempts 3→10 fix, which was validated directly by this project's own live runs (fewer stuck-loops, more varied real progress) independent of any external literature. It only means that fix should not be framed as backed by convergent published evidence of a general "more budget helps" principle — AgentFloor's actual finding is closer to "no intervention we tried generalizes broadly, including budget," which is consistent with, not contradictory to, this project's result being one narrow case that happened to work.

---

Question 3: Candidate models, grounded in benchmarks

Searched BFCL (Berkeley Function-Calling Leaderboard) v3/v4 results and the AgentFloor paper directly. Read AgentFloor's abstract/intro/methods pages primarily (pp.1-3); BFCL numbers below are relayed from a WebSearch summary, not independently verified against the primary leaderboard page — flagged accordingly, treat as a lead to confirm, not settled fact.

Candidate	Size	Why it fits (multi-turn/self-correction specifically)	ROCm/AMD status	First validation step
gemma4:26b	3.8B active / MoE	AgentFloor's own corpus found this the single BEST open-weight model overall, matching GPT-5 at the aggregate level (primary source, read directly)	Already disqualified this session on pure VRAM grounds (18-19GB even at its lightest quant, exceeds this machine's 17GB budget) — not re-recommendable without more VRAM	N/A unless hardware changes
Llama-xLAM-2-8b-fc-r (Salesforce)	8B	Relayed BFCL v3 multi-turn score ~69.25%, reported to exceed GPT-4o (~47.62%) on multi-turn specifically — the exact capability tonight's candidates lacked. Purpose-built for function-calling, Llama-architecture base	GGUF available directly on HuggingFace (Salesforce/Llama-xLAM-2-8b-fc-r-gguf) — same hf.co/ pull pattern used for Agents-A1-4B tonight, should work the same way, not yet confirmed live	Point-1 raw checks: think:false on native + OpenAI-compat paths, raw tool-call format test, ollama ps VRAM/context check across the range — exact same 4-step check this session ran on every candidate
EGPO-4B	4B	Relayed BFCL v3 multi-turn Overall ~56.25%, reported to beat GPT-4o (~42.50%) and Grok-4 (~36.12%) on multi-turn specifically, at only 4B	Not found as a packaged GGUF/Ollama-pullable checkpoint in this search — likely a research-only release; verify a real downloadable checkpoint exists before investing further	Confirm a GGUF/HF checkpoint exists at all before any other step
gemma4:e4b	4.5B effective	Already in this project's own original backlog (untested), same family as the AgentFloor-strongest gemma4:26b but at a size that fits this hardware	Not tested this session; gemma4:12b's throughput surprise (32.2 tok/s, slower than gpt-oss:20b despite fewer params) means this family's ROCm performance should NOT be assumed fast just because it's smaller — verify raw tok/s FIRST, before any full benchmark run	Raw throughput test (/api/generate, eval_count/eval_duration) BEFORE any live benchmark — this session's own gemma4:12b experience shows this family's ROCm speed is not predictable from parameter count alone

Explicitly not recommended without further verification: nothing new from the vLLM ecosystem — per this project's own prior history (session_status/2026-07-26.md era, referenced but not re-read in full this session), vLLM+bitsandbytes on this exact ROCm/consumer-GPU combination was already disqualified across 9 candidates as too immature; that finding was not re-litigated here and should stand unless something has materially changed upstream.

Overall ranked recommendation for next test: Llama-xLAM-2-8b-fc-r first (real checkpoint exists, purpose-built for the exact multi-turn tool-calling reliability gap observed tonight, same proven hf.co/ pull path as Agents-A1-4B) → gemma4:e4b second (untested backlog item, but verify raw throughput before committing real benchmark time, per gemma4:12b's surprise result) → EGPO-4B only if a real downloadable checkpoint can be confirmed to exist.

---

Sources

• src/prompts.py, src/engine/completion.py, src/engine/completion_checks.py, src/engine/orchestrator.py, src/tools/core.py, src/engine/tui.py, src/app.py, src/deepdelve.egg-info/PKG-INFO, RESEARCH.md, ROADMAP.md — all read directly this session (primary source: this repository).
• session_status/2026-08-26.md, session_status/CURRENT.md — read in full, primary source for every run number cited above.
• AgentFloor: How Far Up the Tool Use Ladder Can Small Open-Weight Models Go? — arXiv:2605.00334 (Karmakar & Chatterjee, Harvard, May 2026). Read directly, full text, all 15 pages (confirmed real page count via pdfinfo on papers/agentfloor_2605.00334.pdf), including Results, Section 5 ("Where the long-horizon gap moves"), Limitations, and Conclusion — completed 2026-08-27, correcting the prior session's partial pp.1-3-only read (see the Finding D correction above, which the full read surfaced). https://arxiv.org/pdf/2605.00334
• MAST (Cemri et al., NeurIPS 2025) — cited here as a SECONDARY reference via this project's own existing code comments (src/tools/core.py, src/engine/orchestrator.py, src/deepdelve.egg-info/PKG-INFO), not independently re-read this session. The full paper is reportedly already reviewed elsewhere in this project's wiki literature review (https://github.com/g0elles/deepdelve/wiki/Literature-Review-Architecture-Synthesis, not re-fetched this session).
• WebSearch results (titles/summaries only, not independently verified against primary sources): Berkeley Function Calling Leaderboard v3/v4 general pages (gorilla.cs.berkeley.edu/leaderboard.html, github.com/ShishirPatil/gorilla), a BFCL v3 multi-turn summary via emergentmind.com, and Salesforce's Llama-xLAM-2-8b-fc-r-gguf HuggingFace listing (https://huggingface.co/Salesforce/Llama-xLAM-2-8b-fc-r-gguf). The specific BFCL multi-turn percentage figures for xLAM-2-8b-fc-r and EGPO-4B in the table above are RELAYED from search summaries, not confirmed against the primary leaderboard page directly — treat as a lead worth verifying at gorilla.cs.berkeley.edu/leaderboard.html before citing further.
• session_status/2026-07-26.md (vLLM+ROCm disqualification history) — referenced from memory of this project's own prior session archives, not re-read in full this session; flagged as a lead to re-confirm if vLLM is ever reconsidered.