-
Notifications
You must be signed in to change notification settings - Fork 0
Changelog Findings
Part 2 of the Changelog, informational findings not yet acted on.
MAST's 14 mode failure taxonomy maps closely onto DeepDelve's own bug catalog, confirming these are known, published patterns, not idiosyncratic bugs. Three independent sources converge on "verification and architecture amplify a capable model, they don't rescue an incapable one." A third candidate mechanism for the "content vanishes during synthesis" pattern: models don't naturally allocate reasoning to synthesis, only to planning (PIVOT). See the Literature Review for full detail.
sub_agent_timeout_minutes can cut a dispatch off after a real URL is fetched but before the summary is written, leaving a findings entry with a real URL and zero content. A later retry sometimes produces real content but without re fetching, landing in a non citable fallback bucket instead, splitting real content and its real URL across two un-mergeable entries. Ground truthed against 107 run files on disk, not a one off. Fixed by ring fencing the timeout for a task showing real fetch activity, and excluding cutoff only summaries from the citable list.
Found DeepDelve's grounding pipeline more elaborate than any of them for post hoc citation verification specifically, not "most sophisticated deep research agent, period." See the Literature Review.
Done on 2026-07-18, re verifying all 12 README claimed guarantees against the actual code. All 12 genuinely implemented and reachable, no dead code, no always false gate. The README didn't overclaim.
A GOA (Grasshopper Optimization Algorithm) query got a citation about the Indian state of Goa instead, term overlap passed since it only checks "was this fetched" and "do terms overlap." Fixed 2026-07-14 by the topical relevance cross encoder check.
The fetcher doesn't execute JavaScript, so a JS-gated page just returns its challenge shell. Fixed for most cases (Springer, MDPI) via a headless/headed browser fallback. ScienceDirect resists both, a genuine Cloudflare Turnstile CDP fingerprint, deliberately not pursued further.
Across several runs, more than half of named sources were never actually fetched, recalled from training data, and independently fact checked figures tied to them were measurably wrong.
An excluded sector showed up in the final report anyway via a sibling task's tangential findings. Fixed via check_excluded_topic.
(a bare "Expert opinion from..." attribution with no URL). Fixed via the non URL citation check.
A 5 sector re run had more plausible looking citations, but cross referencing the real fetch log showed most were still fabricated.
The Colombia cultural context section vanished entirely from a report despite real research happening, since other tasks burned the shared quota first. Partially fixed via a ring fence for tasks showing real fetch activity; per task reserved quota and fairness ordering stayed open.
Invented function names fired 3 times in one run. Investigated further, no code fix needed, a generic tool result error nudge already covers the recovery.
Also observed inside Builder once the Builder architecture shipped, the same narrate instead of write failure, just one level down, once Builder's own write quota ran out. The quarantine restore fallback worked as designed both times.
Claim checks now compare each line's citation against its own source instead of the whole report's terms against every source, closing a mask where generic shared terms hid per claim fabrication.
A new eval_type: structural scores rubric tier 1 deterministically from the real run state, not narration.
_strip_trailing_punct didn't strip a trailing asterisk, false flagging real citations as hallucinated (Builder's own **[Title](URL)** format), burning half a run's retry budget on a checker bug, not a model failure. Sub agent "tool not found" errors had zero recovery path, since they come back as in band text, not exceptions, fixed via a new tool_result_error_nudge. web_search had no outer wall clock timeout, confirmed via a real 9 minute hang to an unrelated IP, fixed with a real daemon thread and join timeout. And sub agent status widgets had no staleness indication, showing "executing..." forever with no timer, fixed with an animated widget mirroring the existing pattern. A fifth, related fix the same day: Builder's write quota, shared with the Planner, could starve a later corrective dispatch, fixed with a dedicated headroom topup before every write/review/fix cycle.
History
Model Research
Reviews & Audits
Reference