-
Notifications
You must be signed in to change notification settings - Fork 0
Changelog Recent
Part 1 of the Changelog, most recent entries first. Continues into Part II.
2026-08-18 through 2026-08-19: the coordination-layer ablation study closed, plus a real Searcher/Analyzer root cause fix
Closed the C.8 controlled-ablation study started 2026-08-17 (full detail in the
Literature Review's ablation section). The two remaining
mechanisms, rename_reject_escalation and tool_failure_streak_guard, deliberately did not get
their own ablation run: both were built in direct response to a real live incident already, so their
load-bearing-ness already had independent evidence, and any further run was confounded anyway by a
same-day config change (below). The study's actual goal, MAST/Illusion-of-Multi-Agent-Advantage
style causal validation of coordination-layer complexity, was satisfied by the two mechanisms that
were tested.
The single biggest quality lever found in that same ablation data: the Searcher/Analyzer instance of the "zero trailing text" mechanism this project already tracked for writer roles (a dispatch calls a tool then ends its turn with no narration at all) turned out to be the original, longest-standing case of the pattern, and had only ever been detected downstream, never actually recovered. Auditing one ablation run's own workspace found real, freshly fetched sources landing with a completely empty finding summary at rates from 60 to 100 percent on several tasks, not a rare edge case, a major, previously invisible tax on real research capacity, since the downstream exclusion made it look like "no sources found" rather than "sources found, never synthesized." Fixed by granting one extra turn with a synthesis nudge whenever a research-tier dispatch called a tool but produced zero text, gated to Searcher/Analyzer roles specifically since writer roles already have their own mechanism.
A separate fork analysis mining the same 6 ablation workspaces for problems beyond the two already- fixed bugs found every visa or regulatory topic task hitting the per-task delegation cap and burning turns on rejected retries instead of ever completing real analysis. The cap had been deliberately tightened from 4 to 3 on 2026-07-26 for the opposite reason, a trivial single-fact query over- delegating 6+ Analyzers, so bumping it back to 4 was a real trade-off, not a free fix, and confounds every already-recorded ablation result run at the old cap. A dedicated regression check confirmed the bump didn't reopen the original over-delegation problem it was tightened for.
Separately, the citation-misattribution grounding fix from a few days earlier got its last residual gap closed: the same specific-figure checker now also flags mixed-case named-entity tokens (like a misattributed portal name) verbatim-absent from their cited source, not just numeric figures, closing the exact gap the original incident's own "MiConsulado" reference exposed. Named-requirement phrases stay out of scope by design, a phrase can be paraphrased without changing its truth, so a verbatim check there would over-fire.
2026-08-19 through 2026-08-20: ministral-3:8b investigated to a BLOCKED verdict, three real engine bugs found and fixed along the way
Full verdict and evidence trail live in Model Bake off. This entry covers the three real, permanent DeepDelve-side fixes the investigation surfaced along the way, independent of this candidate's own fate.
check_report_underuses_findings had a boundary bug: ratio >= threshold let a report through when
it dropped exactly half its real sources (2 of 4 cited, ratio exactly 0.5 against a 0.5 default
threshold), silently passing any run landing precisely on the threshold. Caught by directly reading
a 0.750-scored report rather than trusting the score, the report was missing an entire city from a
two-city comparison query despite findings.md having clean sources for it the whole time; three
real baseline gpt-oss runs at the identical score genuinely covered both cities, confirming this
wasn't normal for that score band. Fixed: >= to >, now fails closed on the boundary. The existing
structural test suite's scenario for this exact check had been asserting the old pass-on-boundary
behavior explicitly; updated to assert the corrected fail-closed behavior instead.
The writer-dispatch fallback path's error handling was undiagnosable: five except Exception: sites
around _dispatch_writer_review_fix discarded the real exception entirely, no message, no
traceback, logged nowhere, so a Review/Fix-stage failure could only ever be described as "something
raised," never root caused. Fixed to except Exception as e: with {type(e).__name__}: {e} folded
into the notify message at all five sites. This immediately paid for itself the same session: it
surfaced a crash on the very next run that a prior, unrelated fix had wrongly been credited with
avoiding, see below.
A new _is_transient_ollama_json_error/_dispatch_task_retrying_transient_json_error wrapper
retries a writer dispatch once, unchanged, specifically for a narrow, real, open upstream Ollama bug
(ollama/ollama#6351, #12064) that corrupts a tool call's JSON whenever its argument content
contains a raw newline. Wired into all four dispatch call sites inside _dispatch_writer_review_fix
(Write, Write-retry, Review, Fix), the one shared choke point every writer role routes through, so
one fix covers Builder, FindingsWriter, and both of their per-facet dispatch variants at once. Live
tested: confirmed insufficient on its own at low temperature, since near-deterministic decoding
reproduces the same broken byte sequence on retry, the exact operating mode this model needs per its
own vendor docs for reliable tool calling. Kept in the codebase regardless, since it's still a
correct, real mitigation for any candidate hitting this bug under normal sampling temperatures, just
not sufficient alone for a model pinned to near-zero temperature.
Closed the "per source evidence crowding" item that had stood as the top priority since 2026-07-22, then kept live testing the standing Lisbon versus Mexico City prompt through 4 more runs, root causing each new failure via direct reads of the run state and session transcript.
RunState.coverage() gained an empty summary exclusion: a real URL used to count as covered even
when its summary was completely empty, a sub-agent ending its turn with zero trailing text and no
cutoff marker at all, a third, previously undetected synthesis-vanishing mechanism, measured at 25
to 42 percent of all findings across two live runs. A ledger rollup fix credited a top-level task
with real evidence its nested Analyzer children had found but the verification ledger never counted.
A task-name-churn fix addressed the rename detector's raw text similarity scoring far too low on a
real full-sentence paraphrase, so a facet redispatched 3 times under 3 different names was never
recognized as one rename.
The actual root cause of "per source evidence crowding": add_finding attaches one shared summary
to every URL fetched in a turn, so a stub-source flag about one co-fetched URL wholesale excluded the
record for all of them, confirmed live when a real price figure got thrown away alongside an
unrelated flagged page. Fixed by scoping the exclusion to the finding's own source URL. A
read_workspace_file exact-repeat quota dedup stopped one dispatch from burning its entire quota on
identical repeated calls before finishing its actual edit work. A warning-marker leak, a direct side
effect of the evidence-crowding fix, caused a finding that correctly stayed citable to still carry
raw warning text into findings.md, which then re-flagged itself, a self-inflicted loop confirmed
via 3 byte-identical rejected draft snapshots; fixed by stripping the marker text before rendering.
Two real issues were found and deliberately left open: a writer role dispatch can end with zero output after a structural write gate blocks it, the same zero-trailing-text mechanism now confirmed to hit a writer role too, worse since nothing gets written at all; and why FindingsWriter's rebuild sometimes fails to self-correct a flagged citation across multiple retries. Both need proper scoping, not a tail-end patch.
Five consecutive live runs of the same benchmark, 45 to 68 minutes each, each producing a different real bug, chased down via the run state and report files, not log lines alone. A mid-session user correction ("you're too confident") caught a case where the system log looked like healthy progress while the actual report on disk was still badly incomplete.
A directive-oscillation bug: a task verification check's "stop redelegating" directive could get silently reversed by a later quota refill, reissuing "redelegate" for a task already told to stop; fixed with a sticky per-task ledger flag. A cross-tier starvation bug: the existing starvation guard only protected grounding checks from a repeating problem within the same check, but a run where a different completion check won every attempt starved a grounding check built to catch a report dropping most of its requested facets for an entire run; generalized to the whole tier, live confirmed working across 11 attempts post-fix. A write-gate bug: the structural write-first gate only accepted the full-file write tool, but it's also armed for narrower add-only correction dispatches whose own instructions say to use the edit tool instead, so the gate's own block message actively steered the model toward the full overwrite its instructions forbade, silently destroying facets a prior round had added, firing 20 times in one non-converging run; fixed, zero occurrences post-fix. An entity-mismatch bug in the rename detector false-positived on two independently dispatched tasks sharing a template (two cities' rent facets differing only in name), silently marking one city's real facet superseded, never appearing anywhere with no gap flagged; fixed with proper-noun extraction requiring the two tasks' actual named subjects to overlap before trusting a high text-similarity score, unit tested but not yet live reconfirmed.
A related item found but deliberately not fixed: a single task fetching many real sources can still crowd FindingsWriter's synthesis even after the per-facet dispatch fix, since that fix scopes by task, not by individual source; confirmed live when several source headings got a placeholder despite real content existing, and Builder then fabricated plausible numbers for the gapped sources.
The first hosted, non-local model tested against the real pipeline, 4 runs across both the web UI and
headless harness on the complex Colombia B2B benchmark. Two real, reusable harness bugs surfaced.
First, a new api.backend: "openai_hosted" option: the existing thinking-mode control is a local
serving convention a real hosted API just ignores, confirmed live that DeepSeek defaulted to thinking
on regardless; the new backend looks up each hosted provider's own documented convention instead,
though this alone didn't fix DeepSeek's failure, whose verbosity was ordinary content, not leaked
reasoning. Second, a guardrail starvation bug in the API server's own context-budget cutoff, an
unconditional force-jump to the end that gave a genuinely working check zero real retries before
salvage on a verbose model; brought to parity with the CLI's existing two-stage nudge-then-cutoff.
With both fixed, DeepSeek still failed on genuine model unreliability: 0.2/1.0 for Flash and 0.0/1.0
for Pro (despite ~3x the price) against gpt-oss:20b's 0.7 baseline. The real cause: DeepSeek
re-fabricates the same citations on redo, correctly caught by the task verification check refusing to
let the run advance. Not disqualified for tool-calling mechanics, clean in 3 of 4 runs, or for "no
write access" confusion, since the Planner genuinely has no write tool by design and DeepSeek's own
statement to that effect was accurate.
The user rejected an earlier session's diagnosis that the report-writing stage wasn't structurally overwhelming models, insisting on a fresh whole-repo pass instead of trusting the prior conclusion. A dedicated catalog pass found 17 distinct report-writing incidents across 5 patterns spanning the project's whole history and every model ever tested, including the trusted baseline. Two real root causes got fixed, both grounded in checked literature and both live verified (full detail in Completed's evidence-coverage and check-priority-shadowing entries).
A live smoke test also caught a real gap in the project's own verification process: the first "smoke test passed" read of a 48-minute run checked only for crashes and false positives, never what the report actually said. The user rejected that outright, "this is not acceptable, no proper report made," and was right: the report used a fabricated inline numbering scheme with no reference list at the bottom, unverifiable despite passing URL-presence grounding, plus a claim attributed to a source that, read directly, never mentions it, the exact concrete bug (an uncited-claims check never getting a turn because a different check kept winning) fixed the same session. Read the actual output, not just the check log, before calling any smoke test conclusive.
Full detail in Model Bake off and the Literature Review.
Five live runs surfaced a genuinely strong cold-start synthesis (45 sources, correct architecture
family coverage) but never converged on a clean, verified final report. The looping root cause traced
to the stock chat template's empty think-block injection defect, independently corroborated by the
model's own GitHub issues and Reddit threads, a real model-family trait, not a DeepDelve-specific
one, fixed via a community patched template. Two model-independent DeepDelve architecture bugs also
got found and fixed along the way (a "not delegated" check scoped to the wrong process, and Builder
being handed a directive worded for a delegation tool it doesn't have), plus a new
edit_workspace_file tool for targeted corrections, and a real serving-layer thinking-suppression
leak isolated on the OpenAI-compat endpoint specifically when tools are present. Final verdict: left
inconclusive, not disqualified, not passed, since every failure mode had an independent non-model
explanation attached.
Resumed on the explicit condition that every documented reward dimension be accounted for first,
after auditing found 3 real gaps (missing generators, one reward function never actually trained)
and closing all three the same day. The round trained on Qwen/Qwen3-4B with a rank-16 LoRA across 7
task types and 302 rows (up from 3 types and 158 rows), the project's "one combined retrain, never
piecemeal" rule honored fully this time. It took 5 attempts, with 2 real crash bugs found and fixed
via live training, both genuine unguarded model-output edge cases, not hypothetical, one fixed with a
shared helper rather than four separate patches, with checkpointing added afterward so a future crash
costs at most ~50 steps.
Held-out evaluation on 39 prompts never seen in training, extended from 2 dimensions to all 7:
| Dimension | base | fine-tuned | delta |
|---|---|---|---|
| thin_coverage | 0.542 | 1.000 | +0.458 |
| citation_grounding | 0.615 | 0.781 | +0.167 |
| findings_evidence | 0.562 | 1.000 | +0.438 |
| tool_name | 0.925 | 1.000 | +0.075 |
| stale_findings | 1.000 | 0.875 | -0.125 |
| uneven_task_investment | 1.000 | 1.000 | 0.000 |
| task_verification_flagged | 1.000 | 1.000 | 0.000 |
| combined overall | 0.747 | 0.926 | +0.179 |
A real, generalized improvement, not memorization: the two hardest, most load-bearing dimensions improved substantially on topics never trained on. Two caveats: stale findings regressed on only 2 held-out prompts, plausibly noise, and citation grounding, the dimension tied most to actual hallucination risk, improved but didn't reach ceiling. Not yet a live-verified result, just an offline reward score; deployability would need loading the LoRA into a real run and benchmarking it live, the same standard every candidate is held to.
History
Model Research
Reviews & Audits
Reference