-
Notifications
You must be signed in to change notification settings - Fork 0
Changelog Recent
Part 1 of the Changelog, most recent entries first. Continues into Part II.
Closed the "per source evidence crowding" item that had stood as the top priority since 2026-07-22, then kept live testing the standing Lisbon versus Mexico City prompt through 4 more runs, root causing each new failure via direct reads of the run state and session transcript.
RunState.coverage() gained an empty summary exclusion: a real URL used to count as covered even
when its summary was completely empty, a sub-agent ending its turn with zero trailing text and no
cutoff marker at all, a third, previously undetected synthesis-vanishing mechanism, measured at 25
to 42 percent of all findings across two live runs. A ledger rollup fix credited a top-level task
with real evidence its nested Analyzer children had found but the verification ledger never counted.
A task-name-churn fix addressed the rename detector's raw text similarity scoring far too low on a
real full-sentence paraphrase, so a facet redispatched 3 times under 3 different names was never
recognized as one rename.
The actual root cause of "per source evidence crowding": add_finding attaches one shared summary
to every URL fetched in a turn, so a stub-source flag about one co-fetched URL wholesale excluded the
record for all of them, confirmed live when a real price figure got thrown away alongside an
unrelated flagged page. Fixed by scoping the exclusion to the finding's own source URL. A
read_workspace_file exact-repeat quota dedup stopped one dispatch from burning its entire quota on
identical repeated calls before finishing its actual edit work. A warning-marker leak, a direct side
effect of the evidence-crowding fix, caused a finding that correctly stayed citable to still carry
raw warning text into findings.md, which then re-flagged itself, a self-inflicted loop confirmed
via 3 byte-identical rejected draft snapshots; fixed by stripping the marker text before rendering.
Two real issues were found and deliberately left open: a writer role dispatch can end with zero output after a structural write gate blocks it, the same zero-trailing-text mechanism now confirmed to hit a writer role too, worse since nothing gets written at all; and why FindingsWriter's rebuild sometimes fails to self-correct a flagged citation across multiple retries. Both need proper scoping, not a tail-end patch.
Five consecutive live runs of the same benchmark, 45 to 68 minutes each, each producing a different real bug, chased down via the run state and report files, not log lines alone. A mid-session user correction ("you're too confident") caught a case where the system log looked like healthy progress while the actual report on disk was still badly incomplete.
A directive-oscillation bug: a task verification check's "stop redelegating" directive could get silently reversed by a later quota refill, reissuing "redelegate" for a task already told to stop; fixed with a sticky per-task ledger flag. A cross-tier starvation bug: the existing starvation guard only protected grounding checks from a repeating problem within the same check, but a run where a different completion check won every attempt starved a grounding check built to catch a report dropping most of its requested facets for an entire run; generalized to the whole tier, live confirmed working across 11 attempts post-fix. A write-gate bug: the structural write-first gate only accepted the full-file write tool, but it's also armed for narrower add-only correction dispatches whose own instructions say to use the edit tool instead, so the gate's own block message actively steered the model toward the full overwrite its instructions forbade, silently destroying facets a prior round had added, firing 20 times in one non-converging run; fixed, zero occurrences post-fix. An entity-mismatch bug in the rename detector false-positived on two independently dispatched tasks sharing a template (two cities' rent facets differing only in name), silently marking one city's real facet superseded, never appearing anywhere with no gap flagged; fixed with proper-noun extraction requiring the two tasks' actual named subjects to overlap before trusting a high text-similarity score, unit tested but not yet live reconfirmed.
A related item found but deliberately not fixed: a single task fetching many real sources can still crowd FindingsWriter's synthesis even after the per-facet dispatch fix, since that fix scopes by task, not by individual source; confirmed live when several source headings got a placeholder despite real content existing, and Builder then fabricated plausible numbers for the gapped sources.
The first hosted, non-local model tested against the real pipeline, 4 runs across both the web UI and
headless harness on the complex Colombia B2B benchmark. Two real, reusable harness bugs surfaced.
First, a new api.backend: "openai_hosted" option: the existing thinking-mode control is a local
serving convention a real hosted API just ignores, confirmed live that DeepSeek defaulted to thinking
on regardless; the new backend looks up each hosted provider's own documented convention instead,
though this alone didn't fix DeepSeek's failure, whose verbosity was ordinary content, not leaked
reasoning. Second, a guardrail starvation bug in the API server's own context-budget cutoff, an
unconditional force-jump to the end that gave a genuinely working check zero real retries before
salvage on a verbose model; brought to parity with the CLI's existing two-stage nudge-then-cutoff.
With both fixed, DeepSeek still failed on genuine model unreliability: 0.2/1.0 for Flash and 0.0/1.0
for Pro (despite ~3x the price) against gpt-oss:20b's 0.7 baseline. The real cause: DeepSeek
re-fabricates the same citations on redo, correctly caught by the task verification check refusing to
let the run advance. Not disqualified for tool-calling mechanics, clean in 3 of 4 runs, or for "no
write access" confusion, since the Planner genuinely has no write tool by design and DeepSeek's own
statement to that effect was accurate.
The user rejected an earlier session's diagnosis that the report-writing stage wasn't structurally overwhelming models, insisting on a fresh whole-repo pass instead of trusting the prior conclusion. A dedicated catalog pass found 17 distinct report-writing incidents across 5 patterns spanning the project's whole history and every model ever tested, including the trusted baseline. Two real root causes got fixed, both grounded in checked literature and both live verified (full detail in Completed's evidence-coverage and check-priority-shadowing entries).
A live smoke test also caught a real gap in the project's own verification process: the first "smoke test passed" read of a 48-minute run checked only for crashes and false positives, never what the report actually said. The user rejected that outright, "this is not acceptable, no proper report made," and was right: the report used a fabricated inline numbering scheme with no reference list at the bottom, unverifiable despite passing URL-presence grounding, plus a claim attributed to a source that, read directly, never mentions it, the exact concrete bug (an uncited-claims check never getting a turn because a different check kept winning) fixed the same session. Read the actual output, not just the check log, before calling any smoke test conclusive.
Full detail in Model Bake off and the Literature Review.
Five live runs surfaced a genuinely strong cold-start synthesis (45 sources, correct architecture
family coverage) but never converged on a clean, verified final report. The looping root cause traced
to the stock chat template's empty think-block injection defect, independently corroborated by the
model's own GitHub issues and Reddit threads, a real model-family trait, not a DeepDelve-specific
one, fixed via a community patched template. Two model-independent DeepDelve architecture bugs also
got found and fixed along the way (a "not delegated" check scoped to the wrong process, and Builder
being handed a directive worded for a delegation tool it doesn't have), plus a new
edit_workspace_file tool for targeted corrections, and a real serving-layer thinking-suppression
leak isolated on the OpenAI-compat endpoint specifically when tools are present. Final verdict: left
inconclusive, not disqualified, not passed, since every failure mode had an independent non-model
explanation attached.
Resumed on the explicit condition that every documented reward dimension be accounted for first,
after auditing found 3 real gaps (missing generators, one reward function never actually trained)
and closing all three the same day. The round trained on Qwen/Qwen3-4B with a rank-16 LoRA across 7
task types and 302 rows (up from 3 types and 158 rows), the project's "one combined retrain, never
piecemeal" rule honored fully this time. It took 5 attempts, with 2 real crash bugs found and fixed
via live training, both genuine unguarded model-output edge cases, not hypothetical, one fixed with a
shared helper rather than four separate patches, with checkpointing added afterward so a future crash
costs at most ~50 steps.
Held-out evaluation on 39 prompts never seen in training, extended from 2 dimensions to all 7:
| Dimension | base | fine-tuned | delta |
|---|---|---|---|
| thin_coverage | 0.542 | 1.000 | +0.458 |
| citation_grounding | 0.615 | 0.781 | +0.167 |
| findings_evidence | 0.562 | 1.000 | +0.438 |
| tool_name | 0.925 | 1.000 | +0.075 |
| stale_findings | 1.000 | 0.875 | -0.125 |
| uneven_task_investment | 1.000 | 1.000 | 0.000 |
| task_verification_flagged | 1.000 | 1.000 | 0.000 |
| combined overall | 0.747 | 0.926 | +0.179 |
A real, generalized improvement, not memorization: the two hardest, most load-bearing dimensions improved substantially on topics never trained on. Two caveats: stale findings regressed on only 2 held-out prompts, plausibly noise, and citation grounding, the dimension tied most to actual hallucination risk, improved but didn't reach ceiling. Not yet a live-verified result, just an offline reward score; deployability would need loading the LoRA into a real run and benchmarking it live, the same standard every candidate is held to.
History
Model Research
Reviews & Audits
Reference