-
Notifications
You must be signed in to change notification settings - Fork 0
Completed I
Part 1 of the Completed list of finished ROADMAP items.
(2026-08-17). An edit_workspace_file call anchored its
old_string on a bare heading, so the pre-existing bullet under it never got matched, stayed in
place, and a retyped copy landed right in front of it, producing two near-identical report sections.
Fixed at two layers: prompt instructions now say to anchor an adding edit on the boundary of
existing content, never retype it, and a new structural check,
find_duplicate_report_sections/check_duplicate_report_sections, compares every section of a
report against every other for word overlap. Live validated against the real incident's report: it
correctly flags the one real duplicate and nothing else.
(2026-08-17). Four real, accurately worded figures were traced verbatim
to one fetched source but cited to a different, also-fetched URL, passing the hard grounding gate
(both URLs real) and the term overlap gate (both pages happened to share a bare year). The number
regex behind that gate was too narrow to ever extract the actual figures as checkable terms. Fixed
with a new sibling function, find_unsupported_specific_figures, generalizing the existing
regulation-ID check to dollar figures and day/month counts, verified line by line against its own
cited source. A real bug got caught during validation: the first version stripped thousands-separator
commas from the claim but not the source, false-positiving genuinely supported figures; fixed by
normalizing both sides. Re-validated: the false positives resolved and two more previously unnoticed
real grounding failures in the same report got caught for free.
(2026-08-02). Fixed folder name
collisions for long queries sharing a prefix (a hash suffix), added a references.bib built from
what the report actually cites (not the raw fetched list), added --seed-doc//seed-doc local
document ingestion mirroring the existing seed-URL pattern, and an optional PDF export via pandoc.
The bigger piece was src/api.py and src/static/index.html, an optional FastAPI HTTP API and web
UI. Since the engine's session state is module-level, not per-request, the API uses a single
in-process FIFO job queue, always single-flight, a third copy of the run lifecycle rather than a
shared abstraction. Endpoints cover starting/resuming/following-up/cancelling a run and fetching its
report, bibliography, or PDF. Several real bugs surfaced during implementation: a follow-up request
originally ran the full completion-check pipeline like a fresh run, rewriting findings.md
repeatedly and never touching the report, fixed by mirroring the TUI's own skip_completion_check
condition; an in-memory job registry 404'd on real artifacts from a prior server process, fixed to
check the workspace directly; and FastAPI doesn't auto-add HEAD support to a GET route, so every
existence check in the frontend silently failed, hiding every download button. A new
settings.api_password option was added given that full settings CRUD plus LAN/phone access raises
the stakes. The same session also folded in an SSRF guard on outbound fetches (checking every
resolved target, including redirect hops, against private/loopback ranges), trimmed traceback
leakage to the model, and added test_tools.py pinning the new guard.
(2026-07-31). A check that isn't Builder/FindingsWriter-fixable and never caps its own firing wins
first match on every attempt, permanently starving every check below it. Six live incidents got
patched one by one the same night before the user stopped incident-by-incident patching: "we're
having too much issues with starvation we need to create a defined structure, we're guessing." The
structural fix: one canonical counting function (_consecutive_occurrences with a skip_problems
set, replacing three drifting copies), a hoisted module-level escalation threshold, a required
_capped() call for any non-self-resolving check, and a declarative starvation-yield mechanism
replacing a buggy hand-written lambda that a live trace proved was dead code. A systematic audit
(not another incident) found 2 more previously unknown instances missing a cap. A new standing test
asserts every non-self-resolving check calls _capped, so a future check that skips it fails CI
immediately. Grounded in real prior art (OS scheduler aging, the circuit breaker pattern, Chain of
Responsibility). Live verified: a fresh run converged cleanly through the full check chain with no
starvation.
Note
This doesn't fix the model's own tendency to abandon the harder half of a multi-facet query, a separate, still open capability gap.
(2026-07-29). Two live runs
got stuck cycling the same problem for their entire retry budget without ever escalating, root
caused via _run_state.json: a sibling check (check_untracked_delegation) firing once in between,
itself a symptom of the model not complying with the first check's directive, was resetting the
escalation streak to zero. Fixed by treating that specific sibling problem as a continuation, not a
break. A related bug (.warning's human-facing text was a static string regardless of which
escalation branch actually fired) was fixed with three distinct messages.
(2026-07-29). The
existing findings-stage check guarantees every covered task has a URL surviving into
findings.md, but nothing guaranteed Builder's own selection represented every task; a flat
citation ratio can pass while every surviving citation comes from one task, confirmed live when
gpt-oss dropped an entire task for an off-topic citation but still cleared the ratio threshold on a
larger sibling task's raw source count. Literature grounded (arXiv:2412.08795 validates a binary
per-cluster presence check as a cheap fairness proxy). Live smoke tested: correctly stayed silent on
a report that unevenly but genuinely covered both tasks.
(2026-07-29). The same smoke test surfaced a second bug once the actual
report content, not just crash-freedom, got read: check_uncited_claims never got a turn across 3
attempts because a higher-priority check kept winning, traced to an internal ordered if-chain
returning only its first hit, so any check keyed off the single shared problem string could be
permanently shadowed by a persistently recurring one. The terminal message reported only one
problem even though a second, independently true one (6 uncited figure-bearing lines) was never
even computed. Literature grounded (arXiv:2607.01855 on multi-turn regression via a later fix
silently breaking an earlier requirement). Fixed by extracting the cheap sub-checks into a function
that returns every hit, not just the first, and surfacing the extras as addendum text without
changing which single verdict is "the" recorded problem or touching the escalation counters.
(2026-07-21 through
2026-07-22). A live run abandoned a complete, in-budget 30-finding evidence base ("we don't see
them here," its own words) and re-read raw source files by hand instead, writing findings.md from
1 of 33 findings. Fix 1 filtered relevance-flagged findings out of the citable list. Fix 2 threaded
an already-extracted page title through to the evidence base, closing a format mismatch that forced
the model to invent titles. Both fixes held at their own layer, but two independent live re-tests
confirmed the model's raw file exploration habit was unrelated to evidence base noise or format,
real progress, not the actual fix.
The real fix landed the next day (commit f1562f7): a structural write-first gate
(writer_gate_ctx/check_writer_gate) blocks read_workspace_file/grep_workspace_file until the
armed FindingsWriter dispatch has actually called write_workspace_file; a prompt-only reorder was
tried first and live disconfirmed. A second contributing bug, found while investigating why even a
gate-forced write stayed thin: a task fetching N URLs generated N near-identical findings sharing
one summary, inflating raw counts and letting whichever task fetched the most URLs dominate a
position-bias reorder. Fixed by grouping citable findings by task and summary while keeping every
real URL individually citable. Live confirmed closed: the evidence base collapsed to 5 real clusters
from 15 raw findings, findings.md wrote all 5, and the first dispatch called the write tool
directly with zero blocked reads, the fastest, cleanest convergence of any tested run.
(2026-07-21). A live benchmark failed outright, thin_coverage firing 4
times, after one sibling task consumed 26 tool-call log lines while its four siblings got 1 each,
the model's own narration admitting the search quota was exhausted. Two fixes: the existing per-task
quota rescue required proof of a prior successful fetch, which web_search never populates, exactly
backwards from what a starved task needs, dropped the requirement so every task gets one guaranteed
grace top-up. And a new _reserve_batch_quota_headroom pre-reserves headroom for a whole
delegate_tasks batch before any task in it starts, since tasks otherwise drain a shared pool
roughly in listed order, giving early tasks a structural first-mover advantage. Live re-test: zero
thin_coverage attempts versus 4 before, 25 findings balanced across 9 task angles versus one task
dominating 14 of 28.
A second bug in that same re-test: findings.md carried an auto-recovered draft banner because a
single, role-blind budget nudge told a writer role to stop calling tools right when its whole job is
to call the write tool. Fixed by routing Builder/FindingsWriter to a distinct "write the file now"
nudge, caught by the user reading the code directly, not a live failure. A follow-up re-test
confirmed this fixed but surfaced a deeper, still-open one: findings.md got written via a real
tool call but held only 1 of 33 real findings, since no check verifies "findings.md reflects
everything found," only that it exists and is formatted correctly. Converted a total-loss failure
into a partial-loss one, real progress but not a full fix.
(2026-07-22). Findings ordering: a
pure positional zigzag/sandwich reorder (drawing on "Lost in the Middle" and PING's Anchor Effect)
keeps every finding within one hop of a context edge instead of drifting toward the middle as a run
accumulates findings. A propagation-aware hallucination check, narrowed from PING's full design
(DeepDelve has no claim dependency structure and PING's own released code doesn't implement that
mechanism either) down to the specific split-brain pattern already observed: a citable finding whose
content term-overlaps an uncited or cutoff sibling for the same task, only fires if the flagged
content also reached the report itself. Forcing reasoning at synthesis time (drawing on PIVOT)
turned out to already exist in the prompts, so it shipped as verification instead: snapshotting
think_tool quota usage before and after a write dispatch. Full artifact rebuild escalation
(drawing on the planning-horizon paper) corrected a wrong premise, the existing threshold already
gives up early rather than nudging gently, so on the third consecutive identical problem the system
now grants one extra attempt for a genuine full rebuild instead of a targeted fix.
(2026-07-19). Each round's findings carry a follow-up
directions section; when thin coverage fires, unconsumed directions get geometrically narrowed
(newBreadth = ceil(breadth/2)) and dispatched directly, bypassing the Planner, so the engine drives
the extra research round rather than hoping the model chooses to loop.
(2026-07-20). Small/mid models fail disproportionately at structured serialization, confirmed live at ~4.9% of real delegation calls using a hallucinated but well-formed agent id. Routing moved from free generation to a frozen sentence embedding plus logistic regression, reject-and-nudge, not silent override. Trained on 814 deduplicated real pairs extracted from 101 session logs; held-out per-class results ranged from 0.89/0.88 (DocumentAnalyzer) down to 0.44/0.64 precision/recall for the smallest class, AcademicSearcher. Shipped disabled by default.
The first live end-to-end test found a real regression and the feature got reverted to disabled: a real AcademicSearcher dispatch got wrongly rejected 8 consecutive times at 0.67-0.75 confidence against a flat 0.6 threshold, until the Planner gave up on that angle entirely, a real content coverage loss. Root cause: a flat confidence threshold treats the model's weakest, smallest class as equally trustworthy as its strongest. A same-day data audit then found the actual root cause was bad training data, 13 near-identical mislabeled examples from one session. Two general detectors (text similarity, embedding cosine similarity) were tried and rejected for this cleanup, both failed to cleanly separate "same ask, inconsistently labeled" from "different ask, same topic," so the team used a direct, documented manual exclusion instead. Retraining brought held-out recall for AcademicSearcher from 0.64 to 0.80 at the same precision; all 3 real failing instructions now predict correctly, and a live re-test of the exact query that failed before confirmed zero rejections. A second, unrelated bug found in that same re-test (the report dropped the entire academic section because a run hit 3 remediation cycles that each burned read quota) was newly found, not fixed, in this session.
An earlier fix gave the CLI's top-level
loop a real deadline via asyncio.wait_for, but that fix never propagated to the orchestrator's
single-task dispatch path, which relied entirely on the raw HTTP client's blunt ~600 second default
timeout, discarding the whole in-progress response on trip instead of degrading gracefully. Root
caused live after the user pushed back on accepting repeated timeouts as "just model slowness":
cross referencing Ollama's own logs found a request that had been continuously, validly decoding for
the full 600 seconds up to nearly 20,000 tokens, a single runaway turn with no cutoff watching it. A
first fix attempt (giving the sub-task the same deadline as the top-level guard) was itself
incomplete, caught by live testing: the outer guard's own cancellation was pre-empting the inner one
before its deadline check ever got a chance to run, making the new code effectively dead in realistic
configs. The real fix was an independent settings.sub_agent_timeout_minutes (default 10) computed
fresh per dispatch, plus bumping the client's own SDK timeout past both budgets so it stopped winning
the race. Live verified with a tight 1 minute sub-agent budget: the dispatch was cut short cleanly and
the Planner's very next turn correctly adapted instead of hanging or crashing. This retroactively
explained several earlier "timeout" observations that had been wrongly attributed to model slowness.
History
Model Research
Reviews & Audits
Reference