-
Notifications
You must be signed in to change notification settings - Fork 0
Changelog Recent
Part 1 of the Changelog, most recent entries first. Continues into Part II for the rest of 2026-07-26.
This closed the "per source evidence crowding" item that had stood as the number 1 priority since 2026-07-22. Every open item from the prior day's session got closed out, then the same "Lisbon vs Mexico City" prompt got live tested through 4 more full runs, root causing each new failure via direct reads of the run state and session transcript rather than trusting log lines alone. Roughly in the order fixed:
First, RunState.coverage() gained an empty summary exclusion. A real, http prefixed source URL
used to count as "covered" even when its summary was completely empty, a sub agent ending its own
turn immediately after a tool call with zero trailing text, no cutoff marker at all, a third,
previously undetected synthesis vanishing mechanism, measured at 25 to 42 percent of all findings
across two live runs. Coverage now excludes those via the same predicate used elsewhere, fixing the
two downstream checks that consume coverage's output automatically.
Second, a ledger rollup and FOLLOW-UP DIRECTIONS grounding fix. A top level task whose own findings were all empty still had real evidence in its nested Analyzer children that the verification ledger never credited back to it, fixed with a new contextvar. Separately, a specialist's suggested next URLs section, never a citation, was firing false verification warnings that invalidated genuinely cited real content sitting right next to it.
Third, a task name churn denominator inflation fix. The rename detector's raw text similarity scored 0.11 on a real live full sentence paraphrase, nowhere near its 0.6 threshold, so a facet redispatched 3 times under 3 different names was never recognized as one rename, and even when caught, coverage never actually read the superseded ledger status, so the denominator kept inflating anyway. Fixed both: a content word overlap trigger, and coverage excluding superseded task names.
Fourth, a URL scoped verification warning exclusion, which turned out to be the root cause of the
"per source evidence crowding" item first named on 2026-07-22 and repeatedly flagged since as the
clear number 1 priority. add_finding attaches one shared synthesis text to every URL fetched in a
turn, so a stub source or unverified URLs flag about one co fetched URL wholesale excluded the
record for all of them, confirmed live when a real price figure got thrown away alongside an
unrelated stub page flag. The citability check now scopes the exclusion to the finding's own
source URL when the marker names specific bad URLs.
Fifth, a read_workspace_file exact repeat quota dedup. A FindingsWriter dispatch called
read_workspace_file with identical arguments 2 to 3 times in a row, burning its entire quota
before finishing its actual edit work, confirmed via a quota exhaustion trace forcing a
final_report.md never written outcome. Live confirmed working, zero quota exhaustion events in
the very next live run.
Sixth, a warning marker leak fix, a direct side effect of fix four, found in the next live run
after it shipped: a finding correctly staying citable, since its own URL wasn't the flagged one,
still carried the raw warning marker text verbatim in its summary, which got rendered into
findings.md by the deterministic fallback and then re flagged by findings.md's own grounding
check as if the warning's named URL were a real citation, a self inflicted loop, confirmed via 3
byte identical rejected draft snapshots. Fixed by stripping the marker text before rendering a
citable finding's block.
Two real issues got found but deliberately left open, not rushed. First, a writer role dispatch can end its own turn with zero output after a structural write gate blocks it, the same "zero trailing text" mechanism as fix one, now confirmed to also hit a writer role, where the consequence is worse since nothing gets written at all, and the existing one shot retry only partially absorbs it. Second, why FindingsWriter's rebuild sometimes fails to self correct a flagged citation across multiple retries. Both need their own properly scoped investigation, not a tail end patch.
Running the medium "Lisbon vs Mexico City" eval prompt as a smoke test for an unrelated open item instead produced a real, different bug on each of 5 consecutive live runs, 45 to 68 minutes each. Each one got chased down to its actual root cause via the run state, findings, and final report files, not log lines alone, a mid session user correction ("you're too confident") caught a case where the run's own system log messages looked like healthy progress while the actual report on disk was still badly incomplete.
The first fix addressed a directive oscillation bug: the task verification check's quota exhausted "stop redelegating" directive could get silently reversed by a later quota refill flipping the exhaustion flag back to false, reissuing "redelegate" for a task already told to stop. A new per task ledger flag now makes that decision sticky across the ledger's full recompute.
The second fix addressed cross tier completion check starvation. The existing starvation guard only protected the grounding checks when the same completion check problem repeated consecutively, but a run where a different completion check problem won every attempt starved a grounding check built to catch exactly the case of a report dropping most of its requested facets, for an entire run, despite the final report having dropped 3 of 4 requested facets. A new mechanism generalizes the guard to the whole tier. Live confirmed working: post fix, the same check correctly kept firing and dispatching Builder across 11 attempts instead of never getting a turn.
The third fix addressed the structural write first gate and a narrower correction tool. The gate
forcing FindingsWriter to write before reading only accepted the full file write tool, but it's
also armed for per facet, add only correction dispatches whose own instructions say to use the
narrower edit tool instead and never touch the rest of the file. The gate's block message actively
steered the model toward exactly the full file overwrite its own instructions forbade, silently
destroying facets a prior round had already added. This fired 20 times in one run that never
converged. Live confirmed working: zero occurrences post fix, findings.md converged cleanly on
the first pass.
The fourth fix addressed an entity mismatch in the rename detector. The heuristic meant to catch
the Planner renaming a flagged task instead of retrying it under the same name false positived on
two independently dispatched, differently named tasks sharing a template, two cities' rent facets
differing only in city and neighborhood names, at 0.89 raw text similarity. This silently,
permanently marked one city's real facet "superseded" the moment the other verified, it never
appeared in findings.md or the final report, with no gap ever flagged anywhere in the run. Fixed
via proper noun extraction plus an overlap override that only trusts a high text similarity ratio
when the two tasks' actual named subjects overlap. This one was found too late in the session for a
fresh live run to finish before the write up, unit tested but live reconfirmation stayed open.
A related item was found but deliberately not fixed this session: a single task that legitimately fetches many real sources, 8 or more seen live, can still crowd FindingsWriter's one shot synthesis even after the existing per facet dispatch fix, which scopes by task rather than by individual source, confirmed live when most source headings in one findings file got a placeholder despite real content existing for at least one of them, and Builder then fabricated plausible numbers for those gapped sources in the final report.
This was the first ever hosted, non local model tested against this project's real pipeline, via both the web UI and the headless eval harness. 4 total runs, two flash runs via the web UI, one flash headless CLI run, one pro headless CLI run, all against the complex difficulty Colombia B2B item. DeepSeek's own thinking mode and tool calling docs got read directly before any config change, not assumed from memory, after an initial wrong guess at DeepSeek's current model names was caught and corrected via the real pricing docs first.
Two real, reusable harness bugs got found and fixed along the way, both worth more than this one
model's own verdict. First, a new api.backend: "openai_hosted" option: the existing backend's
thinking mode control is a vLLM or local serving convention that a real hosted API just silently
ignores, confirmed live that DeepSeek defaulted to thinking on at effort "high" regardless, and
that the local serving convention's own field was meaningless to its API at all. The new backend
looks up each hosted provider's own documented convention instead. This alone didn't fix DeepSeek's
failure, which isolated that its narration is verbose as content, not leaked hidden reasoning.
Second, a guardrail starvation bug in the API server's own context budget cutoff, which was an
unconditional force jump straight to the end. Confirmed live that a verbose model can blow the same
character budget gpt-oss:20b never approaches, before the run's first completion check attempt
even happens, giving a check that was actively working literally zero real retries before salvage.
The API server's own code comment had explicitly, deliberately chosen this blunter behavior for
shared queue protection reasons, but it got brought to parity with the CLI's existing two stage
nudge then cutoff anyway, since one bounded extra wrap up turn doesn't meaningfully weaken that
protection and the alternative, a guardrail that can never fire, is worse.
With both fixed, DeepSeek still failed, and the root cause is genuine model unreliability. Scores
came in at 0.2 out of 1.0 for flash and 0.0 out of 1.0 for pro, worse despite roughly 3 times the
price, against gpt-oss:20b's 0.7 baseline on the same rubric. The real, confirmed cause: DeepSeek
re fabricates the same citations on redo, and the task verification check correctly refused to let
the run advance to the stage that dispatches the real writer roles, because the same flagged task
names recurred across retry attempts instead of resolving. This wasn't disqualified for tool
calling mechanics, which were clean in 3 of 4 runs, or for "no write access" confusion, since the
Planner role genuinely has no file writing tool by design and DeepSeek's own statement to that
effect was accurate, not hallucinated. Full verdict and evidence trail live in the
Model Bake off page's Hosted section.
The user pushed back on an earlier session's own diagnosis that the findings and report writing
stage wasn't structurally overwhelming models, insisting on a fresh, whole repo pass over every
historical run rather than trusting the prior audit's conclusion. A dedicated catalog pass over the
full history, literature review, and every session status archive found 17 distinct report writing
stage incidents across 5 patterns, spanning the whole project's life and every model this project
had ever tested, including the trusted gpt-oss:20b baseline. Two real, distinct root causes got
fixed as a result, both grounded in checked literature and both live verified, the first via a real
48 minute gpt-oss:20b run against the standing two facet benchmark query, the second by feeding
that same run's actual saved final report into the fixed code directly and confirming a previously
silent problem now surfaced. See the Completed list's evidence coverage and check
priority shadowing entries for the actual fixes.
A live smoke test also caught a real gap in the project's own verification process, worth recording: the first "smoke test passed" read of that 48 minute run was wrong, it checked for crashes and false positive regressions on the new check, but never read what the report actually said. The user rejected that read outright, "this is not acceptable, no proper report made," and was right to: the report used a fabricated inline numbering scheme with no numbered reference list at the bottom, unverifiable to a human despite passing URL presence grounding, plus a claim attributing something to a source that, read directly, never mentions it, and the concrete bug fixed this session, an uncited claims check never getting a turn across 3 attempts because a different check kept winning the priority chain, even though the uncited claims problem was independently, simultaneously true. Read the real output, not just the check log, before calling any smoke test conclusive.
Candidate: deepreinforce-ai/Ornith-1.0-9B, a dense, Qwen3.5 architecture model under MIT license,
the untested middle ground identified after gpt-oss:20b between the already exhausted sub 14B
local model space and paid frontier APIs. GLM 4.7 Flash and Ornith 1.0 35B were both ruled out on
hardware grounds first, their smallest available quants exceed this hardware's VRAM budget, without
spending GPU time on either.
Five live runs, a cold pull and benchmark, then three resumed attempts, surfaced a genuinely strong cold start synthesis, 45 real sources, correct architecture family coverage matching the benchmark's own gold reference, the best findings file of any candidate this project had tested, but never converged on a clean, fully verified final report. The root cause of the looping and self rejection pattern traced to the stock chat template's empty think block injection defect, independently corroborated by the model author's own GitHub issues and two Reddit threads, a real, model family wide trait, not a DeepDelve specific one. Fixed via a community patched chat template, applied directly into the pulled GGUF's own metadata.
Two real, model independent DeepDelve architecture bugs got found and fixed along the way, and would affect any model hitting these paths, not just this candidate. First, a "not delegated" check was scoped to the current process's live quota pool, always zero at the start of a resumed process, contradicting the resume logic's own "don't re delegate" instruction, live confirmed to derail a resumed run into a reflection loop. Fixed by also checking the run's persisted fetched URLs. Second, Builder was dispatched with grounding check text worded for the Planner, "delegate a Searcher," but Builder has no delegation tool at all, live confirmed to get it stuck narrating "I will delegate..." instead of ever rewriting the file. Fixed with a shared clarification string appended to both Builder dispatch branches.
A new edit_workspace_file tool got added after confirming live that a "drop 3 flagged citations,
keep everything else" correction cycle was a genuine capacity edge for full document regeneration,
one attempt made zero write or edit calls across about 8 minutes, successive full rewrites fixed old
stub citations while introducing new ones, never converging. Live tested once after adding it, the
model didn't spontaneously reach for the new tool even on a textbook case, inconclusive on one
sample, not contradicted, left open for a future retest.
A real serving layer gap also got isolated and confirmed: four direct API tests holding everything else constant showed thinking suppression works cleanly via Ollama's native endpoint, even with tools present, but leaks via the OpenAI compat endpoint DeepDelve actually used at the time, specifically triggered by tools being present in the request. Since DeepDelve was built entirely on an OpenAI compatible client, this likely affected every model ever tested through Ollama on this project whenever it made a tool call with thinking nominally disabled, not just this candidate. Not fixed at the time, proposed as a real architecture question, a backend adapter abstraction for how different serving backends want thinking toggled or tool schemas shaped.
A process hygiene lesson also got recorded: launching the app in the background with nohup
returns the bash wrapper's own process ID, not the actual Python process's, so killing the wrapper
alone left two real app processes orphaned in the background mid session, each still holding a
request against the model.
The final verdict: Ornith 1.0 9B left inconclusive, not disqualified, not passed. Every failure mode hit had an independent non model explanation attached, two fixed DeepDelve bugs, one serving layer gap, one missing tool now added, so no run that night was a clean, unconfounded test of this candidate's real ceiling per the Model Evaluation Standard. A future clean re test was seen as the natural next step, not a repeat of that night's confounded conditions.
The user explicitly decided to resume fine tuning after an indefinite pause, on the condition that every documented reward dimension be accounted for first rather than training with partial coverage again. Auditing the reward module against what the training script actually trained, versus what its own docstring merely documented, found 3 real gaps, all closed the same day: one reward function had real calibration data but no synthetic prompt generator and had never actually been trained, and two other checks had no reward function at all. New generators got built for all three, plus new scenarios added to the existing generator for a fourth check that reuses an existing reward function after confirming the response shapes were structurally identical, not assumed.
The training round used the Qwen/Qwen3-4B base with a rank 16 LoRA, across 7 task types, 302
rows, and 5 distinct reward functions, up from 3 task types and 158 rows in the prior round, honoring
the project's own "one combined retrain, never piecemeal" methodology rule fully this time, not
partially like an earlier attempt that added a reward function but never actually launched.
It took 5 attempts to complete, with 2 real bugs found and fixed via live training, both genuine
model output edge cases the reward functions weren't defensive against, not hypothetical. First,
one reward function crashed on a None per task URL list, since the dataset library unions every
row's dict keys into one schema, padding keys absent from a given row with None rather than an
empty list. Fixed by treating None like an empty list. Second, the schema compliance reward
function crashed when a real completion emitted its arguments as a JSON array instead of an object,
ironic, since that's exactly the malformed shape that function exists to score zero for. This hit
96 percent through a roughly 95 minute run with nothing saved yet, forcing a full restart. Fixed
once at the root with a new shared helper rather than patched separately at the four call sites
that had the same unguarded pattern, and checkpointing got added afterward so a future crash costs
at most about 50 steps, not the whole run.
Held out evaluation, extended that day from 2 dimensions to all 7, ran 39 held out prompts on topics never in the 302 training rows.
| Dimension | base | fine-tuned | delta |
|---|---|---|---|
| thin_coverage | 0.542 | 1.000 | +0.458 |
| citation_grounding | 0.615 | 0.781 | +0.167 |
| findings_evidence | 0.562 | 1.000 | +0.438 |
| tool_name | 0.925 | 1.000 | +0.075 |
| stale_findings | 1.000 | 0.875 | -0.125 |
| uneven_task_investment | 1.000 | 1.000 | 0.000 |
| task_verification_flagged | 1.000 | 1.000 | 0.000 |
| combined overall | 0.747 | 0.926 | +0.179 |
The verdict: a real, generalized improvement got confirmed, not memorization, since the two hardest, most load bearing dimensions, thin coverage and findings evidence, improved substantially on topics the model never trained on, directly answering the overfitting question raised mid session. Two real caveats stood: stale findings regressed, but on only 2 held out prompts, plausibly noise, and citation grounding, the dimension most tied to actual hallucination risk, improved but didn't reach ceiling, meaning the residual gap there mattered most of any dimension. This was explicitly not yet a live verified result, just an offline reward function score, and before it could count as deployable it would need loading the LoRA into a real run and benchmarking it live, the same standard every model candidate is held to.
History
Model Research
Reviews & Audits
Reference