Skip to content

2.5.0

Latest

Choose a tag to compare

@nathansutton nathansutton released this 31 Oct 15:47
· 1 commit to main since this release
0ecb4ae

Against 2.4.0 on the 30-task Linux pool (3 reps each, same server and model), 2.5.0 passed 89/90 to 88/90: one task up, none down. Both of 2.4.0's failures were a think that ran into the wall; 2.5.0 had none, and its one failure fell during a tool call. Wall clock was 1.07x, all but 2% of it one task (javascript/complex-numbers) whose occasional long rep shows up in every arm.

The think ceiling is on by default, at 2048 tokens. Every wall-cap failure in the published polyglot runs was one that never closed: 13 of 20 were killed mid-think, two more right after a 5–7k-token think. CHAD_THINK_CEILING force-closes the block past N tokens and keeps decoding the action in the same step, and was off. A paired A/B on the 30-task pool (gv-base vs gv-ceil2048, 3 reps each) put 2048 at 90/90 against 88/90: the never-pass task flipped up, nothing flipped down, 0.92x the wall and 0.87x the generated tokens. The ceiling fired 31 times in 20 trials and every salvaged step went straight to a write, edit, bash or todo. CHAD_THINK_CEILING=0 restores the old behaviour. Every trials.jsonl row now also says how the trial ended (stop: {kind, step, where}; engine failures are infra rows a resume redoes and stats.py compare leaves out), and trace.py --autopsy reports each trial's longest think and what a ceiling would have cut.

GGUFs of the qwen35 family from other converters load. The loader required a BOS id in the header and quantized GDN gate projections, which held for Unsloth's files but not for bartowski's or Ornith's own: Qwen prepends no BOS, so those converters omit the key, and bartowski's also leaves the 32-row ssm_alpha/ssm_beta gates as F32. A missing BOS id now falls back to the padding id (the same <|endoftext|> the donor config names for both), and a float gate takes the dense path it already had for small projections. A large float projection is still refused. The shipped model's files are unaffected.

A tool call whose arguments mention the reasoning tags is parsed whole. The parser removed every … span from a turn before reading its calls, including one spelled out inside an edit's old/new text, so the middle of the call vanished and chad could not edit about sixty lines of its own source (the selfsearch improver found this; its fix did nothing). Reasoning is now removed only outside closed tool-call blocks, in the parser, the trajectory writer and the reasoning split alike.

The syntax warning stops crying wolf. .h files were parsed as C, so a correct C++ header with a namespace or a class drew "this edit introduced a syntax error" on the write that created it: 18 of the 19 warnings in a 90-trial eval run. An ambiguous header is now judged by whichever of the C and C++ grammars fits it. A corpus of valid files in some sixty languages (tests/fixtures/syntaxgate_corpus) also caught grammars that reject idiomatic code (CSS nesting, SCSS, Groovy, PowerShell, Tcl, V, assembler), which are no longer policed; JSON files that are JSONC by convention (tsconfig.json, .vscode/) are left alone; a byte-order mark no longer fails the Python check; and the Python warning names the interpreter it ran under, since chad's may be older than the project's.

A slip in how a tool call is filled in no longer costs a step. Every such slip used to be answered with a rejection, and the model regenerated the whole call: once, a 990-token Java file re-emitted because the first attempt had no path. Now an edit with no path is applied to the one file this turn read or changed that contains old (never for write, never to a file the turn did not touch). A hallucinated name like write_file or str_replace_editor runs the tool it meant, and a built-in called with another built-in's parameters runs that tool. The result says so in its first line. A key that is not a parameter is named in the rejection. Two silent faults in the XML call dialect are gone: a literal inside a write's content truncated the file, and a space after <parameter= dropped the parameter. Each new behaviour is a lever (tool_name_alias, edit_path_inferred), and a correct call's result is byte-identical. On 30 tasks × 3 the block went 86/90 → 87/90 with zero rejections of these kinds.

CHAD_BASH_MAX_CHARS sets the per-result cap on bash output before the head/tail clip (default 20,000, floor 2,000; the full body still spills to disk). At the default nothing changes. A 6,000 cap was measured on the 11 tasks whose output crossed it: no pass moved, and it cost 1.47x the wall clock in re-fetch and re-run turns, so the default stays where it was.