bundle-20261015c: images from agent tools on the Responses API (+ everything in 20261015b)

bundle-20261015c fixes agent tools that return images through the Responses API (patch 0058, a cherry-pick of
ggml-org/llama.cpp 4098fdc922). A harness on /v1/responses whose tool returned a screenshot got HTTP 400 ("Output
of tool call should be 'Input text'"), and every later request in that session failed too (#112). Now an
input_image in a tool output reaches the model like an image in a user message (with BONSAI_MMPROJ set).
Server-only change; the attention and cache code are the same as bundle-20261015b.

bundle-20261015b fixes the launcher of bundle-20261015: its q3_0 check never matched (two control characters in
the pattern), so 8 GB cards stayed on q4_0. The engine is the same; with 15b the 8 GB default is q3_0 as described
below. If you run bundle-20261015, take the new zip or set BONSAI_CTK=q3_0.

bundle-20261015 makes 8 GB cards faster and puts more of the window in VRAM (patches 0056 and 0057).

• q3_0 K/V is the new 8 GB default (patch 0056: 3.5 bits per value, 14,336 B per position instead of 18,432 for
  q4_0). The VRAM line holds 1.31x the positions (RTX 2060 SUPER, 262k: 61,696 -> 80,640). Decode against q4_0 on the
  same build: 4k -1.8 %, 64k +3.5 %, 128k +44 %, 192k +34 %. Perplexity at 16k +0.56 % against q4_0;
  HumanEval 164 163/164 (q4_0 158/164); exact copies and needles at 160k equal to q4_0. The rule for the default was
  written before the runs (docs/research/q3port-20261013/). BONSAI_CTK=q4_0 keeps the previous default; an
  engine without q3_0 keeps q4_0 by itself.
• Faster K/V tile loads in the tensor-core attention kernel (patch 0057, bit-identical output): RTX 2060 SUPER,
  q4_0 at 64k 35.7 -> 39.6 tok/s (+11 %), 128k 15.7 -> 16.9 (+7.6 %), the same answers. RTX 4070 (12 GB, q8_0):
  within 1 %, HumanEval answers identical on 164 of 164.
• BONSAI_SPARSE=1 now keeps 1/8 of the host tail (was 1/16). On wider item sets at 245k, 1/16 lost a fact past
  the VRAM line on both test cards; 1/8 kept every item. RTX 2060 SUPER: 160k 1.59x, 245k 1.76x dense (1/16:
  1.65x / 2.2x). Still opt-in.

Zip: bonsai-bundle-win-x64.zip, 616,639,105 bytes, SHA-256 a46ae100baef2ba38ad48a0a40c28c0e975d394e958e29dbe7b140a272f2560a. Engine c2cf6d6450 (PrismML adfffbe + 58 patches), sm 75 / 86 / 89 + PTX; launcher and docs from 0d45cd03. Gates: docs/receipts/release_0058.log (op tests, the 8 GB launcher path, the Responses tool-image repro, the chat path).

bundle-20261015b: q3_0 K/V default on 8 GB (launcher fix), faster tile loads, sparse budget 1/8

Superseded by bundle-20261015c (the same launcher and attention code, plus the Responses API tool-image fix, #112).

bundle-20261015b fixes the launcher of bundle-20261015: its q3_0 check never matched (two control characters in
the pattern), so 8 GB cards stayed on q4_0. The engine is the same; with 15b the 8 GB default is q3_0 as described
below. If you run bundle-20261015, take the new zip or set BONSAI_CTK=q3_0.

bundle-20261015 makes 8 GB cards faster and puts more of the window in VRAM (patches 0056 and 0057).

• q3_0 K/V is the new 8 GB default (patch 0056: 3.5 bits per value, 14,336 B per position instead of 18,432 for
  q4_0). The VRAM line holds 1.31x the positions (RTX 2060 SUPER, 262k: 61,696 -> 80,640). Decode against q4_0 on the
  same build: 4k -1.8 %, 64k +3.5 %, 128k +44 %, 192k +34 %. Perplexity at 16k +0.56 % against q4_0;
  HumanEval 164 163/164 (q4_0 158/164); exact copies and needles at 160k equal to q4_0. The rule for the default was
  written before the runs (docs/research/q3port-20261013/). BONSAI_CTK=q4_0 keeps the previous default; an
  engine without q3_0 keeps q4_0 by itself.
• Faster K/V tile loads in the tensor-core attention kernel (patch 0057, bit-identical output): RTX 2060 SUPER,
  q4_0 at 64k 35.7 -> 39.6 tok/s (+11 %), 128k 15.7 -> 16.9 (+7.6 %), the same answers. RTX 4070 (12 GB, q8_0):
  within 1 %, HumanEval answers identical on 164 of 164.
• BONSAI_SPARSE=1 now keeps 1/8 of the host tail (was 1/16). On wider item sets at 245k, 1/16 lost a fact past
  the VRAM line on both test cards; 1/8 kept every item. RTX 2060 SUPER: 160k 1.59x, 245k 1.76x dense (1/16:
  1.65x / 2.2x). Still opt-in.

Zip: bonsai-bundle-win-x64.zip, 616,638,401 bytes, SHA-256 a6bf7adda3ed97de03a8105fb97b91afa68207544abea24e3b33572bb4035776. Engine b86298dba9 (unchanged from bundle-20261015), launcher and README from 444de562. Gates: docs/research/release-20261014b/ (S01-S06, S06b).

bundle-20261015: q3_0 K/V default on 8 GB, faster tile loads, sparse budget 1/8

Superseded by bundle-20261015b. This zip's launcher does not select q3_0 on 8 GB (a control character in its check, #108); 8 GB cards stay on q4_0. Take 15b, or set BONSAI_CTK=q3_0.

bundle-20261015 makes 8 GB cards faster and puts more of the window in VRAM (patches 0056 and 0057).

• q3_0 K/V is the new 8 GB default (patch 0056: 3.5 bits per value, 14,336 B per position instead of 18,432 for
  q4_0). The VRAM line holds 1.31x the positions (RTX 2060 SUPER, 262k: 61,696 -> 80,640). Decode against q4_0 on the
  same build: 4k -1.8 %, 64k +3.5 %, 128k +44 %, 192k +34 %. Perplexity at 16k +0.56 % against q4_0;
  HumanEval 164 163/164 (q4_0 158/164); exact copies and needles at 160k equal to q4_0. The rule for the default was
  written before the runs (docs/research/q3port-20261013/). BONSAI_CTK=q4_0 keeps the previous default; an
  engine without q3_0 keeps q4_0 by itself.
• Faster K/V tile loads in the tensor-core attention kernel (patch 0057, bit-identical output): RTX 2060 SUPER,
  q4_0 at 64k 35.7 -> 39.6 tok/s (+11 %), 128k 15.7 -> 16.9 (+7.6 %), the same answers. RTX 4070 (12 GB, q8_0):
  within 1 %, HumanEval answers identical on 164 of 164.
• BONSAI_SPARSE=1 now keeps 1/8 of the host tail (was 1/16). On wider item sets at 245k, 1/16 lost a fact past
  the VRAM line on both test cards; 1/8 kept every item. RTX 2060 SUPER: 160k 1.59x, 245k 1.76x dense (1/16:
  1.65x / 2.2x). Still opt-in.

Zip: bonsai-bundle-win-x64.zip, 616,638,283 bytes, SHA-256 23db0bc477274314e817f3eaeda185b9ac314f31d346f45fc4861b2e3ae7dd16. Engine b86298dba9 (PrismML adfffbe + 57 patches), sm 75 / 86 / 89 + PTX. Gates: docs/research/release-20261014b/ (S01-S05) and release-20261014/ R10-R11 (sparse budget).

bundle-20261014: sparse host-tail reads for deep sessions on 8 GB (patch 0055, opt-in)

Sparse host-tail reads for deep sessions on 8 GB cards (patch 0055, opt-in: BONSAI_SPARSE=1).

Same model files. New engine (4de6185494 = PrismML adfffbe + 55 patches) and launcher.

• What it does. Past the VRAM line, the K/V cache lives in system RAM and every decode step reads all of it over PCIe. With sparse on, a decode attention op scores each 64-cell page of that system-RAM part with int8 page bounds (an upper bound on its attention scores), reads only the best pages (a budget that grows with the host tail, at least 12,288 cells) plus the last 1,024 cells, and skips the rest.

• Speed, mean decode (code / prose / bash):

  card	depth	dense	sparse	ratio
  RTX 2060 SUPER, 8 GB recipe, 262k	160k	10.6 tok/s	17.5	1.65x
  RTX 2060 SUPER, 8 GB recipe, 262k	245k	6.3	13.7	2.17x
  RTX 4070, line pinned 8 GB-like	160k	14.4	24.1	1.67x
  RTX 4070, line pinned 8 GB-like	245k	8.8	20.4	2.32x

• Quality. Exact code copies and needle answers equal to dense on the same items at 160k, 200k (20 items) and 245k, on both cards. Op tests pass.

• Why opt-in. The page bounds take VRAM (about 170 MiB at 262k), so the VRAM line moves down: decode below about 100k is about 8 % slower on PCIe 3.0 (2.6 % on PCIe 4.0). The launcher prints the trade. With sparse off (the default), nothing changes.

• Also: with a separate draft model (-md), the launcher turns the shared CUDA pool off (it aborted at start). The README has a new Vision (images) section.

Upgrade

git pull, unzip bonsai-bundle-win-x64.zip over the repo, restart start-server.ps1. For deep sessions on 8 GB: $env:BONSAI_SPARSE = '1' before start-server.ps1.

SHA-256 c91dfc98f82c74db93370568e47210d2e6c98c2907ed7aef872ba3bf5ab24cd0

bundle-20261013: faster prefill on RTX 30 and 40 (patch 0054)

Faster prefill on RTX 30 and 40 cards: chunked gated-delta-net prefill with BF16 tensor-core math (patch 0054).

Same model files and launcher. New engine (de873117e1 = PrismML adfffbe + 54 patches).

• Prefill (patch 0054, a port of ggml-org/llama.cpp#29353). The model's 48 gated-delta-net layers processed a prompt one token after another. Now they process it in chunks with BF16 MMA (tensor-core) math. Below 64 tokens, and with speculative-decoding rollback steps, the old kernel still runs.
  RTX 4070, 12 GB recipe, 262k window, bench/depth_arms.sh, the same launcher settings for both arms:

  depth	prefill before	prefill after	mean decode before	mean decode after
  4k	1197 tok/s	1280 (+6.9 %)	97.0	95.4
  32k	1020	1082 (+6.1 %)	102.6	102.8
  128k	583	602 (+3.3 %)	55.1	54.8

  Quality: wikitext-2 perplexity 8.0714 -> 8.0733 (+0.02 %, inside the error bar of 0.117). HumanEval 164 at medium, greedy, on the bundle's own binaries: 158/164 (160/164 before: two long-thinking answers changed with the float order).

• RTX 20 cards have no BF16 MMA. They keep the old kernel and log one line: chunked GDN prefill (MMA) skipped: device cc 750 .... RTX 2060 SUPER, 8 GB recipe, 32k: prefill 347.3 vs 344.9 tok/s, decode 45.1 vs 44.8 (within 0.7 %), 0 one-token answers.

• GGML_CUDA_GDN_MMA_DISABLE=1 turns the new kernel off.

Upgrade

git pull, unzip bonsai-bundle-win-x64.zip over the repo, restart start-server.ps1.

SHA-256 dda336ab1473707724078e42f81e720cbf7c286954004db85827c602b03e5a51

bundle-20261011: faster decode past the VRAM line (patch 0053)

Faster decode past the VRAM line: CUDA graphs off where attention reads the host tail (patch 0053).

Same model files and launcher. New engine (90ca41e40 = PrismML adfffbe + 53 patches).

• Decode at depth (patch 0053). The tier prefetch pipeline (the copy engine moves the next attention op's K/V rows from system RAM into VRAM while the current op computes) ran only without a CUDA graph, and the launcher keeps CUDA graphs on. So past the VRAM line, every copy waited for compute. Now a graph whose attention reads the system-RAM part of the cache runs without a CUDA graph, and the copies overlap with compute. Graphs below the line keep their CUDA graphs. The server logs one line on each switch. GGML_CUDA_KV_TIER_GRAPHS=1 keeps the old behaviour.
  RTX 4070, 12 GB recipe, 262k window, bench/quick_tps.py mean decode (code / prose / bash):

  depth	before	after
  4k	95.2 tok/s	95.4
  32k	101.8	102.3
  128k (below the line)	71.9	71.8
  192k	23.6	29.4
  250k	12.7	14.6

  Lookup off, 192k: 19.8 -> 25.0 tok/s.

  Same text: HumanEval 164 at medium, greedy, on the bundle's own binaries: 160/164 with the same 4 fails, and all 164 outputs identical to the previous engine.
  RTX 2060 SUPER, 8 GB recipe (#22, P103), mean decode: 128k 12.8 -> 14.8, 192k 7.9 -> 8.6, 245k 6.2 -> 6.7, 192k lookup off 7.8 -> 8.7 tok/s; 4k / 32k unchanged (53.8-54.0 / 48.2-48.3, alternated twice); 0 one-token answers.

Upgrade

git pull, unzip bonsai-bundle-win-x64.zip over the repo, restart start-server.ps1.

SHA-256 f8899cbfd98091b6830615dc83182626b0b551d8b833ceea58c4e2e52cda8bf8

bundle-20261010: system RAM sized from the machine (#16), faster 8 GB prefill at depth (#19)

System RAM sized from the machine (#16), faster 8 GB prefill at depth (#19), a faster verification step (patch 0050).

Same model files. New engine (2cadbfb64 = PrismML adfffbe + 52 patches) and launcher.

• System RAM (#16, reported by @howdya; #18). The server's memory is its VRAM (Windows charges it to the process as well), the K/V past the VRAM line, the prompt cache (default limit 8 GiB; one entry for a 26k-token prompt is ~2 GB with its checkpoint copies) and the checkpoints. On 16 GB of RAM this ended in "bad allocation" and a server exit. The launcher now sizes the cache and the checkpoints from the RAM (ram line):
  • 48 GB or more: the server defaults;
  • 24-47 GB: a 4 GiB prompt cache and 32 checkpoints (RTX 4070, 12 GB recipe, 7 thinking requests with 26k-token prompts: peak 21.7 -> 17.5 GB, 7/7);
  • under 24 GB: sized from the free memory at start (RTX 2060 SUPER limited to what a 16 GB machine has: 7/7, peak 10.5 GB).
    BONSAI_RAM_BUDGET=0 keeps the server defaults; LLAMA_ARG_CACHE_RAM / LLAMA_ARG_CTX_CHECKPOINTS set by hand win.
• 8 GB: chunked prefill attention at depth (#19, patches 0051-0052). Past the f16 prefill limit, prefill attention runs on 32k-cell chunks converted to f16, and rows in system RAM cross PCIe once per op. RTX 2060 SUPER at 262k: prefill 121 -> 171 tok/s at 128k, 58 -> 116 at 192k, 25 -> 90 at 250k; decode unchanged; perplexity 7.2107 (f16) / 7.2074 (chunked). The 12 GB recipe keeps it off for now; GGML_CUDA_FA_CHUNK=32768 turns it on (RTX 4070: +12-14 % prefill at 128k-249k, ~200 MiB less free VRAM).
• Patch 0050: the gated-delta-net conv-state concat in multi-token steps (draft verification) uses the engine's transpose kernel on every NVIDIA card, not only GB10. Decode +1.4 %, same text, HumanEval 160/164 before and after.

Upgrade

git pull, unzip bonsai-bundle-win-x64.zip over the repo, restart start-server.ps1.

SHA-256 2f3b6544ab2e8cd02bf9a2a16e246ebbdeb66987ce4a14891f0a153c3e46da6c

bundle-20261009-vram: more of a long context in VRAM (12 GB +18% at 131k, 8 GB 262k)

More of a long context in VRAM: 12 GB decode +18% at 131k; 8 GB cards get the full 262k window.

Same model files. The engine adds patches 0047-0049 (from the 8 GB work, PR #13); new launcher.

12 GB recipe

• Packed (1-bit) attention mask (patches 0048-0049). The f16 mask is window x micro-batch x 2 bytes: 256 MiB at 262k. Packed, it is 16 MiB. The launcher adds the 240 MiB to the VRAM line: 120,064 -> 128,000 positions on the RTX 4070. Decode: 40.9 -> 48.2 tok/s at 131k (+18%), 21.2 -> 23.0 at 180k (+8.5%); prefill the same (receipts/mask12_ab.log). With MTP drafting + lookup at 32k and 64k, 22 of 22 texts are identical to the f16 mask (receipts/pr13_ada.jsonl). LLAMA_ARG_KQ_MASK_PACKED=0 turns it off.
• Also on the 12 GB recipe since today's earlier bundles: the shared CUDA pool, and checkpoints inside long messages.

8 GB cards

• The full 262,144-token window in drafting mode, with 24 GB of system RAM or more (less RAM: 131k as before). Partial staging (0047) stops the VRAM staging buffer from growing with the window, the packed mask and the shared pool free more VRAM, and the line moves up, so 262k decodes faster than the old 131k preset at every measured depth (RTX 2060 SUPER: 30.5 -> 37.8 tok/s at 64k, 13.0 -> 14.2 at 128k; 10.4 at 160k). Prefill past the staged depth is slower (59 tok/s at 160k). Details and receipts: PR #13, docs/8GB.md, receipts/262k_8gb/.

Checks

Release smoke: greedy outputs identical to the previous engine (receipts/release_smoke_0049.log). Fresh 20k-60k prompts: normal answers on the 12 GB recipe (line 128,000) and on the 8 GB preset at 262k (receipts/one_token_ab.log, receipts/release_vram_checks.log).

Upgrade

git pull, unzip bonsai-bundle-win-x64.zip over the repo, restart start-server.ps1. Binaries: sm_75 / 86 / 89 machine code (RTX 20 / 30 / 40) plus compute_89 for RTX 50, CUDA 13 runtime included, NVIDIA driver only. Engine commit d90e35528 (branch bonsai-q8-product of the fork; the 49 patches in patches/ give the same tree).

SHA-256 430aa2be0c2a03512641dd68fdc08895ee9bc3b12e284793b0bf9edfe440207d

bundle-20261009-probe: launcher fix for Windows PowerShell 5.1 (#14)

Launcher fix: Windows PowerShell 5.1 with Python but without wasmtime no longer stops before the server starts (#14).

Same model files and engine as bundle-20261009-vram (engine d90e35528). New launcher.

• The launcher probes the optional layer's wasmtime package. Under $ErrorActionPreference = 'Stop', Windows PowerShell 5.1 turned the Python traceback on stderr into a terminating error, so on a machine with Python on PATH and without wasmtime the launcher stopped before the server started (reported by @picmax in #14; fixed in #15). The probe now runs only when the layer's runtime is present, and it cannot stop the launcher. Without the runtime the launcher prints layer off: run layer\fetch_runtime.ps1 once to enable it and starts the server.
• Everything else is as in bundle-20261009-vram (the packed KQ mask on 12 GB, the 262k window on 8 GB).

Upgrade

git pull, unzip bonsai-bundle-win-x64.zip over the repo, restart start-server.ps1.

SHA-256 38a4000c3c57bf7bea15cd278f552fe333bebbbc758e1d96307c3cfd3bf6d7ee

bundle-20261009-pool: shared CUDA pool on for 12 GB

12 GB recipe: the shared CUDA pool is on. A little more of a long context stays in VRAM.

Same model files and engine as bundle-20261009-ckpt (engine 5251968b0). New launcher.

• The 12 GB recipe now uses one CUDA memory pool for the main and the MTP draft context, and a 256-token draft micro-batch. That saves 74 MiB of VRAM, so the VRAM line moves up about 1.3k positions (RTX 4070: 118,784 -> 120,064). Decode past the line: 39.3 -> 41.0 tok/s at 131k, 20.7 -> 21.2 at 180k; the same below the line. Fresh 20k-60k prompts: normal answers (receipts/pool_ab.log).
• It is on only with an engine that has patch 0045, which orders the two CUDA streams so the pool is safe with drafting. BONSAI_SHARED_POOL=0 turns it off. The startup kv line says when it is on.
• The 8 GB preset is unchanged.

Upgrade

git pull, unzip bonsai-bundle-win-x64.zip over the repo, restart start-server.ps1.

SHA-256 1144b9042f3542cb9d26a84d4b32c2cbc562fc41c2cb1ab46246cccb10fadd94