NoLlama 2026-08-24

VLM slots are agent-grade

The headline. A vision model slot used to be the second-class one — it could
see, but it couldn't call tools, couldn't cache a prefix, and paid full
prefill on every restart. All three are fixed, on both API surfaces.

Tool calling works on VLM slots. Qwen3.5-4B on an Arc 140V and Muse
Glimmer on an Arc Pro B60 both return a structured
get_weather({"city":"Oslo"}) with finish_reason=tool_calls, and images
may ride along with tools in the same turn. Glimmer's reasoning stays inside
<think> with no channel leak.

Prefix caching works on VLM slots. VLMPipeline honours
scheduler_config — the long-standing "the CB backend is LLM-only" belief
was simply stale. Measured through NoLlama's serving path:

	prompt	cold	repeat
Arc 140V iGPU, 2026.3 release	~9k tokens	21.7 s	3.9 s
Arc Pro B60, 2026.4 nightly	33k tokens	53.7 s	1.4 s

Prewarm covers VLM slots, so the first real turn after a restart is a
cache hit rather than a cold prefill. Glimmer on the B60, measured through
the network API: 12.4 s → 0.65 s TTFT. The 12.1 s prefill still happens
— it is just paid at startup, before the port answers requests.

One honest caveat, because it will bite someone: the CB backend prefills
slower when cold. That same 33k prompt takes ~8.7 s on the plain pipeline
versus 53.7 s under CB, then 1.4 s on every repeat. Agents win from turn
two; one-shot prompts pay more once. --no-prompt-cache restores the plain
pipeline if that trade is wrong for you.

Slots whose runtime rejects the property fall back to the plain pipeline and
zero their KV pool, so /health stays honest about a dead cache and prewarm
skips them instead of burning a 30B prefill for nothing.

Muse Glimmer no longer needs --backend optimum

It is a VLM, so it is now served on the GenAI path like any other. That
un-does the one regression the earlier reroute carried: Glimmer agent use
works on the fast path, tool calls included.

Gemma 4 E4B: use our re-export for agent work

Intel's OpenVINO/gemma-4-E4B-it-int8-ov traced attention decomposed, so
it can get no prefix caching at all — it logs prefix caching unavailable
and drops to the plain pipeline, on every runtime we tried. We re-exported
the same weights with the fused SDPA op:
aweussom/gemma-4-E4B-it-int8-ov.

Caching works there, ~2.6× faster per turn on a repeated agent prompt,
with byte-identical answers. Reported upstream.

Intel's build is still the better pick for one-shot vision — roughly
2.2× faster on a cold turn. Two different jobs, two different builds; the
registry now points at ours and says why.

models.json: 35 models → 24

Pruned to measured-or-best-pick. Every remaining entry is one we have run,
or the best available option in a class we could not test. Entries carrying
opinions rather than measurements are gone, and the Gemma 4 notes were
rewritten to match what the hardware actually did.

Memory sizing: two fixes

Container memory limits are now respected. /proc/meminfo reports the
host total inside a container, so a --memory=4g container was sizing a
4 GB KV cache pool on top of 1.6 GB of weights — silently, with no warning.
That is issue #21's Got unfinished GenerationStatus waiting to happen.
NoLlama now takes the smaller of physical RAM and the cgroup limit (v2 and
v1). Same container now sizes 2 GB and warns that agent prompts will exhaust
it.

Models with a huge single tensor now load where the device caps
allocations. Every GPU caps a single allocation below its total memory —
OpenCL only guarantees a quarter of global memory, and real devices measured
between 0.12 and 1.0 of total. When one weight tensor exceeds that cap the
load dies with "Exceeded max size of memory object allocation". NoLlama now
compares the model's biggest tensor against the device's cap and passes
GPU_ENABLE_LARGE_ALLOCATIONS only when it actually needs to.

Flags and installer

• --prewarm with a nonzero --idle-timeout is now refused at startup
  rather than quietly doing something useless. An idle unload throws away
  the warmed cache, and re-warming on reload would stall the triggering
  request — so the two settings never made sense together.
• install.ps1 rejects unknown arguments instead of ignoring them, and
  no longer dies on PATH entries it cannot traverse.

Documentation

Four logic diagrams of the core flows — request routing, slot lifecycle,
token streaming, web UI rendering — indexed in docs/DIAGRAMS.md. Node IDs
are the real function names, and a checker (check-docs.ps1) verifies that
every function a diagram claims to cover still exists and flags diagrams your
diff has invalidated. Production code is at 100% docstring coverage, with
measured claims tagged so a future rewrite can tell evidence from prose.

Not in this release

Docker support (#31) lives
on the docker-support branch. It works — an Intel GPU in a container runs
at native speed — but the /dev/dri path on native Linux is still unverified,
and one model computes wrong on the container GPU path for reasons not yet
understood. It ships when those are answered. There is no NPU in a
container on Windows, in any WSL channel, and that is not going to change:
wslc has no device-passthrough flag at all.