🚀 FastFlowLM v1.0.11 — Decision Models, a Prefix Cache for Every Chat Model & Logprobs

This release does four things:

• New models. The Decision 3.0 (d3) family from vLLM Semantic Router: five decision models from 0.8B to 27B, served through a new /v1/systemone endpoint. Also LFM3-1.7B and Nemotron 3.5 Bolt E4B.
• The prefix cache now covers almost every chat model. flm serve keeps several saved prompts at once, so agents that share a system prompt and tools, or switch between conversations, don't re-read the whole prompt on every request.
• More OpenAI API fields: logprobs / top_logprobs, seed, continue_final_message and repeat_last_n.
• max_tokens is exact. A reply used to be one token longer than max_tokens asked for.

It also fixes a gemma4-it:12b crash on repeated image or audio requests.

---

🆕 New Models

🧭 Decision 3.0 (d3): d3:lite, d3:nano, d3:mini, d3:turbo, d3:ultra

d3 is the decision model that vLLM Semantic Router calls for routing, guardrails and tool selection. It doesn't generate text. You send it a state (any JSON) and one or more questions, and it returns a probability for every option:

• choice: pick one of up to 255 options
• noul: no / yes
• score: 2 to 10 ordered levels

FLM serves the same HTTP API as the upstream d3 server, so vLLM Semantic Router can use FLM as its d3 backend without changes.

A decision model runs in server mode, on its own or next to a chat model:

flm serve --decision d3:lite                 # decision model only
flm serve qwen3.5:4b --decision d3:lite      # next to a chat model

Tag	Backbone	Download	Inputs	One short question*
d3:lite	Qwen3.5-0.8B	0.8 GB	text, images	~0.16 s
d3:nano	Qwen3.5-2B	2.0 GB	text, images	~0.26 s
d3:mini	Qwen3.5-4B	3.4 GB	text, images	~0.56 s
d3:turbo	Qwen3.5-27B	16.6 GB	text, images, videos, box / track / assets	~3 s
d3:ultra	Qwen3.5-27B (newer fine-tune)	16.6 GB	same as d3:turbo	~3 s

* About 330 tokens, over HTTP, on an idle machine.

• Endpoints.
  • POST /v1/systemone (alias /v1/decisions) answers decision requests.
  • GET /v1/systemone/models lists the decision models this build can serve.
• Input limit: 32,768 tokens per question. A longer prompt gets max_length_exceeded for that question. Nothing is truncated.
• Accuracy. On the Decision Index public suite, every model stays close to the upstream BF16 model. Each model's numbers are on the benchmark page.
• d3:turbo and d3:ultra:
  • The state is computed once per request. Every question then reads it, so many questions about one state cost little more than one. That is 2.5× faster at 16 questions and up to 32× with a 4k-token state.
  • box and track questions locate objects in images and follow them through videos. assets put images in questions, options and instructions.
  • Videos need the FFmpeg shared libraries (FFmpeg 6, 7 or 8) on the machine. FLM doesn't ship FFmpeg. It loads it the first time a video arrives. Without it, encoded videos get a clear 422, and frame lists ({"frames": [...]}) still work. See the d3 page for install steps.
  • Memory. Plan for about 28 GB of free memory.
  • long_context: true is refused with a 422.

📖 d3 models · d3 benchmarks

---

💧 LFM3-1.7B: lfm3:1.7b

LiquidAI LFM3-1.7B runs fully on the NPU, prefill, all 24 layers and the LM head included.

• Thinking is on by default. You can turn it off.
• Supports tool calling.
• Context: up to 32k tokens.
• Uses LiquidAI's Q4_0 quantization-aware build.
• Uses LiquidAI's recommended sampling defaults: temperature 0.1, top_k 50, repetition penalty 1.1.

flm run lfm3:1.7b

---

⚡ Nemotron 3.5 Bolt E4B: nemotron3.5-bolt:e4b

NVIDIA Nemotron-3.5-Bolt-E4B is Gemma 4 E4B post-trained for multi-turn tool use, instruction following and fast voice and device assistants.

• Takes text, images and audio, like gemma4-it:e4b.
• Context: up to 64k tokens (32k by default).
• "conversational_mode": true in a request adds Bolt's short-answer instruction to the prompt.

flm run nemotron3.5-bolt:e4b

Better tool calls on Gemma 4 E2B/E4B too. gemma4-it:e2b and gemma4-it:e4b now render tool calls and tool results in Gemma 4's own format. Before, FLM turned them into a JSON blob the model wasn't trained on. Prompts now match Hugging Face's template on all 14 test cases, up from 2.

---

✨ New Features

🗂️ Prefix Cache for Every Chat Model (Multi KV cache slots)

Until now, FLM reused a cached prompt only when the new prompt continued the last one. If an agent sent a side call (a title, a summary) or another conversation ran in between, the next request prefilled everything again.

In v1.0.11, flm serve keeps up to 8 saved prompts in host memory and resumes from the longest one that matches the start of the new prompt. FLM also saves the end of the system prompt and tools as its own snapshot, so different conversations that share them all reuse it. Images are matched by content, so a conversation with a picture resumes too.

Time to first token, cold vs. cached:

Model	Prompt	Cold	Cached
gemma4-it:12b	2.2k tokens	9.3 s	1.7 s
gpt-oss:20b	2.1k tokens	7.7 s	1.5 s
gemma3:4b	2.2k tokens	3.3 s	1.0 s
minicpm-v-4.7:1b	1.6k tokens, with a picture	1.3 s	0.2 s
qwen3.8-mtp:27b	1.3k tokens (shared system prompt only)	16.5 s	3.6 s

On Qwen3.5-4B, four rotating 15k-token conversations take 1.7–2.1 s per turn instead of 27 s.

Covered models: Qwen3.5, Qwen3.6-MoE, Qwen3, Qwen3-VL, Qwen2.5 and Qwen2.5-VL, Llama 3.x, DeepSeek-R1 (both 8B models), Phi-4-mini, Nanbeige, LFM2 / LFM2.5, LFM3, gpt-oss and gpt-oss-safeguard, Gemma 3 (and MedGemma, TranslateGemma), Gemma 4, Nemotron 3.5 Bolt, MiniCPM-V 4.7. Qwen3.8-27B (MTP) keeps only the shared system prompt and tools.

Not covered:

• the single-turn *-flash models;
• flm run and the Ollama endpoints (/api/chat, /api/generate), which keep the older reuse. A request on an Ollama endpoint also empties the prefix cache.

Options:

Option	Default	What it does
--prefix-slots N	8 (max 32, 0 = off)	How many prompts to keep
--prefix-cache-mb N	4096	Host memory budget for saved prompts
--prefix-min-gain N	128	Save a prompt only if it adds at least N tokens over a saved prompt that already covers it

You can also turn the cache off with FLM_DISABLE_PREFIX_SLOTS=1.

Checking it works:

• usage.prompt_tokens_details.cached_tokens shows how many tokens were reused.
• A new usage.flm_prefix_cache object, and one server log line per cache decision, tell you which slot was used and why a request missed.
• Cancelling a request or clearing the context no longer throws away other conversations' saved prompts.

📖 Prefix Cache

---

📊 OpenAI API: logprobs, seed, continue_final_message and repeat_last_n

logprobs and top_logprobs on /v1/chat/completions, streamed or not, as in the OpenAI API:

• top_logprobs accepts 0 to 100.
• The values come from the model before penalties, constraints, temperature and top-k/p are applied. This matches OpenAI and vLLM.
• Only content tokens get entries. Reasoning, tool calls and template markup get none.
• qwen3.8-mtp is supported too, speculated tokens included. Its logprobs can differ slightly from plain decoding. Set FLM_DISABLE_SPECULATION=1 to compare.

seed is now applied on /v1/chat/completions and /v1/completions. The same request with the same seed gives the same text. To make this hold:

• the engines now clear the KV cache fully between prompts;
• gpt-oss:20b's Windows prefill now gives the same result on every run. ⚠️ This makes short gpt-oss:20b prompts on Windows a little slower to start, about +11% at 91 tokens. Long prompts are unchanged.

continue_final_message: true (the same field as in vLLM) continues a partial assistant message instead of starting a new reply. Combined with max_tokens: 1 and logprobs, you can score a classification or multiple-choice answer through the model's own chat template:

{"messages": [
   {"role": "user", "content": "Is Paris the capital of France? Answer yes or no."},
   {"role": "assistant", "content": "The answer is:"}],
 "continue_final_message": true, "max_tokens": 1, "logprobs": true, "top_logprobs": 5}

• It works on every multi-turn model except gpt-oss.
• Single-turn models, gpt-oss, and requests it can't serve return a 400. That includes a last message that isn't from the assistant, forced JSON or tool calls, and add_generation_prompt: true.

repeat_last_n sets how many generated tokens the presence, frequency and repetition penalties look at. It is an FLM extension with the same name as in llama.cpp and Ollama. -1 is the whole reply and 0 turns the penalties off. The default is still 64.

📖 Request Parameters

---

🐛 Bug Fixes

• max_tokens was off by one. max_tokens: N produced N+1 tokens while usage.completion_tokens reported N. This is fixed for every model, the MTP speculative loop included.
• The prompt cache can no longer skip or repeat a token. A token cut off by max_tokens, a cancel or the context limit was recorded as if it had run. The next request's cache match could then skip or misplace a token.
• gemma4-it:12b no longer crashes on a repeated image or audio request. Sending the same request with audio or an image twice made flm serve exit.
• Gemma 3 no longer writes <0x0D> (and other byte tokens) as literal text, for example before a JSON answer.
• Single-turn models now return a 400 for conversation history. Before, they returned a 200 with an error body.
• Requests with no messages or prompt now get a 400.
• Qwen3.5-VL images with an odd side size (for example 80 px) now produce the same number of image tokens as Hugging Face, which rounds half to even.

---

🌟 Summary

	Highlight
🧭	Decision 3.0 (d3:lite to d3:ultra) on the NPU, served on /v1/systemone for vLLM Semantic Router
💧	New model: lfm3:1.7b
⚡	New model: nemotron3.5-bolt:e4b, plus Gemma 4's own tool-call format on E2B/E4B
🗂️	Prefix cache on almost every chat model: up to 8 saved prompts, much faster agent turns
📊	logprobs / top_logprobs, including qwen3.8-mtp
🎲	seed gives the same output on every run; repeat_last_n added
✍️	continue_final_message to continue or score a partial answer
🎯	max_tokens is exact, and no more gemma4-it:12b crash on repeated image or audio requests

Thanks for your support — see you in the next one! 🚀