🚀 FastFlowLM v0.9.43 - FLM Benchmarking Tool, KV Cache, Chat Templates, and Tool Calling

📊 FLM Benchmarking Tool

You can now use the FLM benchmarking tool to test models and compare performance across context lengths.

Each benchmark runs from 1k to 32k context length for 8 iterations.

flm bench <model_tag>

The benchmark reports:

• TTFT: Time to first token in seconds.
• Prefill speed: Prompt processing speed in tokens per second.
• Decoding speed: Token generation speed in tokens per second.

Results are printed as a table in the shell and saved as a CSV file in the current folder for later reference.

---

⚡ KV Cache Improvements

FLM now uses a safer KV cache flow that preserves the full conversation history before applying the chat template.

🔁 What Changed

Previously, FLM computed a checksum for incoming messages and handled cache state as follows:

• Cache hit: Manually removed the old message from the message list.
• Cache miss: Cleared the context.

This approach caused several issues: message-list manipulation happened before chat templating, reasoning content could remain in the KV cache indefinitely, and the bos token could be prefilled repeatedly on each conversation turn. Together, these issues could lead to fragmented or incorrectly formatted conversation histories.

The updated flow keeps the original message history intact:

• Checksum tracking: FLM still computes a checksum for incoming messages.
• Cache hit: The existing context is retained while the complete original message history is processed.
• Cache miss: The context is cleared as before.
• Consistent templating: The chat template is always applied to the complete, unbroken message history sent by the client.
• Token-level diffing: FLM tokenizes the fully templated prompt, compares the token IDs against the cached token IDs from the previous turn, removes the overlapping cached portion, and keeps only the newly generated token IDs for prefill.

The new implementation uses a checkpoint and restore mechanism to record the current KV cache state and restore it in the next round of conversation.

This keeps reasoning content out of the KV cache, making cached conversations more consistent and reliable, especially for complex prompts. It also helps ensure the final templated prompt remains correctly formatted.

Thanks to @NVolcz for inspiring this improvement and providing feedback on the previous flow!

---

🧩 Chat Template Updates

The Gemma4 chat template has been updated to match the latest official release from Google. To support the improved KV cache logic, we also updated chat templates for several model families so they can safely handle complete message histories.

Updated model families:

• gemma4-it
• qwen3.5
• medgemma
• deepseek-r1

You do not need to redownload all model files manually. Use flm list to check whether your local model files are compatible with the latest chat templates.

There are two ways to update a model's chat template:

1. Run flm check <model_tag> and then flm pull <model_tag> to update the model before your next use.
2. Run flm run <model_tag> or flm serve <model_tag>, and FLM will automatically check and update the chat template if needed.

⚠️ Note: The checking process may take about 20 seconds, depending on the model, but it only needs to run once per model version. After the chat template is updated, you can run or serve the model as usual without the extra delay.

---

🔄 Smarter Force Pulls

flm pull <model_tag> --force now checks existing model files first and only re-downloads missing or incorrect files, instead of removing and re-downloading the entire model.

---

🧰 Gemma4 Tool-Calling Message Formatting

Gemma4 uses a different tool-calling message format from OpenAI-compatible tool-calling APIs. Previously, upstream OpenAI-formatted tool-call messages could be sent directly to Gemma4 without conversion, which could lead to parsing issues.

FLM now detects OpenAI-formatted tool-call messages and converts them to the Gemma4 tool-calling format before sending them to the model. This improves parsing reliability for Gemma4 tool calls.

For more details about the format differences, see the Gemma4 tool-calling documentation.

Thanks to @NVolcz for pointing this out!

---

🛠️ Gemma4 Tool-Calling Reliability

We received a lot of feedback about Gemma4 tool-calling reliability, especially around tool-argument parsing.

Tool arguments are generated by the model as text and then parsed into JSON. Parsing can fail for several reasons, including malformed strings, complex nested arguments, or edge-case formatting.

This release improves Gemma4 tool-call argument parsing across several edge cases, making tool calling more robust overall.

Some malformed outputs may still be impossible to recover automatically, such as tool calls that completely miss the expected chat-template format. We will continue improving coverage for these cases where possible.

We will also continue monitoring feedback and improving tool-calling reliability for Qwen3 and Qwen3.5.

Thanks to the community members who shared their tool-calling experiences and edge cases. @TatuLund, @jtmonroe, @cqh963852, @antrv, and more than a dozen others contributed suggestions that helped improve tool-calling reliability.

---

🔕 Tool Calling Disabled for Low-Reliability Models

Tool-calling support has been disabled for the following models due to low usage and low reliability:

• lfm2.5-tk:1.2b
• nanbeige4.1:3b

---

🌟 Summary

FastFlowLM v0.9.43 adds benchmarking output for easier performance comparison, makes cached conversations more reliable by preserving complete message histories, keeping reasoning content out of the KV cache, and using token-level diffing after chat templating. This release also updates chat templates for Gemma4, Qwen3.5, MedGemma, and DeepSeek-R1, makes flm pull <model_tag> --force more efficient by re-downloading only missing or incorrect files, improves Gemma4 tool-call formatting and argument parsing, and disables tool calling for low-reliability models.