🚀 FastFlowLM v0.9.46 — We're Moving!

🏠 FastFlowLM is Now an Official AMD Project

🎉 FastFlowLM has joined the ROCm organization and is now officially maintained by AMD. This is the last release under FastFlowLM/FastFlowLM — starting from v1.0.0, everything moves to ROCm/FastFlowLM. Please update your bookmarks, forks, and remotes accordingly. See you there!

---

🌐 ModelScope Support

Models are pulled from HuggingFace by default. You can now opt into ModelScope as an alternative source with a single flag.

Pull a model from ModelScope:

flm pull llama3.2:1b --modelscope 1

Auto-pull from ModelScope in CLI mode:

flm run llama3.2:1b --modelscope 1

Auto-pull from ModelScope in server mode:

flm serve --modelscope 1

Check the compatibility of your local model with ModelScope:

flm check llama3.2:1b --modelscope 1

---

🖼️ More Image Resize Levels for Qwen-VL Models

Fine-grained control over input image resolution is now available via the -r flag:

flm serve -r <img-pre-resize-level>

Level	Resolution
0	Original size
1	Height = 480
2	Height = 720
3	Height = 1080
4	Height = 1440
5	Height = 2160
6	Height = 2880
7	Height = 3240
8	Height = 4320

---

⚡ Speed Improvements for Qwen3.6-MoE

Both prefill and decoding throughput have been improved for Qwen3.6-MoE across all context lengths.

Decoding throughput (tokens/s):

Context	Old	New	Gain
1k	12.41	13.65	+9.99%
2k	12.26	13.41	+9.38%
4k	11.96	13.09	+9.45%
8k	11.38	12.51	+9.93%
16k	10.40	11.24	+8.08%
32k	8.88	9.51	+7.09%

Prefill throughput (tokens/s):

Context	Old	New	Gain
1k	75.18	78.98	+5.05%
2k	109.85	118.04	+7.46%
4k	150.93	156.43	+3.64%
8k	181.56	197.93	+9.02%
16k	214.46	218.84	+2.04%
32k	219.72	221.96	+1.02%

---

🌟 Summary

	Highlight
🏠	FastFlowLM is now an official AMD project — repo moved to ROCm/FastFlowLM
🌐	ModelScope support: pull, serve, and check model compatibility
🖼️	9-level image resize control for Qwen-VL models
⚡	Up to ~+10% decoding and ~+9% prefill speedup for Qwen3.6-MoE

🚀 FastFlowLM v0.9.45 — Qwen3.6-35B-A3B & Smoother KV Cache Through Multi-Backend Support

Here's what's new 🎉

🤖 New Model: Qwen3.6-35B-A3B

Say hello to Qwen3.6-35B-A3B — the second MoE model in FLM, joining GPT-OSS. It packs 35B total parameters with only 3B activated per forward pass, so you get strong reasoning quality at a fraction of the compute cost.

Tag: qwen3.6-moe:35b-a3b

Run in CLI mode:

flm run qwen3.6-moe:35b-a3b

Run in server mode:

flm serve qwen3.6-moe:35b-a3b

Check out the model card and benchmark results for more details.

---

⚡ Smoother KV Cache Through Multi-Backend Support

FLM now supports per-round KV cache checks, making context management more precise and robust when mixing inference backends.

Here's a real example — imagine a Lemonade user running gemma4-it:e2b on both NPU (via FLM) and GPU (via llama.cpp):

1. They send an initial prompt to the NPU and get a response.
2. They continue on the GPU and get a second response.
3. They switch back to the NPU with the full conversation history.

Previously, the NPU would see gaps from the GPU round, fail the KV cache check, and re-prefill everything from scratch. Now, with per-round KV cache checks, it knows the first round is already cached — so it only prefills the GPU round and the new prompt. Mixing backends is no longer a headache! 🙌

---

🌟 Summary

• New model: Qwen3.6-35B-A3B — our second MoE model, with 35B total parameters and just 3B activated per token 🧠
• Smarter KV cache — per-round checks let you mix FLM and other backends without losing cache or re-prefilling the whole context 🔄

Thanks for your support — more good stuff is on the way. See you in the next one! 🚀

🚀 FastFlowLM v0.9.44 — Portable Linux Support

🐧 Portable FLM for Linux

FastFlowLM is now available as a portable build for Linux! This makes it easy to bundle and integrate the app into any higher-level application or custom environment — no installation required, just plug it in and go.

⚡ Quick Start

Download the portable package from the releases page and place it in your home directory, then run:

cd ~/portable
tar -xzf fastflowlm_v0.9.44_linux.tar.gz
cd fastflowlm_v0.9.44_linux
./flm run llama3.2:1b

🙌 Acknowledgements

A huge thank you to @superm1 and the Lemonade team — your invaluable insights and contributions made this portable Linux release possible. We truly appreciate your support! 🙏

🚀 FastFlowLM v0.9.43 - FLM Benchmarking Tool, KV Cache, Chat Templates, and Tool Calling

📊 FLM Benchmarking Tool

You can now use the FLM benchmarking tool to test models and compare performance across context lengths.

Each benchmark runs from 1k to 32k context length for 8 iterations.

flm bench <model_tag>

The benchmark reports:

• TTFT: Time to first token in seconds.
• Prefill speed: Prompt processing speed in tokens per second.
• Decoding speed: Token generation speed in tokens per second.

Results are printed as a table in the shell and saved as a CSV file in the current folder for later reference.

---

⚡ KV Cache Improvements

FLM now uses a safer KV cache flow that preserves the full conversation history before applying the chat template.

🔁 What Changed

Previously, FLM computed a checksum for incoming messages and handled cache state as follows:

• Cache hit: Manually removed the old message from the message list.
• Cache miss: Cleared the context.

This approach caused several issues: message-list manipulation happened before chat templating, reasoning content could remain in the KV cache indefinitely, and the bos token could be prefilled repeatedly on each conversation turn. Together, these issues could lead to fragmented or incorrectly formatted conversation histories.

The updated flow keeps the original message history intact:

• Checksum tracking: FLM still computes a checksum for incoming messages.
• Cache hit: The existing context is retained while the complete original message history is processed.
• Cache miss: The context is cleared as before.
• Consistent templating: The chat template is always applied to the complete, unbroken message history sent by the client.
• Token-level diffing: FLM tokenizes the fully templated prompt, compares the token IDs against the cached token IDs from the previous turn, removes the overlapping cached portion, and keeps only the newly generated token IDs for prefill.

The new implementation uses a checkpoint and restore mechanism to record the current KV cache state and restore it in the next round of conversation.

This keeps reasoning content out of the KV cache, making cached conversations more consistent and reliable, especially for complex prompts. It also helps ensure the final templated prompt remains correctly formatted.

Thanks to @NVolcz for inspiring this improvement and providing feedback on the previous flow!

---

🧩 Chat Template Updates

The Gemma4 chat template has been updated to match the latest official release from Google. To support the improved KV cache logic, we also updated chat templates for several model families so they can safely handle complete message histories.

Updated model families:

• gemma4-it
• qwen3.5
• medgemma
• deepseek-r1

You do not need to redownload all model files manually. Use flm list to check whether your local model files are compatible with the latest chat templates.

There are two ways to update a model's chat template:

1. Run flm check <model_tag> and then flm pull <model_tag> to update the model before your next use.
2. Run flm run <model_tag> or flm serve <model_tag>, and FLM will automatically check and update the chat template if needed.

⚠️ Note: The checking process may take about 20 seconds, depending on the model, but it only needs to run once per model version. After the chat template is updated, you can run or serve the model as usual without the extra delay.

---

🔄 Smarter Force Pulls

flm pull <model_tag> --force now checks existing model files first and only re-downloads missing or incorrect files, instead of removing and re-downloading the entire model.

---

🧰 Gemma4 Tool-Calling Message Formatting

Gemma4 uses a different tool-calling message format from OpenAI-compatible tool-calling APIs. Previously, upstream OpenAI-formatted tool-call messages could be sent directly to Gemma4 without conversion, which could lead to parsing issues.

FLM now detects OpenAI-formatted tool-call messages and converts them to the Gemma4 tool-calling format before sending them to the model. This improves parsing reliability for Gemma4 tool calls.

For more details about the format differences, see the Gemma4 tool-calling documentation.

Thanks to @NVolcz for pointing this out!

---

🛠️ Gemma4 Tool-Calling Reliability

We received a lot of feedback about Gemma4 tool-calling reliability, especially around tool-argument parsing.

Tool arguments are generated by the model as text and then parsed into JSON. Parsing can fail for several reasons, including malformed strings, complex nested arguments, or edge-case formatting.

This release improves Gemma4 tool-call argument parsing across several edge cases, making tool calling more robust overall.

Some malformed outputs may still be impossible to recover automatically, such as tool calls that completely miss the expected chat-template format. We will continue improving coverage for these cases where possible.

We will also continue monitoring feedback and improving tool-calling reliability for Qwen3 and Qwen3.5.

Thanks to the community members who shared their tool-calling experiences and edge cases. @TatuLund, @jtmonroe, @cqh963852, @antrv, and more than a dozen others contributed suggestions that helped improve tool-calling reliability.

---

🔕 Tool Calling Disabled for Low-Reliability Models

Tool-calling support has been disabled for the following models due to low usage and low reliability:

• lfm2.5-tk:1.2b
• nanbeige4.1:3b

---

🌟 Summary

FastFlowLM v0.9.43 adds benchmarking output for easier performance comparison, makes cached conversations more reliable by preserving complete message histories, keeping reasoning content out of the KV cache, and using token-level diffing after chat templating. This release also updates chat templates for Gemma4, Qwen3.5, MedGemma, and DeepSeek-R1, makes flm pull <model_tag> --force more efficient by re-downloading only missing or incorrect files, improves Gemma4 tool-call formatting and argument parsing, and disables tool calling for low-reliability models.

🚀 FastFlowLM v0.9.42 - Tool Calling Reliability Updates

🛠️ Tool Calling Improvements

🔧 Gemma4 JSON Validation

Added a sanity check for Gemma4 tool-call JSON output to catch malformed responses more reliably.

This should help prevent failures caused by invalid JSON in tool calls.

Thanks to @TatuLund for reporting this issue!

🧰 Qwen3.5 Non-Streaming Tool Calls

Fixed incorrect parsing that could cause Qwen3.5 tool-calling support to fail in non-streaming mode.

🧱 Qwen3.5 Tool-Call Robustness

Enhanced Qwen3.5 tool-call parsing to better handle cases where the model may miss closing tool tags.

This improves robustness for tool-calling workflows and reduces failures caused by incomplete tool-call markup.

---

🌟 Summary

FastFlowLM v0.9.42 focuses on more reliable tool calling. This release adds Gemma4 JSON sanity checks and improves Qwen3.5 tool-call behavior in both non-streaming and edge-case parsing scenarios.

🚀 FastFlowLM v0.9.41 - Tool Calling + Usability Updates

🐛 Bug Fixes

🛠️ Tool Calling Cache Redundancy

Fixed an issue where tool schemas could be injected into the KV cache on every cached turn.

This could fill the model context window with repeated copies of the same schema, wasting context tokens, increasing compute time, and making tool-calling behavior less reliable as instructions appeared repeatedly mid-conversation.

Thanks to nvolcz from Discord for reporting this issue!

---

✨ Improvements

📊 Chat Completions Logging

OpenAI-compatible chat completions logging now reports KV cache usage for each conversation round.

Example fields include:

{
    "active_kv_tokens": 4096,
    "max_kv_token_capacity": 32768,
    "kv_token_occupancy_percentage": 12.5
}

kv_token_occupancy_percentage = 4096 / 32768 × 100% = 12.5%

🎧 Gemma4 Audio Logging

Improved Gemma4 audio logging to make long-audio handling easier to understand.

The previous wording could suggest that audio was clipped or partially dropped. The updated message now makes it clear when audio has been split into chunks for processing.

Example:

Audio in message is split into 33 chunks for processing.

Thanks to @gdkrmr for the suggestion.

🐧 Arch Linux Guide

Added an Arch Linux installation guide to the documentation.

For installation instructions, see the Arch Linux guide.

Thanks to @filipenf for the contribution!

🔕 Version Check Control

FastFlowLM can now disable the automatic version check before run and serve modes.

To disable the startup version check, set system environment variable FLM_DISABLE_UPDATE_CHECK to 1:

Linux:

export FLM_DISABLE_UPDATE_CHECK=1

Windows:

setx FLM_DISABLE_UPDATE_CHECK 1

Thanks to @heliosran for the contribution.

🧰 Linux make install

Improved Linux installation behavior by avoiding accidental third-party submodule installs and placing bundled FLM shared libraries under ${CMAKE_INSTALL_LIBDIR}/flm.

Thanks to @J-Bu for the contribution.

---

🌟 Summary

FastFlowLM v0.9.41 fixes redundant tool schema injection in cached conversations and adds clearer observability for KV cache usage and Gemma4 audio processing. This release also improves Linux installation workflows with a new Arch Linux guide, better make install behavior, and an option to disable automatic version checks.

🚀 FastFlowLM v0.9.40 - Gemma4 E4B + Reliability Updates

📦 New Model Support

🌎 Gemma4-IT-E4B

FastFlowLM now supports gemma4-it:e4b for language, vision, audio workloads, including concurrent multimodal input for omni-model use cases.

• Tag: gemma4-it:e4b

Run in CLI mode:

flm run gemma4-it:e4b

Run in server mode:

flm serve gemma4-it:e4b

For more details, see the model card and benchmark results.

---

✨ Improvements

🔥 Performance Boosts for gemma4-it:e2b

This release brings meaningful speed improvements to the gemma4-it:e2b model:

• Prefill: up to 11.4% faster
• Decoding: up to 10.2% faster

⚡ Chunk Prefill

This release adds chunk prefill support, significantly reducing memory usage for long prompts and larger workloads.

You can configure the prefill chunk length with --prefill-chunk-len in both CLI and server modes. The default value is 4096.

Run in CLI mode:

flm run gemma4-it:e4b --prefill-chunk-len 8192

Run in server mode:

flm serve gemma4-it:e4b --prefill-chunk-len 8192

In server mode, you can now cancel a request even while it is still in the prefill stage. No more waiting around for a huge prompt to finish prefill: just hit the stop button in higher-level apps such as Open WebUI and move on.

🔐 Hash Checking

A new hash checking command is now available to help verify downloaded model files.

If you have trouble running a model and suspect a corrupted download, run:

flm check gemma4-it:e4b

If corrupted files are detected, you will see output like this:

[FLM]  Checking model: llama3.2:1b...

[FLM]  Checking file: config.json...
[FLM]  Fail!
[FLM]  Removing corrupted file: config.json...
[FLM]  Successfully removed config.json!
[FLM]  Checking file: model.q4nx...
[FLM]  Success!
[FLM]  Checking file: tokenizer.json...
[FLM]  Success!
[FLM]  Checking file: tokenizer_config.json...
[FLM]  Success!
[FLM]  Model check completed with errors. Please use `flm pull llama3.2:1b` to re-download corrupted files.

---

🐛 Bug Fixes

🛠️ Tool Calling

Fixed an issue where tool calls could return an incorrect finish reason.

Thanks to @antrv for reporting this issue!

⛓️‍💥 Empty Multimodal Input Handling

Fixed an issue where empty image or audio input in server mode could cause the server to break.

Thanks to @antrv for reporting this issue!

🧠 Memory Limits

Fixed a memlock limit issue that could affect loading ASR or embedding models standalone.

Thanks to @sofiageo for reporting this issue!

---

🌟 Summary

FastFlowLM v0.9.40 expands the Gemma4 lineup with gemma4-it:e4b. This release delivers meaningful speed improvements to gemma4-it:e2b, introduces chunk prefill for more efficient handling of long prompts, adds check command for verifying model files, and improves reliability across tool calling, multimodal input, and memory handling.

🚀 FastFlowLM v0.9.39 - Gemma4 + Smoother Workflows

FastFlowLM v0.9.39 adds support for gemma4-it:e2b and smooths out several key workflows across reasoning, embeddings, sampling, and tool calling.

---

📦 New Model Support

👁️ Gemma4-IT-E2B

FastFlowLM now supports gemma4-it:e2b for language, vision, and ASR workloads.

• Tag: gemma4-it:e2b

Run in CLI mode:

flm run gemma4-it:e2b

Run in server mode:

flm serve gemma4-it:e2b

For more details, see the model card and benchmark results.

---

✨ Smoother Experience

🧠 Reasoning Effort Controls

OpenAI-compatible requests now support reasoning_effort for the qwen3 and qwen3.5 model families.

• low, medium, and high enable thinking mode
• none disables thinking mode

The Ollama think field is still supported, but reasoning_effort takes priority when both are provided.

🔢 Embedding API

Thanks to @nitedani for reporting the issue.

Fixed an issue in v1/embedding where multi-batch input could incorrectly return only a single batch in the response.

v1/embedding now correctly supports both multi-batch input and multi-batch output.

⚡ Sampling Optimization

Thanks to @xieyanbo for the optimization contribution.

Sampling performance has been improved in this release for faster generation workflows.

🛠️ Tool Calling

Fixed a Qwen3.5 tool-calling parsing issue where numeric arguments could appear as strings in the JSON schema.

Thanks to @gabrielgiersch-ai for reporting the issue.

Thanks to @J-Bu for the contribution.

---

🌟 Summary

FastFlowLM v0.9.39 expands model support with gemma4-it:e2b and delivers a smoother experience across reasoning, embeddings, sampling, and tool calling.

🚀 FastFlowLM v0.9.38 - New Models!

FastFlowLM v0.9.38 adds qwen3.5:0.8b and nanbeige4.1:3b.

📦 New Models

🖼️ Qwen3.5:0.8B

FastFlowLM now supports a lightweight vision-language model from the Qwen3.5 family:

• Tag: qwen3.5:0.8b

You can control image pre-resizing in both run and serve modes with --img-pre-resize (or -r).

Run in CLI mode:

flm run qwen3.5:0.8b -r 1

Run in server mode:

flm serve qwen3.5:0.8b -r 1

For more details, see the model card and benchmark results.

---

🧠 Nanbeige4.1-3B

FastFlowLM now supports the new Nanbeige4.1-3B model, a reasoning model with tool-calling capabilities.

• Tag: nanbeige4.1:3b

Run in CLI mode:

flm run nanbeige4.1:3b

Run in server mode:

flm serve nanbeige4.1:3b

For more details, see the model card and benchmark results.

---

🐛 Bug Fixes

Some feature labels were missing from the model list JSON file, which could cause issues when integrating with downstream apps that rely on this list.

This issue is fixed in this release.

🌟 Summary

FastFlowLM v0.9.38 expands the Qwen3.5 lineup with Qwen3.5-0.8B and adds Nanbeige4.1-3B for reasoning and tool-calling workflows. Give both models a try and explore the latest performance improvements! ✨

🚀 FastFlowLM v0.9.37 - New Qwen3.5 Models + Better Defaults

FastFlowLM v0.9.37 adds two new Qwen3.5 vision-language models and improves the default model path experience for new Windows users.

---

🖼️ 1. New Qwen3.5 Models

FastFlowLM now supports two additional vision-language models from the Qwen3.5 family:

• Tag: qwen3.5:2b
• Tag: qwen3.5:9b

You can control image pre-resizing in both run and serve modes with --img-pre-resize (or -r).

⚠️ Note: The default pre-resize setting for all Qwen VL models (2.5, 3 and 3.5) is now 720p.

Run in CLI mode:

flm run qwen3.5:2b -r 1

Run in server mode:

flm serve qwen3.5:9b -r 1

For more details, see the model card (2B & 9B) and benchmark results.

---

🪟 2. Better Default Model Path on Windows

For new Windows users, FLM_MODEL_PATH now defaults to C:\Users\<username>\.flm.

Existing users are unaffected: the previous default path, C:\Users\<username>\Documents\flm, is still supported, so current model installations continue to work without changes.

---

🌟 Summary

FastFlowLM v0.9.37 expands the Qwen3.5 lineup with qwen3.5:2b and qwen3.5:9b, while improving the out-of-box experience for new Windows users with a cleaner default model path.