🚀 FastFlowLM v1.0.6 — Flash Models, tool_choice, and Tool-Call Fixes

This release introduces three flash models — kernel-optimized runtimes over the existing weights, so no re-download is needed — makes hy-mt2:1.8b single-turn, adds tool_choice support in server mode, fixes a tool-call finish_reason bug, and changes where the MSI installer writes FLM_MODEL_PATH.

---

⚡ New Flash Models

FastFlowLM now supports three flash models. These are not new checkpoints — they run the same weights as their standard counterparts through optimized kernels, so there is no weights update and nothing new to download:

Tag	Prefill @ 128 ctx	Model card
gemma4e-flash:e2b	~390 tokens/s	Gemma 4 E2B-IT · Flash
gemma4e-flash:e4b	~256 tokens/s	Gemma 4 E4B-IT · Flash
qwen3vl-flash:4b	~350 tokens/s	Qwen3-VL 4B-Instruct · Flash

Run in CLI mode:

flm run gemma4e-flash:e2b

Run in server mode:

flm serve qwen3vl-flash:4b

📖 What "Flash" Means

Flash models trade multi-turn flexibility for speed: the same weights run through optimized kernels, under a set of constraints that make those kernels possible. Please note the following behavior before deploying them:

• Same weights, no re-download. Flash models reuse the weights you already have — no flm pull required.
• Single-turn only. Any request containing an assistant message is rejected. Send a system prompt (optional) and a single user message.
• System KV cache supported. The system prompt is prefilled once and reused across requests, so repeated calls sharing a system prompt skip that prefill cost.
• 1k maximum context length. Requests over 1k tokens are rejected with a warning — they are not silently truncated. Size your prompts accordingly.

🖼️ Image & Audio Handling

Flash models apply a fixed media budget. No per-request tuning is required — oversized input is reduced automatically rather than rejected:

Model	Images	Audio
qwen3vl-flash:4b	Resized so the longer side is 256 pixels	Not supported
gemma4e-flash:e2b / :e4b	Resized to 70 tokens	Truncated to the first 30 seconds

⚠️ Note the difference from the context limit: oversized media is silently reduced (images downscaled, audio cut at 30 s), while an over-1k prompt is rejected outright. Audio longer than 30 seconds will not raise an error — the model simply never sees the remainder, so split long clips yourself if you need full coverage.

🐍 Example: Python + OpenAI SDK (Streaming)

Flash models speak the standard OpenAI chat-completions API, so the official openai Python client works as-is — just point it at your local FLM server.

pip install openai
flm serve gemma4e-flash:e2b

Text, streaming, with a pinned system prompt:

Flash models are single-turn, so each call is independent — do not append the reply to messages and send it back. Keep the system prompt fixed and swap only the user message:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:52625/v1",
    api_key="dummykey",  # FLM runs locally; the key is not checked
)

# Keep everything fixed in the system prompt — it is prefilled once and
# reused across requests. Vary only the user message.
SYSTEM_PROMPT = "You are a concise assistant. Answer in one short sentence."

for question in ["Why is the sky blue?", "Why is grass green?"]:
    print(f"\n> {question}")
    stream = client.chat.completions.create(
        model="gemma4e-flash:e2b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},   # same every time
            {"role": "user", "content": question},          # only this changes
        ],
        stream=True,
    )
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)
    print()

Vision, streaming:

Serve the vision flash model with flm serve qwen3vl-flash:4b, then pass an image as a base64 data URL. Resizing is automatic, so there is no need to downscale beforehand:

import base64
from openai import OpenAI

image_path = r"C:\path\to\image.png"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(image_path, "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="qwen3vl-flash:4b",
    messages=[
        {"role": "system", "content": "Describe images in one sentence."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is in this image?"},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image}"},
                },
            ],
        },
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()

Note: the image counts against the same 1k context budget as your text. Keep prompts short when sending an image.

Audio, streaming:

Audio goes to the same endpoint with the same message structure — only the content part changes. Use a gemma4e-flash model, since qwen3vl-flash:4b does not accept audio:

import base64
from openai import OpenAI

audio_path = r"C:\path\to\audio.wav"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(audio_path, "rb") as audio_file:
    audio = base64.b64encode(audio_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="gemma4e-flash:e2b",
    messages=[
        {"role": "system", "content": "Transcribe and summarize audio briefly."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is said in this clip?"},
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio},
                },
            ],
        },
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()

⚠️ Only the first 30 seconds are processed. Longer clips are truncated without an error, so split them client-side if you need full coverage.

Audio + image in one request:

gemma4e-flash accepts both in the same content array, so a single call can reason over audio and an image together:

import base64
from openai import OpenAI

audio_path = r"C:\path\to\audio.wav"   # <-- edit this
image_path = r"C:\path\to\image.png"   # <-- edit this

client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="dummykey")

with open(audio_path, "rb") as audio_file:
    audio = base64.b64encode(audio_file.read()).decode("utf-8")
with open(image_path, "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")

stream = client.chat.completions.create(
    model="gemma4e-flash:e2b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Finish two tasks: 1. Summarize the audio. 2. Describe the image."},
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image}"},
                },
            ],
        }
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()

---

🔄 hy-mt2:1.8b Is Now Single-Turn

hy-mt2:1.8b, the multilingual translation model added in v1.0.5, is now a single-turn model as well. Like the flash models, it rejects any request containing an assistant message.

This matches how a dedicated translation model is actually used: every translation is independent, and prior turns add prefill cost without improving the result.

What this means for you: if you were replaying conversation history back to hy-mt2:1.8b, remove it — send only the instruction and the source text. The recommended shape from the v1.0.5 prompt guide is unchanged and remains the best option:

{"role": "system", "content": "将以下文本翻译为英语，注意只需要输出翻译后的结果，不要额外解释。输出必须全部使用英语，不要输出源语言或原文"},
{"role": "user", "content": "{TEXT}"}

Pinning the instruction as the system prompt and sending only the source text as the user message keeps the instruction out of every request's prefill — especially worthwhile for batch workloads such as translating subtitle lines.

---

🛠️ Tool Calling

tool_choice Support (Server Mode)

Server mode now honors the tool_choice field. Two modes are supported:

Value	Behavior
auto (default)	The model decides whether to call a tool
none	Tools are still sent to the model, but tool-token logits are masked during decoding, so no tool call can be emitted

Any other value falls back to auto.

🐛 Bug Fix: Truncated Tool Calls Reported as tool_calls

Fixed a bug where a tool call truncated by max_tokens was reported with finish_reason: "tool_calls" alongside a tool call whose name and arguments were empty strings. Truncated generations now correctly report finish_reason: "length".

---

📦 MSI Installer

The installer now sets FLM_MODEL_PATH as a user environment variable instead of a system one.

🌟 Summary

	Highlight
⚡	Three new flash models: gemma4e-flash:e2b, gemma4e-flash:e4b, qwen3vl-flash:4b — kernel-optimized over the same weights (no re-download), single-turn only, 1k context, system KV cache
🔄	hy-mt2:1.8b is now single-turn — assistant messages are rejected; send instruction + source text only
🛠️	tool_choice support in server mode: auto (default) and none
🐛	Truncated tool calls now report finish_reason: "length" instead of an empty tool_calls
📦	MSI installer sets FLM_MODEL_PATH in the user environment instead of the system environment

Thanks for your support — see you in the next one! 🚀