Skip to content

🚀 FastFlowLM v1.0.10 — Constrained Decoding & More Accurate Whisper

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 22 Oct 22:59
v1.0.10
e2d0144

This release does three things:

  • Reasoning, tool calls and structured output are handled one way for every model on /v1/chat/completions.
  • tool_choice and response_format are enforced while the model generates, not just requested.
  • Whisper transcription is more accurate, and /v1/audio/transcriptions now supports language, prompt and response_format.

It also fixes several flm serve bugs, including Lemonade failing to launch FLM on Linux.

✨ New Features

🧩 OpenAI API: Shared Chat Parsing and Constrained Decoding

PR #790

Every model writes its answer in its own markup. Qwen3 uses <think> and <tool_call>, Gemma 4 uses a thought channel and call:f{…}, GPT-OSS uses Harmony channels, and LFM2 writes Python calls. Until now each model had its own parser for this. In v1.0.10, one shared parser handles all of them, so /v1/chat/completions returns the same OpenAI-shaped message whatever the model:

  • reasoning_content, content and tool_calls are split the same way for every model, streamed or not.
  • The streamed deltas add up to exactly the non-streamed reply.
  • Tool-call markup counts as a call only when the request includes tools. A reply that is only tool calls has content: null, as in the OpenAI API.

Constrained decoding. Before each token is sampled, FLM works out which tokens can still lead to valid output and rules out the rest. This uses xgrammar. The model can't misspell a tool name, give an argument the wrong type, leave out a required field, or wrap your JSON in a Markdown fence. You control it with the standard OpenAI request fields:

Field What FLM enforces
tool_choice: "none" No tool call
tool_choice: "auto" Any tool call is well-formed and names a real tool
tool_choice: "required" At least one tool call
tool_choice: {"type": "function", ...} A call to that function
parallel_tool_calls One call or several
response_format: {"type": "json_object"} Valid JSON
response_format: {"type": "json_schema", ...} JSON that matches your schema

Bad values return a clear 400 error instead of being ignored. This covers a malformed tool_choice, an unknown function name, "required" with no tools, and a schema that can't be compiled. Small models that don't see tools in their prompt (Qwen3 up to 1.7B, Qwen3.5 0.8B) also return a 400 for a forced tool call, instead of making up a call to a tool they never saw.

When the model's thinking is off, it can no longer slip its answer into a reasoning block. It answers directly in the requested format.

Coming later: a grammar request field, so you can pass your own grammar for output that isn't JSON (for example, digits only or a fixed set of labels). The constrained-decoding engine in this release already supports this. Only the API field is still to come.

Other API changes:

  • Streaming usage now follows OpenAI. A streamed response carries usage only when the request sets stream_options: {"include_usage": true}. In that case the usage comes in one final chunk before [DONE]. ⚠️ If your client read usage from streamed responses, add include_usage.
  • null request fields are treated as unset. Some OpenAI clients send null for every option they leave out. Before, this could return a 400 or 500 error, for example for parallel_tool_calls: null or tool_choice: null.

New docs:

🎙️ Audio Transcription: /v1/audio/transcriptions

PR #794, by community contributor @quine00

The endpoint used to read only model and file. It now honors the other OpenAI parameters:

  • language (ISO 639-1, for example en, zh). Use it to skip language detection. An unknown code returns a 400 instead of falling back silently.
  • prompt. Prior text that guides spelling and style, such as names and terms. It is applied the same way as in openai-whisper.
  • response_format. json (the default, unchanged), text, srt, vtt and verbose_json. verbose_json returns segments with timestamps, the detected language and the clip duration. text, srt and vtt are returned as text/plain.

Privacy: transcripts are no longer printed to the server log by default. Under a service manager, dictated speech used to end up in the system journal. To log them again, set FLM_LOG_RAW_OUTPUT.

🐛 Bug Fixes

🎯 Whisper Transcribes More Accurately

PR #794, by community contributor @quine00

The Whisper decode loop now behaves like OpenAI's reference Whisper:

  • Long sentences are no longer cut mid-word. FLM forced a timestamp after every 16 text tokens, which often ended the segment early. On 82 English clips, this fix alone took the word error rate from 7.2% to 3.1%.
  • The language token is fed to the decoder, as Whisper expects. Before, it was detected but never passed back.
  • Whisper's timestamp rules are applied, and <|30.00|> is now treated as a timestamp. Segments no longer start late or get text after a closing timestamp.
  • Audio over 30 seconds no longer repeats text at window boundaries, or repeats the same timestamp several times in a row.
  • The Whisper engine and kernels were rebuilt for accuracy. Padding is now masked, rounding is fixed, and the exact GELU is used. On the ASR test set, a prompt that repeats words from the audio no longer drops the start of the transcript. The model weights are unchanged.

🖥️ Server Fixes

PR #783, PR #820, PR #821, PR #823

  • Lemonade can launch FLM on Linux again. In v1.0.8 and v1.0.9, an NPU that couldn't be opened produced [ERROR] No NPU device available and [ERROR] NPU device instance is nullptr, instead of the clean Error: mmap(...) failed message. This broke Lemonade's launch of FLM. The v1.0.7 behavior is restored for flm run, flm serve and flm bench.
  • Streams no longer hang on cut-off UTF-8. If a reply hit max_tokens or was cancelled partway through a multi-byte character (CJK text, emoji), the stream never ended and the client waited until its own timeout. Now the incomplete character is dropped. An error after streaming has started is sent as an error event followed by [DONE].
  • Switching models in one flm serve process is reliable.
    • A failed switch no longer leaves a half-loaded model behind.
    • The prefill chunk size is set for each model, not taken from the first one loaded.
    • Aliases no longer reload the model on every request.
    • Multi-turn conversations no longer restore a stale prompt cache after a reload.

🙌 Community Contributions

Thank you to everyone who contributed to this release:

  • @quine00 reworked the Whisper decode loop and added language, prompt and response_format to /v1/audio/transcriptions (PR #794).

Several community PRs found these same problems and proposed fixes before this release. Their changes weren't merged as-is, because the code they touched was rewritten in v1.0.10. Their reports and fixes shaped what shipped:

  • @Anjielon found Whisper's forced timestamp after 16 tokens cutting transcriptions short (PR #699).
  • @ademasi proposed verbose_json and forwarding the multipart fields on /v1/audio/transcriptions (PR #510).
  • @hvico proposed grammar-constrained sampling with GBNF grammars, the first constrained decoding for FLM (PR #487). v1.0.10 builds constrained decoding on xgrammar instead, and a grammar request field like the one in this PR is planned.
  • @waw2637 proposed enforcing the OpenAI tool_choice contract and stream_options.include_usage (PR #650).
  • @Javinator9889 found the think block leaking into content in non-streaming responses (PR #740).
  • @AshtonVaughan found tool-call arguments losing their closing brace (PR #531).

🌟 Summary

Highlight
🧩 One parser for reasoning and tool calls across all models on /v1/chat/completions
🔒 tool_choice and response_format (JSON / JSON Schema) enforced by constrained decoding
⚠️ Streamed usage now needs stream_options.include_usage
🎙️ /v1/audio/transcriptions supports language, prompt and response_format (text, srt, vtt, verbose_json)
🎯 More accurate Whisper: no mid-sentence cutoffs, correct timestamps, no repeated text on long audio
🔐 Transcripts no longer written to the server log by default
🐧 Lemonade can launch FLM on Linux again
🖥️ No more hung streams on cut-off UTF-8, and reliable model switching in flm serve

Thanks for your support — see you in the next one! 🚀

Assets 11
Loading