Whisper: hidden_size > 0 assertion in lm_config.hpp:121 crashes flm serve --asr 1 and flm list after Whisper pull

[istitov]

Summary

On Linux FLM 0.9.41, every documented invocation that loads whisper-v3:turbo crashes with an unconditional assert(this->hidden_size > 0) in LM_Config::from_pretrained. Whisper's config.json does not (and should not) have a hidden_size field — it has d_model, encoder_layers, etc. — so the assertion fires whenever this code path is reached for a Whisper config. The assertion is still present in trunk (src/include/lm_config.hpp:121), so this affects current main too.

Additional side effect: once flm pull whisper-v3:turbo succeeds, every subsequent flm list invocation crashes at exit because the listing code iterates all installed config.json files and hits the assertion on Whisper.

Reproducer (all three paths produce identical crash)

$ flm pull whisper-v3:turbo                # succeeds; downloads 4 files (~622 MB) into ~/.config/flm/models/Whisper-V3-Turbo-NPU2/

$ flm serve --asr 1 --port 11434           # docs: "standalone ASR model" mode
[FLM]  Using custom model list path: /opt/fastflowlm/share/flm/model_list.json
[FLM]  ASR mode enabled: reserving additional 1GB of memory
[FLM]  Using user-specified port: 11434
flm: src/include/lm_config.hpp:121:
void LM_Config::from_pretrained(std::string): Assertion `this->hidden_size > 0' failed.
Aborted (core dumped)

$ flm serve llama3.2:3b --asr 1 --port 11434  # docs: sidecar pattern
[same assertion crash, same line]

$ flm list                                  # iterates installed configs at exit
[FLM]  Using custom model list path: /opt/fastflowlm/share/flm/model_list.json
Models:
  - deepseek-r1-0528:8b ✅
  - llama3.2:1b ✅
  - llama3.2:3b ✅
  - qwen3:0.6b ✅
  - qwen3:4b ✅
  - qwen3:8b ✅
  - qwen3-it:4b ✅
  - qwen3.5:4b ✅
  - qwen3.5:9b ✅
flm: lm_config.hpp:121: void LM_Config::from_pretrained(std::string): Assertion `this->hidden_size > 0' failed.
Aborted

The flm list output is complete before the abort (so it's user-visible), but the non-zero exit code breaks any scripting that checks flm list's return status.

Expected behavior

flm serve --asr 1 should start the Whisper ASR server cleanly (per the docs "standalone ASR model" example). flm list should not crash regardless of whether Whisper is installed.

Mechanism

whisper-v3:turbo's config.json (downloaded by flm pull) has:

{
  "_name_or_path": "/raid/yoach/tmp_whisper_turbo",
  "architectures": ["WhisperForConditionalGeneration"],
  "model_type": "whisper",
  "d_model": 1280,
  "encoder_layers": 32,
  "encoder_attention_heads": 20,
  "decoder_layers": 4,
  ...
  "flm_version": "0.9.14"
}

No hidden_size field — Whisper's encoder/decoder transformer uses d_model for its width, not the chat-LLM hidden_size terminology.

LM_Config::from_pretrained in src/include/lm_config.hpp (line ~121 in 0.9.41, still present in current main) unconditionally asserts:

assert(this->vocab_size > 0);
assert(this->hidden_size > 0);          // ← crashes on Whisper
assert(this->intermediate_size > 0);
assert(this->num_attention_heads > 0);
assert(this->num_hidden_layers > 0);
assert(this->num_key_value_heads > 0);

The same class has fields for audio_model_weight and audio_config, which suggests the design intent is "audio support is embedded inside the chat model's config.json" — but the actual --asr 1 codepath ends up loading Whisper's own config and tripping the assertion regardless.

Regression bisection: introduced in v0.9.39

The assert(this->hidden_size > 0) itself isn't new — it's been in
lm_config.hpp since at least v0.9.14 (verified by fetching the file
at v0.9.14, v0.9.22, v0.9.30, v0.9.36, v0.9.37, v0.9.38, v0.9.39, v0.9.40,
v0.9.41 — all contain the assertion). So Whisper used to take a code
path that didn't go through LM_Config::from_pretrained.

What changed: commit e2ccfd5d ("feat: update config for audio", 2026-04-07) added audio-config handling to LM_Config itself:

+   std::string audio_model_weight;
+   nlohmann::json _audio_config;
+   bool is_audio;

…and added the corresponding JSON loads inside the from_pretrained
body, right next to the existing hidden_size > 0 assertion.

This landed in v0.9.39 (released 2026-04-15). v0.9.38 (2026-04-02)
predates it.

Cross-checks:

• Issue [Feature Request] Support custom ASR model path / --asr-model flag #441 (closed 2026-03-27) confirms Whisper-via-FLM worked then
  on the same Ryzen AI 9 HX 370 chip — that user was likely on
  v0.9.36 or v0.9.37, both predating the regression.
• FLM's own benchmark page fastflowlm.com/docs/benchmarks/qwen3.5_results/
  cites measurements taken on v0.9.38 — i.e. the last known-working
  release for the audio loader.

Regression range: introduced between v0.9.38 (works) and v0.9.39 (broken).

Likely mechanism: before v0.9.39, the --asr 1 codepath either
bypassed LM_Config::from_pretrained for Whisper, or chained the
chat-LLM config as the "main" config (which has hidden_size). The
v0.9.39 refactor routed both standalone (flm serve --asr 1) and
sidecar (flm serve <chat> --asr 1) Whisper loads through
LM_Config::from_pretrained, where the unconditional hidden_size > 0
assert has always been waiting.

Suggested fix

from_pretrained should branch on model_type: when model_type == "whisper", validate Whisper-specific fields (d_model, encoder_layers, encoder_attention_heads, etc.) instead of the chat-LLM ones. Or the chat-LLM assertions should be gated to apply only when the config represents a chat LLM.

A minimal patch that avoids breaking existing behavior: relax the asserts to early-return-with-error when model_type == "whisper", deferring to the dedicated Whisper loader path (whichever currently exists for the ASR/-a flag).

Confirmed not a Linux-packaging artifact

This was triaged on Gentoo Linux. ldd /opt/fastflowlm/bin/flm confirms the binary IS linked against the full audio stack at runtime:

libavformat.so.62, libavcodec.so.62, libswresample.so.6, libswscale.so.9,
libavutil.so.60, libfftw3f.so.3, libboost_program_options.so.1.90.0,
libreadline.so.8, libcurl.so.4, and /opt/fastflowlm/lib64/flm/libwhisper_npu.so

So the Whisper compute path is compiled in; the crash is in FLM's own config-parsing code, not in missing native deps.

Environment

• Hardware: Framework 13, AMD Ryzen AI 9 HX 370 (Strix Point), XDNA 2 NPU,
• NPU FW: 1.1.2.64
• amdxdna: 1.0 (kernel 7.0.2)
• OS: Gentoo Linux
• FLM: 0.9.41 (built from upstream tag via Gentoo sci-ml/fastflowlm ebuild)
• XRT / xrt-xdna: 2.21.75

flm validate output:

[FLM]  Using custom model list path: /opt/fastflowlm/share/flm/model_list.json
[Linux]  Kernel: 7.0.2
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 1.0
[Linux]  Memlock Limit: infinity

Cross-references

• Docs page that documents the --asr 1 standalone pattern: fastflowlm.com/docs/models/whisper/
• Model registry entry: models.whisper-v3.turbo in src/model_list.json (flm_min_version: "0.9.14")
• Trunk source: src/include/lm_config.hpp line ~121 (assertion is present)

[ZaneNi]

@istitov Sorry, I couldn’t reproduce this error.

The LM_Config in src/include/lm_config.hpp is only instantiated for language models such as gemma4-it:e2b, and every language model has a hidden_size field in config.json, so the assertion works for all language models.

However, Whisper does not use LM_Config. It uses a special Whisper_Config, which does not include an assertion for the hidden_size field. Reference:

FastFlowLM/src/include/whisper/modeling_whisper.hpp

Line 136 in c7a9807

std::unique_ptr<Whisper_Config> lm_config = nullptr;

FastFlowLM/src/include/lm_config.hpp

Line 159 in c7a9807

class Whisper_Config : public LM_Config{

Honestly, I am not sure why Whisper would trigger the hidden_size assertion.

May I ask whether you built from source and accidentally modified the code? Could you try pulling the latest code or installing the latest released version and testing again?

[istitov]

I hit this on 0.9.43 as well. Suspicion is the asymmetry between how we build and your default: we build RelWithDebInfo with no -DNDEBUG (Gentoo's cmake.eclass wipes the standard release-flag defaults so user CXXFLAGS take over, and ours don't carry NDEBUG), while the linux-default preset here is Release, which CMake silently adds -O3 -DNDEBUG to. With NDEBUG the assert(hidden_size > 0) at lm_config.hpp:121 is a no-op - so the crash never surfaces on your end, even though the buggy code path runs.

For 0.9.43 specifically: the Whisper_Config : LM_Config subclass at lm_config.hpp:159 looks like the right shape already - reads hidden_size from d_model, skips the chat-LLM asserts. But flm list and flm serve --asr 1 still walk the base LM_Config::from_pretrained for every model dir they scan, so the subclass never gets dispatched on Whisper configs. Routing by model_type == "whisper" at the scan site (or making the base method virtual + overriding it in Whisper_Config) should close it.

Minimal repro on a build with asserts live:

      flm pull whisper-v3:turbo
      flm list
      # → Assertion `this->hidden_size > 0' failed.

No patches, no config edits — pristine pull is enough.

P.S. otherwise I'd like to say kudos for you flm -- it is really cool thing!

[ZaneNi]

I hit this on 0.9.43 as well. Suspicion is the asymmetry between how we build and your default: we build RelWithDebInfo with no -DNDEBUG (Gentoo's cmake.eclass wipes the standard release-flag defaults so user CXXFLAGS take over, and ours don't carry NDEBUG), while the linux-default preset here is Release, which CMake silently adds -O3 -DNDEBUG to. With NDEBUG the assert(hidden_size > 0) at lm_config.hpp:121 is a no-op - so the crash never surfaces on your end, even though the buggy code path runs.

For 0.9.43 specifically: the Whisper_Config : LM_Config subclass at lm_config.hpp:159 looks like the right shape already - reads hidden_size from d_model, skips the chat-LLM asserts. But flm list and flm serve --asr 1 still walk the base LM_Config::from_pretrained for every model dir they scan, so the subclass never gets dispatched on Whisper configs. Routing by model_type == "whisper" at the scan site (or making the base method virtual + overriding it in Whisper_Config) should close it.

Minimal repro on a build with asserts live:

      flm pull whisper-v3:turbo
      flm list
      # → Assertion `this->hidden_size > 0' failed.

No patches, no config edits — pristine pull is enough.

P.S. otherwise I'd like to say kudos for you flm -- it is really cool thing!

@istitov Thanks for your kind words!

Your explanation makes sense to me, but I really cannot reproduce this issue on my side.

flm -v
FLM v0.9.41

flm pull whisper-v3:turbo
[FLM]  Model: whisper-v3:turbo
[FLM]  Name: Whisper-V3-Turbo-NPU2
[FLM]  Missing files (4):
  - config.json
  - tokenizer.json
  - tokenizer_config.json
  - model.q4nx
[FLM]  Downloading 4 missing files...
[FLM]  Files to download (622.91 MB): 
  - config.json (0.00 MB)
  - tokenizer.json (2.58 MB)
  - tokenizer_config.json (0.27 MB)
  - model.q4nx (620.06 MB)
[FLM]  Downloading 1/4: config.json
[FLM]  Checking Hash...
[FLM]  Download completed: /home/zane/.config/flm/models/Whisper-V3-Turbo-NPU2/config.json
[FLM]  Overall progress:  1/4 files
[FLM]  Downloading 2/4: tokenizer.json
[FLM]  Checking Hash...
[FLM]  Download completed: /home/zane/.config/flm/models/Whisper-V3-Turbo-NPU2/tokenizer.json
[FLM]  Overall progress:  2/4 files
[FLM]  Downloading 3/4: tokenizer_config.json
[FLM]  Checking Hash...
[FLM]  Download completed: /home/zane/.config/flm/models/Whisper-V3-Turbo-NPU2/tokenizer_config.json
[FLM]  Overall progress:  3/4 files
[FLM]  Downloading 4/4: model.q4nx
[FLM]  Downloading: 84.2% (521.9MB / 620.1MB)
[FLM]  Checking Hash...
[FLM]  Download completed: /home/zane/.config/flm/models/Whisper-V3-Turbo-NPU2/model.q4nx
[FLM]  Overall progress:  4/4 files
[FLM]  All downloads completed successfully!
[FLM]  Model downloaded successfully!
[FLM]  All files verified successfully.

flm list
Models:
  - deepseek-r1:8b ⏬
  - deepseek-r1-0528:8b ⏬
  - embed-gemma:300m ⏬
  - gemma3:1b ⏬
  - gemma3:4b ⏬
  - gemma4-it:e2b ⏬
  - gemma4-it:e4b ⏬
  - gpt-oss:20b ⏬
  - gpt-oss-sg:20b ⏬
  - lfm2:1.2b ⏬
  - lfm2:2.6b ⏬
  - lfm2-trans:2.6b ⏬
  - lfm2.5-it:1.2b ⏬
  - lfm2.5-tk:1.2b ⏬
  - llama3.1:8b ⏬
  - llama3.2:1b ⏬
  - llama3.2:3b ⏬
  - medgemma:4b ⏬
  - medgemma1.5:4b ⏬
  - nanbeige4.1:3b ⏬
  - phi4-mini-it:4b ⏬
  - qwen2.5-it:3b ⏬
  - qwen2.5vl-it:3b ⏬
  - qwen3:0.6b ⏬
  - qwen3:1.7b ⏬
  - qwen3:4b ⏬
  - qwen3:8b ⏬
  - qwen3-it:4b ⏬
  - qwen3-tk:4b ⏬
  - qwen3.5:0.8b ⏬
  - qwen3.5:2b ⏬
  - qwen3.5:4b ⏬
  - qwen3.5:9b ⏬
  - qwen3vl-it:4b ⏬
  - translategemma:4b ⏬
  - whisper-v3:turbo ✅

Would you mind submitting a PR from your side? We would really appreciate your contribution!

[istitov]

created a PR, rechecked the -DNDEBUG hypothesis, and it is real.