Qwen3-Coder-Next randomly looping gibberish

[nawoa]

I'm not an expert in this sort of thing, maybe you could test further, but I had severe issues with repetition using the default KV cache quantization in the provided Qwen3-Coder-Next recipe when using it to debug a fairly long Arduino sketch. Switching to "auto" (BF16) the issue was appeared to be immediately resolved. This is false, it was just luck. I've read suggestions that Qwen3.5 models may have a similar issue but I haven't actually tested those myself yet. (Irrelevant) A sample of the sort of looping I'm describing:

However, the following improvements are recommended:

**Add a way to reset emergencyShutdown (e.g., button press or timer).
**Add logging of emergency events (e.g., serial log) to help debug.
**Add a way to test the flame sensor (e.g., button press to simulate flame).
**Add a way to test the fan sensor (e.g., button press to simulate fan spin).
**Add a way to test the thermistor (e.g., button press to simulate temp change).
**Add a way to test the relay (e.g., button press to toggle relay).
**Add a way to test the LCD (e.g., button press to toggle LCD).
**Add a way to test the encoder (e.g., button press to simulate encoder move).
**Add a way to test the EEPROM (e.g., button press to read/write EEPROM).
**Add a way to test the ADC (e.g., button press to read ADC).
**Add a way to test the PWM (e.g., button press to set PWM).
**Add a way to test the I2C (e.g., button press to read/write I2C).
**Add a way to test the SPI (e.g., button press to read/write SPI).
**Add a way to test the UART (e.g., button press to read/write UART).
**Add a way to test the USB (e.g., button press to read/write USB).
**Add a way to test the Bluetooth (e.g., button press to read/write Bluetooth).
**Add a way to test the Wi-Fi (e.g, button press to read/write Wi-Fi).
**Add a way to test the GPS (e.g., button press to read/write GPS).
**Add a way to test the LoRa (e.g., button press to read/write LoRa).
**Add a way to test the Zigbee (e.g, button press to read/write Zigbee).
**Add a way to test the Thread (e.g, button press to read/write Thread).
**Add a way to test the Matter (e.g, button press to read/write Matter).
**Add a way to test the Bluetooth LE (e.g, button press to read/write Bluetooth LE).
**Add a way to test the Thread (e.g, button press to read/write Thread).
**Add a way to test the Zigbee (e.g, button press to read/write Zigbee).
**Add a way to test the LoRa (e.g, button press to read/write LoRa).
**Add a way to test the Wi-Fi (e.g, button press to read/write Wi-Fi).
**Add a way to test the GPS (e.g, button press to read/write GPS).
**Add a way to test the Bluetooth (e.g, button press to read/write Bluetooth).
**Add a way to test the USB (e.g, button press to read/write USB).
**Add a way to test the UART (e.g, button press to read/write UART).
**Add a way to test the SPI (e.g, button press to read/write SPI).
**Add a way to test the I2C (e, button press to read/write I2C).
**Add a way to test the LCD (e, button press to toggle LCD).
**Add a way to test the encoder (e, button press to simulate encoder move).
**Add a way to test the fan (e, button press to simulate fan spin).
**Add a way to test the flame (e, button press to simulate flame).
**Add a way to test the thermistor (e, button press to simulate temp change).
**Add a way to test the relay (e, button press to toggle relay).
**Add a way to test the ADC (e, button press to read ADC).
**Add a way to test the PWM (e, button press to set PWM).
**Add a way to test the EEPROM (e, button press to read/write EEPROM).
**Add a way to test the Wi-Fi (e, button press to read/write Wi-Fi).
**Add a way to test the Bluetooth (e, button press to read/write Bluetooth).
**Add a way to test the GPS (e, button press to read/write GPS).
**Add a way to test the LoRa (e, button press to read/write LoRa).
**Add a way to test the Zigbee (e, button press to read/write Zigbee).
**Add a way to test the Thread (e, button press to read/write Thread).
**Add a way to test the Matter (e, button press to read/write Matter).
**Add a way to test the Bluetooth LE (e, button press to read/write Bluetooth LE).
**Add a way to test the USB (e, button press to read/write USB).
**Add a way to test the UART (e, button press to read/write UART).
**Add a way to test the I2C (e, button press to read/write I2C).
**Add a way to test the SPI (e, button press to read/write SPI).
**Add a way to test the Wi-Fi (e, button press to read/write Wi-Fi).
**Add a way to test the GPS (e, button press to read/write GPS).
**Add a way to test the Bluetooth (e, button press to read/write Bluetooth).
**Add a way to test the LoRa (e, button press to read/write LoRa).
**Add a way to test the Zigbee (e, button press to read/write Zigbee).
**Add a way to test the Thread (e, button press to read/write Thread).
**Add a way to test the Matter (e, button press to read/write Matter).
**Add a way to test the Bluetooth LE (e, button press to read/write Bluetooth

My full launch parameters (2 stacked Asus Ascent):

./run-recipe.sh qwen3-coder-next-fp8
--no-ray
--gpu-memory-utilization 0.88
--max-num-batched-tokens 65536
--max-num-seqs 16
--enable-chunked-prefill
--attention-backend flashinfer
--max-model-len 262144
--kv-cache-dtype auto
--override-generation-config '{"temperature": 1.0, "top_p": 0.95, "top_k": 40, "min_p":0.01}'

Only change between it working consistently and failing consistently on the same prompt was the kv-cache. This turned out to be merely a string of good luck. The issue is present regardless of kv-cache quantization so this seems to actually be a deeper issue with the backend. A Windows workstation with the same prompt and parameters under Llama.cpp with Q5_K_XL GGUF functioned perfectly 50 runs out of 50. No looping, no gibberish.

The issue might only appear with fairly long prompts; I'm not comfortable sharing the specific Arduino sketch (it's dangerous in its present condition) but it's 540 lines long and about 20KB. It also does not in any way include most of the functions described in the loop above (Matter, Thread, Zigbee, Wifi, Bluetooth, GPS, etc), so this is not merely repetition but also hallucination.

A very simple prompt may not exhibit the same behavior, I can't say; both my Sparks and my workstation are going to be occupied for the next several days so I can't do any further testing immediately.

[eugr]

OK, thanks, looks like it's very sensitive to KV quantization. Too bad, because that would limit the context on a single Spark.
I'll adjust the recipe.

[nawoa]

Honestly it may have been a false postive. I got many bad responses in a row with it at FP8 but this morning it's sometimes been bad even with BF16. Trying more troubleshooting on my end, I might have more to say tomorrow. Is it concerning that the vLLM startup complains about it missing a MoE whatsit?

[SeanThomasWilliams]

I ran into the same fp8 KV coherence failure pattern on our GB10 nodes with Qwen3.5-35B-A3B, so I wanted to share what ended up mattering for me.

The failure mode seems to be a stack of upstream issues depending on exact config/hardware: --calculate-kv-scales on hybrid models can poison scales (vllm#37554, fixed/guarded by vllm#37565), and there was also a Qwen3-MoE FP8 KV scale loading bug that silently fell back to 1.0 (vllm#35656, included in 0.18.x). Since you are on an FP8 weight model, I would also watch the Blackwell DeepGEMM path (vllm#37618, vllm#37804) - that is more likely to bite you than my INT4 setup. I had better stability after moving to 0.18.x, avoiding --calculate-kv-scales, and forcing VLLM_USE_DEEP_GEMM=0 during validation. On my side, fp8 KV has been coherent on v0.18.x with no calc-scales on the INT4 model.

[nawoa]

Yes I'm pretty confident now that my instant-fix of switching to BF16 was less a fix and more a series of lucky coherent responses. Since then I've found regardless of KV cache it can often begin to break down as early as 500-1000 tokens into a response. I apologize for the false report.

I will experiment more as I have time; maybe this report can be repurposed to troubleshooting Qwen3-Coder-Next in general, less so the KV cache quantization specifically being at fault. If this can be made to work properly the performance is truly impressive, over 60 tokens per second and the "good" responses I do get are very good.

[eugr]

I'll do some coding later today and will test this too with the most recent build.

[nawoa]

I decided to instead try the full FP16 version since there's plenty enough RAM on a pair of stacked sparks for 10x concurrency with 256K context length. I gave it the same test, a long Arduino sketch I asked it to check for bugs. It still runs at a very respectable ~42 tps on single-user even at full size, which is encouraging.

It generated a good response.... and then eventually switched to similar gibberish as with FP8. I hit stop and regenerate, and based on the vLLM window in the background I could tell it was still generating the original response at the same time. This time it almost immediately began generating garbage. Hit stop and regenerate again and it gave a perfectly good coherent full-length response. Regenerated again, and it began listing off hundreds of GPIO pins that don't exist and never stopped. Then it made 5 perfect responses in a row. Then I restarted it and the very first response got stuck in a loop. I'm not sure what to think... maybe my expectations of this model are just wrong?

My startup command:
cd ~/spark-vllm-docker && ./launch-cluster.sh --no-ray exec
vllm serve Qwen/Qwen3-Coder-Next
--tensor-parallel-size 2
--gpu-memory-utilization 0.885
--max-model-len 262144
--max-num-seqs 10
--enable-chunked-prefill
--attention-backend flashinfer
--load-format fastsafetensors
--enable-auto-tool-choice
--tool-call-parser qwen3_coder

(edit: also I'm using the recommended parameters)

I tried the same prompt on Qwen/Qwen3.5-122B-A10B-FP8 (with thinking disabled) and it was able to produce consistently good results ten times out of ten, zero looping, excellent reasoning, actionable results.

I wonder if this might tell you anything: as I was monitoring the vLLM command window, I noticed the Qwen3.5 regenerate tests had 80.3% prefix hit rate while the identical prompt with Qwen3-Coder-Next-FP8 (and BF16 kv-cache) had just 46.4% prefix hit rate.

Maybe it would be worth checking if other Qwen3-Next models behave the way Coder does.

[eugr]

You need to add --enable-prefix-caching for the prefix caching to work. I will get to testing this model later - busy with some other stuff right now, but I used it a couple of weeks ago and haven't experienced any of these issues, for both fp8 and int4-autoround

[nawoa]

Prefix caching is enabled as part of the recipe for both models.

Thanks for your efforts.

[dmitrym2025]

I'm not sure if this is connected, but there is an issue with accuracy in flashinfer on fp8 MoE specifically for qwen models:
vllm-project/vllm#31394

[nawoa]

I'm not sure if this is connected, but there is an issue with accuracy in flashinfer on fp8 MoE specifically for qwen models: vllm-project/vllm#31394

Any chance something similar also applies to BF16 in Qwen3-Coder-Next (and possibly other Qwen3 models)? I had similar issues with both. I have yet to retry INT4-Autoround to see if it's affected, maybe today if I get the chance.

[dmitrym2025]

BF16

based on flashinfer-ai/flashinfer#2803 can be, but have not seen such reports for bf16, as it not as popular as fp8/fp4

[nawoa]

A few days ago just as a reality check I loaded a conventional Q5_K_XL GGUF of the same model via Llama.cpp on my Windows workstation and with the identical prompt and parameters it showed absolutely none of the issues I encountered in the custom GB10 vLLM container, so the looping gibberish behavior is certainly not merely a quirk of the model or a result of me having unrealistic expectations of it.