SpeculativeDecodingNotSupportedError: Speculative decoding is not supported for batched MLX models.

[Clete2]

I am running MLX qwen/qwen3-4b and want to use speculative decoding with Qwen2.5 0.5B Instruct, or Qwen3 1.7B, which LM Studio claims are both compatible. However when trying to load either one as the Draft Model in the UI, I get the following error:

2026-02-06 22:24:20 [DEBUG]
 [batched_model_kit][INFO]: Loading model from /Users/clete2/.lmstudio/models/lmstudio-community/Qwen3-4B-MLX-8bit...
2026-02-06 22:24:21 [DEBUG]
 [batched_model_kit][INFO]: BatchedModelKit loaded successfully
2026-02-06 22:24:22 [DEBUG]
 SpeculativeDecodingNotSupportedError: Speculative decoding is not supported for batched MLX models.

At:
  /Users/clete2/.lmstudio/extensions/backends/vendor/_amphibian/app-mlx-generate-mac14-arm64@15/lib/python3.11/site-packages/mlx_engine/generate.py(253): load_draft_model
2026-02-06 22:24:22 [DEBUG]
 lmstudio-llama-cpp: failed to load model. Error: SpeculativeDecodingNotSupportedError: Speculative decoding is not supported for batched MLX models.

Even if I set batch size to 1, or try to make a single call into it, I will still get an error about Speculative decoding not being supported for batched MLX models.

Is it possible to load the model in non-batched mode?

LM Studio 0.4.2 (Build 2)

MacBook Pro M4 Pro 48GB

[piotrchojnowski]

did you manage to solve it ?

[Clete2]

Looks like #272 fixes it but I don’t know until updated LM Studio is released with that patch.

[yeison]

Same error. Was it released yet?

[trigger2k20]

Still not works in Chat and Server mode

[denys-bushuliak]

Set 'Max Concurrent Prediction' to 1 in global settings or in model loading settings.

[trigger2k20]

@Denys-Bushulyak is already set I cannot change it. Beside that I do not find it in the global settings. I have version 0.4.6+1 (0.4.6+1)

More i cannot load any model. I want to use it with the Qwen family.

[yenerguller]

Set 'Max Concurrent Prediction' to 1 in global settings or in model loading settings.

this worked for qwen/qwen2.5-coder-14b -> qwen2.5-0.5b-instruct-mlx, thanks

[trigger2k20]

Set 'Max Concurrent Prediction' to 1 in global settings or in model loading settings.

this worked for qwen/qwen2.5-coder-14b -> qwen2.5-0.5b-instruct-mlx, thanks

Thanks for the info, but 2.5 is already old.... Looking for Qwen 3.X

[yenerguller]

Set 'Max Concurrent Prediction' to 1 in global settings or in model loading settings.

this worked for qwen/qwen2.5-coder-14b -> qwen2.5-0.5b-instruct-mlx, thanks

Thanks for the info, but 2.5 is already old.... Looking for Qwen 3.X

Tests run fine also for qwen/qwen3-8b -> qwen/qwen3-1.7b on my MacBook Pro M4 16GB RAM

[D19cannon]

Set 'Max Concurrent Prediction' to 1 in global settings or in model loading settings.

this worked for qwen/qwen2.5-coder-14b -> qwen2.5-0.5b-instruct-mlx, thanks

Thanks for the info, but 2.5 is already old.... Looking for Qwen 3.X

Tests run fine also for qwen/qwen3-8b -> qwen/qwen3-1.7b on my MacBook Pro M4 16GB RAM

Unfortunately, this is not working on MLX

Failed to send message

Failed to load draft model. SpeculativeDecodingNotSupportedError: Speculative decoding is not supported for batched MLX models.

SpeculativeDecodingNotSupportedError: Speculative decoding is not supported for batched MLX models.

[D19cannon]

Set 'Max Concurrent Prediction' to 1 in global settings or in model loading settings.

this worked for qwen/qwen2.5-coder-14b -> qwen2.5-0.5b-instruct-mlx, thanks

Thanks for the info, but 2.5 is already old.... Looking for Qwen 3.X

Tests run fine also for qwen/qwen3-8b -> qwen/qwen3-1.7b on my MacBook Pro M4 16GB RAM

Unfortunately, this is not working on MLX

Failed to send message

Failed to load draft model. SpeculativeDecodingNotSupportedError: Speculative decoding is not supported for batched MLX models.

SpeculativeDecodingNotSupportedError: Speculative decoding is not supported for batched MLX models.

Ignore this comment. I got it up and running, but it is a hidden hassle to figure out.

To enable speculative decoding:

1. Go to Developer.
2. Load the main model (e.g., one of the models mentioned above, or any other model that supports speculative decoding with a compatible draft model).
3. Click the loaded model to open its settings.
4. In the right-hand sidebar, click Load.
5. Open Advanced settings.
6. Set Max Concurrent Predictions to 1.
7. Reload the model if required.
8. In the right-hand sidebar, navigate to Inference.
9. Open Speculative Decoding.
10. Select a compatible draft model.

Once configured, speculative decoding should be enabled for the loaded model.