Performance questions

[JJOne123]

Hi

First this is not meant as a bash against omlx. I love omlx and use it daily and have been happy to help file issues etc to help improve the product where I could.

But I decided to try a different product recently (mlx-serve) after reading a post on Reddit about it.
It didn't work oob with my jundot qwen3.8 FN oQ4e image, so I had to download a slightly different model, but which has a similar size, here: https://huggingface.co/ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit

Anyways, on mlx-serve it was telling me that my prefill was getting up to 1.9k at times on a M5 Max 128gb. TG was about 50-80 per second depending on the work. I wasn't able to see any option on mlx-serve to offload the ngram to SSD as I do on omlx to preserve more memory for context etc. But it's size loaded into memory to be fair was quite on par with the jundot model so I presume it also had ngram SSD offloading.

This is a substantially performance difference between the 2 engines. Mlx-serve did feel quicker on my own harness when running it too. Omlx tends to give me around 12-1400pp and about 55tg on 0.7.0-dev4 build.

Ultimately I've reverted my long term work to omlx though as I use other models and I found the jundot model more stable.

But I won't lie, the performance difference is noticeable and it's always nice to have more speed.

Is there any idea how or why mlx-serve is running faster, or is there a way to port some of the performance improvements to omlx?

Maybe it's all down to the model?
I'll try finding a model what works across both engines for a closer comparison.

[JJOne123]

I just installed oMLX 0.7.0rc1 build and whoa nelly, thats a speedy meatball! 👍
Much closer if not equal to mlx-serve so far and I seem to get better stability with the oQ4e models.

Anyone reading this, I highly recommend upgrading to 0.7.0rc1 or later.

This humble pie tastes pretty good! XD

Well done and thank you to everyone who made this happen! 🥇

[nexus-studio-dev]

@JJOne123 mind sharing the settings you are using in oMLX for this model?

[JJOne123]

Hi @nexus-studio-dev

Sure. I'm not at my laptop but IIRC I am using:

• model: Qwen3.8 FN oQ4e from jundot
• model settings: I use the recommended settings for the base model from hugging face. You can look them up there to be precise. I think it's temp 1.0, top k 20, top p 0, presence penalty 0, repetition penalty 0.95
• ngram streaming from SSD
• thinking is enabled
• lightning MTP is enabled. I left the default draft number as is (I think it defaults to 3).
• I turned off cold cache
• hot cache is on at about a 30gb size or so
• I don't use turbo quant or do anything to the kv cache. It's left at the default which works best IMO

I think batch size is set to 4096 but I'd need to double check this.
I have number of simultaneous requests set to 4 for any subagents etc. They rarely get used tbh.

If I didn't mention a specific parameter, presume it's key at the default value.

I'm using a M5 Max mbp with 128gb ram.

HTH

[nexus-studio-dev]

Yeah, I ended up using similar settings.
I have M4 Max 126 GB Mac Studio and I'm using Qwen3.8-Flash-Next-oQ4e-mtp model.
After the upgrade from 0.6.4 to 0.7.0 my generation speed is much faster. Here is what I'm getting currently:

And before update I was getting ~480 PP and 39 TG on average. This is a substantial speedup from just updating the oMLX and leaving the rest the same, kudos @jundot 💪 .

Here are my settings if that helps anyone else:

"Qwen3.8-Flash-Next-oQ4e-mtp": {
      "temperature": 1.0,
      "top_p": 0.95,
      "top_k": 20,
      "force_sampling": false,
      "max_tool_result_tokens": 0,
      "qwen4_ple_ssd_offload": true,
      "deepseek_v41_engram_ssd_offload": false,
      "deepseek_v41_ced_prefill_enabled": false,
      "thinking_budget_enabled": false,
      "guided_grammar_enabled": false,
      "turboquant_kv_enabled": false,
      "turboquant_kv_bits": 4.0,
      "turboquant_skip_last": true,
      "qwen35_ane_prefill_enabled": false,
      "qwen35_ane_prefill_sequence_length": 2048,
      "qwen35_ane_prefill_tail_padding_min_tokens": 0,
      "qwen35_ane_prefill_fraction": 0.53,
      "qwen35_ane_prefill_shared_fraction": 1.0,
      "qwen35_ane_prefill_fused_down": false,
      "qwen35_ane_prefill_max_layers": 64,
      "qwen35_ane_prefill_dual_ane": true,
      "qwen35_ane_prefill_gdn": true,
      "qwen35_ane_prefill_gdn_fraction": 0.5,
      "qwen35_ane_prefill_gdn_max_layers": 48,
      "qwen35_ane_prefill_cpu_enabled": false,
      "qwen35_ane_prefill_cpu_fraction": 0.135,
      "qwen35_ane_prefill_cpu_down_fraction": 0.0,
      "qwen35_ane_prefill_cpu_gdn_fraction": 0.0,
      "qwen35_ane_prefill_cpu_threads": 8,
      "qwen35_ane_prefill_cpu_shared_resource": true,
      "qwen35_oq_a8_enabled": false,
      "qwen35_oq_a8_min_tokens": 128,
      "moe_expert_offload_enabled": false,
      "moe_expert_offload_resident_fraction": 0.25,
      "specprefill_enabled": false,
      "dflash_enabled": false,
      "dflash_draft_quant_enabled": false,
      "dflash_in_memory_cache": true,
      "dflash_in_memory_cache_max_entries": 4,
      "dflash_in_memory_cache_max_bytes": 8589934592,
      "dflash_ssd_cache": false,
      "dflash_ssd_cache_max_bytes": 21474836480,
      "dflash_draft_sink_size": 0,
      "mtp_enabled": true,
      "mtp_adaptive_max_depth": 3,
      "vlm_mtp_enabled": false,
      "is_pinned": false,
      "is_default": false,
      "is_hidden": false,
      "is_favorite": false,
      "trust_remote_code": false
    }