How to configure DeepSeek V4 Flash 0731?

[noonkho]

Hi, I am trying out the DeepSeek V4 Flash 0731 and I wonder what is the 'best' setup config for it in omlx?

Model: DeepSeek-V4-Flash-0731-oQ8e-mtp

My current setup:

CTX Window: 1048576 (Load Defaults)
Temperature: 1
Top P: 0.95
Max tokens: 384000 (recommended for high/max reasoning effort levels)
Enable Thinking: True
Lighting MTP: True
Chat Template Kwargs: REASONING_EFFORT max (not sure if I should click Force or not)

---

Something interesting

When I load and run it in the chat, I asked the model "What model are you?" and the response is Gemini which is not the same response if I asked DeepSeek V4 Flash online.

omlx chat:

online Deep Seek:

I don't know if it is because I have other models running at the same time (mlx-community/gemma-4-31B-it-qat-mxfp8 with VLM MTP mlx-community/gemma-4-31B-it-qat-assistant-mxfp8, jwindle47/chandra-ocr-2-8bit-mlx, mlx-community/Qwen3-Embedding-8B-4bit-DWQ) and causing the response to be different?

Fix? If I set the REASONING PARSER to deepseek_v4 (DeepSeek-V4) (Should I choose this option?)

Then the response is more closer, but not the knowledge cutoff date.

---

Performance Benchmarks

Sorry I'm new to LLM and I'm not sure what continuous batching tests mean... (like what this implies in application?).

---

How to get better performance?

System:  M3 Ultra, 512 GB Memory, macOS Tahoe 26.5.1
oMLX version: v0.5.4

Resource Management

Memory guard: aggressive (True)
Hot Cache Limit (In-Memory Cache): 12% (61 GB)
Cold Cache Limit (SSD Cache): 20% (744 GB)
Max Concurrent Requests: 1 (set to 2 will drop the tg TPS by half or even worse)
Embedding Batch Size: 32
Chunked Prefill: False
Prefill Priority: Max Context
Ide Timeout: None (disabled)

Cache

Cache Enabled: True
Hot Cache Only: False

Advanced

Burst Decode: Aggressive (fastest)
Preserve Prefix Cache for Mid-Conversation System Notes: True
Initial Cache Blocks: 1024
SSE Keepalive Mode: Chunk (default, OpenClaw / WorkBuddy compatible)

I wonder if there are any resources recommended to learn all the terms, knowledge, VLM MTP, DFlash, MTP, Quantization, better resource management, reasoning parser, TurboQuant KV Cache, SpecPrefill, etc.? Like I really want to maximize the usage of the project and/or contribute more as I'm a beginner and not sure how to support the community.

Thank you for your patience and reading this entire message.

[ParticleOne]

My settings are like yours, though I reduced the overall context size due to "only" having a 256GB mac. I'd hoped we'd be getting presets when new models like this came out. The settings below don't capture everything as they fall back to defaults for things like repetition, top_k, etc. I've set all of those to be disabled.

"DeepSeek-V4-Flash-0731-MXFP4-MLX": {
    "max_context_window": 512000,
    "max_tokens": 393216,
    "temperature": 1.0,
    "top_p": 0.95,
    "force_sampling": false,
    "chat_template_kwargs": {
      "reasoning_effort": "max"
    },
    "forced_ct_kwargs": [
      "reasoning_effort"
    ],
    "enable_thinking": true,
    "thinking_budget_enabled": false,
    "reasoning_parser": "deepseek_v4",
    "guided_grammar_enabled": false,
    "turboquant_kv_enabled": false,
    "turboquant_kv_bits": 4.0,
    "turboquant_skip_last": true,
    "specprefill_enabled": false,
    "dflash_enabled": false,
    "dflash_draft_quant_enabled": false,
    "dflash_in_memory_cache": true,
    "dflash_in_memory_cache_max_entries": 4,
    "dflash_in_memory_cache_max_bytes": 8589934592,
    "dflash_ssd_cache": false,
    "dflash_ssd_cache_max_bytes": 21474836480,
    "mtp_enabled": true,
    "vlm_mtp_enabled": false,
    "is_pinned": false,
    "is_default": false,
    "is_hidden": false,
    "is_favorite": false,
    "trust_remote_code": false
  }

I'd extend your question by asking what model to use.
I first tried dfp-official/DeepSeek-V4-Flash-0731-oQ8e-mtp. I ran out of memory often when running my test prompt. (This may hae been due to something in a 0.5.4rc build?)
Next I tried DeepSeek-V4-Flash-0731-oQ4e-mtp which worked well and didn't use memory like it was going out of style. Then I noticed DeepSeek-V4-Flash-0731-MXFP4-MLX which said it's bit perfect and uses the same amount of memory. That's the one I've confinued with and I get about 40 tok/s with it, though that speed doesn't matter when on max thinking because the model takes forever with max thinking.

Thanks for this great project, Jundot!

[DiscoStew6082]

I profiled and optimized the DeepSeek V4 Flash 0731 prefill path against oMLX main at 49ec2716 on a Mac Studio M3 Ultra with 512 GB unified memory.

The comparison used the same existing official native mixed FP4/FP8 checkpoint, all native kernels enabled, unchanged DSpark/MTP settings, cache disabled, temperature 0, and repeated cold prompts.

Results:

• 17,219-token cold prefill, median of 5 runs:
  • 35.97s → 30.87s
  • 478.67 → 557.81 prompt tok/s
  • 16.53% higher throughput / 14.18% lower prefill time
• Fixed 5,476-token prompt plus 512-token decode:
  • 25.27s → 22.73s total
  • 10.05% lower end-to-end time
• Paired objective evaluation:
  • upstream 717/864
  • candidate 718/864
  • statistically unchanged overall accuracy; 95% paired interval −0.58 to +0.81 percentage points
• Prefix-cache cold and repeated-hit outputs remained exact.
• Full oMLX suite: 7,926 passed, 0 failed.

The patch is shape-gated to the measured DeepSeek V4 production paths. It combines faster MXFP4 dispatch, native compressed-attention routes, an indexer fast path when the entire pooled set fits under top-k, fused hyperconnection prefill expansion, and fused expert routing/block-plan construction.

Code and reviewable diff:

• Optimize DeepSeek V4 prefill on M3 Ultra DiscoStew6082/omlx#1
• https://github.com/DiscoStew6082/omlx/blob/main/benchmarks/deepseek_v4_flash_prefill_m3_ultra.md

I have not opened an upstream PR. If the maintainers are interested, I’m happy to rebase the focused patch onto the latest main and open one for review.

[jundot]

@DiscoStew6082, thank you for putting this together. The profiling and validation are thorough, and I appreciate that the patch stays focused on the measured production paths. The fused expert route-plan construction and indexer fast path are particularly well done.

I tested the patch independently on the same M3 Ultra. With the official deepseek-ai FP8 checkpoint, the median time for my 2,045-token cold request dropped from 4.215s to 3.440s, an 18.4% reduction, with identical deterministic outputs. The mixed MXFP4/MXFP8 oQ4e checkpoint also preserved the checked output, and the complete oMLX suite passed with 7,926 tests. This is a substantial improvement and a seriously impressive contribution.

I found one deterministic output regression on the more aggressive oQ2.5 variant, so I would like the upstream review to include low-bit quantization coverage and per-path numerical tests. That does not take away from the quality of the work here. It is exactly the kind of edge case that is easier to sort out in a focused PR.

Please open an upstream PR against the current main with the benchmark evidence and reproduction commands. I would be very happy to review this properly and work through the remaining accuracy and cross-chip details with you.

[DiscoStew6082]

Thank you — I’ll open a focused upstream draft PR from the validated branch and treat oQ2.5 parity as the first blocker. Could you share the exact oQ2.5 checkpoint, request/config, and expected deterministic output or hash you used? I’ll add that case plus per-path numerical coverage and the benchmark reproduction commands before marking the PR ready for merge.

[jundot]

@DiscoStew6082 Sure. The checkpoint was Jundot/DeepSeek-V4-Flash-0731-oQ2.5e, revision 5c8b681. I tested on an M3 Ultra with 512 GB, comparing upstream 49ec2716 with candidate 4d3d8e2. The server prefix cache was disabled, and this was the non-MTP checkpoint.

The /v1/chat/completions request was:

{
  "model": "DeepSeek-V4-Flash-0731-oQ2.5e",
  "messages": [
    {
      "role": "user",
      "content": "Return only the integer: Convert binary 101101 to decimal."
    }
  ],
  "temperature": 0,
  "top_p": 1,
  "max_tokens": 64,
  "chat_template_kwargs": {
    "enable_thinking": false
  }
}

The greedy results were repeatable across clean runs.

Upstream:

101101₂ = 1×32 + 0×16 + 1×8 + 1×4 + 0×2 + 1×1 = 32 + 8 + 4 + 1 = 45.

45

Candidate:

135

I also isolated the two numerical changes. For the first generated token, upstream produced logits of 32.500 for 101 and 31.375 for 135. With only the candidate attention path enabled, they became 31.250 and 31.625; with only the fused hyperconnection path enabled, they became 30.500 and 31.875. Each change independently flipped the greedy token, so this did not look like sampling noise.

For scope, the same prompt remained correct with the candidate on the official FP8 and mixed oQ4e checkpoints. The divergence only reproduced on this oQ2.5 checkpoint in my testing. I can also attach the raw responses or a small reproduction script to the PR if useful.

[DiscoStew6082]

sounds good I will have a PR file today as soon as codex finishes some additional testing and verification

[DiscoStew6082]

Draft PR filed: #2553

I downloaded Jundot/DeepSeek-V4-Flash-0731-oQ2.5e at exact revision 5c8b6811550e8ef0c02c006a3fdfc4e724ffec1c and reran your request sequentially with prefix caching disabled.

• upstream 49ec2716: the same 50-token response you reported, ending in 45 (17 prompt tokens, 0 cached tokens, 2.80 s)
• candidate 16945d29: byte-identical response ending in 45 (17 prompt tokens, 0 cached tokens, 2.62 s)

The fix derives a mandatory guard from DeepSeek V4 checkpoint metadata: sub-4-bit configs retain the upstream dense pooled-attention and compiled hyperconnection paths, while the other native MoE improvements remain enabled. I also restored the previous large-block threshold specifically for 2/3-bit affine kernels.

Validation is 210/210 focused tests and 7,900 passed / 70 skipped / 74 deselected for the full suite. I left the PR as a draft for cross-chip review. Thanks again for the exact isolation details.