You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Splash: allow contexts above 262,144 tokens (YaRN) and pass --ple-memory#5071
Environment: oMLX 0.7.3 (macOS app), Splash 1.5.0, macOS 27.2, Apple M5 Max 128 GB. Model: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF (IQ1_M, 2 split files) with incoai/Qwen3.8-Flash-Next-DFlash2.
What I'd like
Two options that Splash 1.5.0 already supports but oMLX does not expose when it launches splash serve:
Context above 262,144 tokens.engine/splash.py caps the requested context with min(options.max_context, SPLASH_MAX_CONTEXT) where SPLASH_MAX_CONTEXT = 262144, and docs/Splash.md says
"(at most 256K)". Splash itself accepts --max-context up to 1,010,000 and turns on static YaRN past the native
window (Qwen's model card: "262,144 natively and extensible up to 1,000,000 tokens").
--ple-memory. Splash can keep a small RAM row cache of the 27 GB per-layer n-gram (PLE) table of a
Flash-Next GGUF and stream the rest from the SSD. oMLX never passes it, so with a full budget the table loads
into memory (--ple-memory auto). The qwen4_ple_ssd_offload model setting only applies to MLX checkpoints.
Today the only way to get either is to run splash serve by hand outside oMLX, which loses the oMLX dashboard,
profiles, idle unload and memory accounting for that model.
Measurements (same GGUF, splash serve started directly)
Setup
Decode, 2 parallel requests
Prefill 128K
Peak memory (all-in)
Through oMLX (PLE table in RAM)
81 tok/s
1,616 tok/s
~61 GB
--ple-memory 2G
89 tok/s
1,795 tok/s
34-40 GB
--ple-memory 2G --max-context 750K (YaRN ×2.93), 32K context per request
59 tok/s
1,704 tok/s
40-46 GB
Streaming the PLE table cost nothing measurable and saved ~20-25 GB.
Long-context recall (5 keys planted at 5-95 % depth in a real Kotlin code base): 5/5 at 128K, 250K, 480K and
730K with --max-context 512K or 750K. 500K prefill: 1,381 tok/s (TTFT 6 min); 730K: TTFT 9 min.
Short-context regression with YaRN 750K on: our agent coding suite stayed at 9/9 (same as native).
Possible shape
Raise SPLASH_MAX_CONTEXT to Splash's own limit (1,010,000), or only cap at the native window unless the model's max_context_window setting asks for more — the same rule patches/qwen35_yarn.py already uses for MLX qwen4_exp models (yarn_factor = window / native).
A per-model setting for the PLE row cache (e.g. reuse qwen4_ple_ssd_offload for Splash GGUFs and map it to --ple-memory <size>), and count only the cache in the memory estimate. That would also change the estimate
discussed in Splash Split GGUF incorrect memory requirement #4876.
Related: #4677 (leave the context to Splash when the budget cannot hold the limit), #4876 (Splash split GGUF memory
requirement), #1077 (first-class YaRN config, MLX side).
Happy to test a build or share the full benchmark data.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Environment: oMLX 0.7.3 (macOS app), Splash 1.5.0, macOS 27.2, Apple M5 Max 128 GB.
Model: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF (IQ1_M, 2 split files) with incoai/Qwen3.8-Flash-Next-DFlash2.
What I'd like
Two options that Splash 1.5.0 already supports but oMLX does not expose when it launches
splash serve:engine/splash.pycaps the requested context withmin(options.max_context, SPLASH_MAX_CONTEXT)whereSPLASH_MAX_CONTEXT = 262144, and docs/Splash.md says"(at most 256K)". Splash itself accepts
--max-contextup to 1,010,000 and turns on static YaRN past the nativewindow (Qwen's model card: "262,144 natively and extensible up to 1,000,000 tokens").
--ple-memory. Splash can keep a small RAM row cache of the 27 GB per-layer n-gram (PLE) table of aFlash-Next GGUF and stream the rest from the SSD. oMLX never passes it, so with a full budget the table loads
into memory (
--ple-memory auto). Theqwen4_ple_ssd_offloadmodel setting only applies to MLX checkpoints.Today the only way to get either is to run
splash serveby hand outside oMLX, which loses the oMLX dashboard,profiles, idle unload and memory accounting for that model.
Measurements (same GGUF,
splash servestarted directly)--ple-memory 2G--ple-memory 2G --max-context 750K(YaRN ×2.93), 32K context per request730K with
--max-context 512Kor750K. 500K prefill: 1,381 tok/s (TTFT 6 min); 730K: TTFT 9 min.Possible shape
SPLASH_MAX_CONTEXTto Splash's own limit (1,010,000), or only cap at the native window unless the model'smax_context_windowsetting asks for more — the same rulepatches/qwen35_yarn.pyalready uses for MLXqwen4_expmodels (yarn_factor = window / native).qwen4_ple_ssd_offloadfor Splash GGUFs and map it to--ple-memory <size>), and count only the cache in the memory estimate. That would also change the estimatediscussed in Splash Split GGUF incorrect memory requirement #4876.
Related: #4677 (leave the context to Splash when the budget cannot hold the limit), #4876 (Splash split GGUF memory
requirement), #1077 (first-class YaRN config, MLX side).
Happy to test a build or share the full benchmark data.
All reactions