Skip to content

vMLX 1.6.35

Choose a tag to compare

View all tags
@jjang-ai jjang-ai released this 22 Aug 03:50
· 436 commits to main since this release
v1.6.35
614f89d

vMLX 1.6.35 — SSD-first prompt cache.

The in-memory paged cache is now OFF by default, and prompts are cached on your SSD instead. Memory use stays close to the model's own size: measured across dots3-note, DeepSeek-V4-Flash, MiniMax-M3, Gemma 4, Qwen 3.8 and others, memory tracks the loaded model and does not grow with the cache. dots3-note peaked +3.0 GB over its 95 GB bundle across a conversation climbing to 48k tokens, with no idle drift. Turning the RAM tier back on buys about 2% on time-to-first-token and costs noticeably more memory — Session Settings now says so at the top, and points at the setting.

The SSD cache budget is now a percent of your disk (10% by default) rather than a flat number of gigabytes, so it stays proportionate to the machine instead of being simultaneously too small for a 4 TB drive and too large for a 512 GB one. A "Clear SSD cache for this session" button reports which cache tiers were actually cleared and which were still in use, rather than claiming success either way. One budget covers every cache type vMLX writes — plain KV blocks, typed companion state, native composite records, rotating sliding-window state and multimodal blocks — across every model and session, and the oldest prompts are removed first when it fills.

Stored prefix caches are now full precision for every model family. TurboQuant stored-KV encoding measured under 1% on time-to-first-token, so exactness wins.

Fixes:

  • The SSD budget could be handed the entire disk while the interface reported a 10% cap. Existing sessions that deliberately chose "Unlimited" keep it.
  • DeepSeek-V4-Flash could refuse to start on the new SSD-only default.
  • A session whose model family the registry does not recognise never received the new defaults at all.
  • The cache panel reported success when the engine had in fact declined to clear a tier that was still in use.
  • A bfloat16 vision embedding could raise inside cache-key hashing on the pinned MLX.
  • The engine now says plainly when a cache budget is smaller than a single prompt's block chain, instead of silently storing nothing.

Verified: prompt caches survive a hard kill and restart across five cache architectures — DeepSeek-V4 composite, rotating sliding-window with images, SSM/GatedDelta companion, dots3 DSA and MiniMax sparse MSA — with 47 deterministic prompts byte-identical after restart, image turns included. Cache eviction protects the prefix you are actively reusing and removes idle ones first, across models, including models that are not running.

Both DMGs are Developer ID signed, notarized by Apple, and stapled.

Assets 2
Loading