Repository navigation
v1.1.0
TheYgent 1.1 runs mixture-of-experts models bigger than your Mac's memory. On the MLX engine, a mixture-of-experts (MoE) chat model can keep its experts on the SSD. Each layer reads only the experts its router picks, and recently used experts stay in a memory cache. The answers are identical to the fully loaded model's; it's slower.
Expert streaming on MLX (#85)
-
Same answers, far less memory. The model's shared part stays loaded, and the routed experts are read from the model's own files as each token needs them. Measured with Qwen3.6-35B-A3B in 4-bit (20 GB) on a 36 GB Apple Silicon Mac:
How it runs Memory Speed Fully loaded 19.7 GB 53 tokens/s Streaming, every expert cached 20 GB 34 tokens/s Streaming, 4 GB cache, cold SSD 5.7 GB 9 tokens/s Streaming, smallest cache, cold SSD 1.8 GB 4 tokens/s -
Off by default, per model. A model that fits in memory is always faster fully loaded. To turn streaming on:
- in the catalog's install dialog;
- in a model's settings on the Registries page;
- through the API, with the registration params
streamExpertsandexpertCacheGb.
-
Cache size. Expert cache (GB) defaults to half your memory, less the model's shared part.
-
Find MoE models in the catalog. The Hugging Face catalog marks mixture-of-experts models MoE, and a new MoE filter shows only those. Installing the MLX variant of one that won't fit offers Stream experts from disk, already ticked.
-
Loud refusals. A model the feature can't stream is refused when its engine starts, with the reason in the engine log. That covers a model without experts, one whose experts are a kind it doesn't read, and a checkpoint that wasn't converted for MLX.
Start with MoE models bigger than memory in the docs, or read how it works, with the numbers.
Also in this release
- Catalog installs can carry registration params. They're checked before the download starts.
- Edit registration. Models registered from a Hugging Face id get an Edit registration button to reach their settings.
- New environment variable
THEYGENT_MLX_PYTHON. It points expert streaming at the Python that mlx-lm is installed in. That Python is normally found frommlx_lm.serveron its own. - Website.
Upgrading
No database migrations and no changes to agents or the IR. Expert streaming needs mlx-lm (make engines installs it on a Mac); it was verified with mlx-lm 0.31.3.
Known limits
- Apple Silicon, MLX, chat only. Expert streaming is for mixture-of-experts chat models in MLX format, on Apple Silicon. llama.cpp and vLLM are not covered.
- Verified architectures. Verified on Qwen3-MoE and Qwen3.6-MoE. Other architectures built on mlx-lm's standard expert block should work but are unverified.
- Long prompts. The first reply to a long prompt is slow from a cold disk, because processing it touches nearly every expert.
- Parallel requests make the cache less effective.
Full changelog: v1.0.2...v1.1.0