Skip to content

v1.1.0

Choose a tag to compare

@al2m4n al2m4n released this 18 Oct 22:21
acb9f36

TheYgent 1.1 runs mixture-of-experts models bigger than your Mac's memory. On the MLX engine, a mixture-of-experts (MoE) chat model can keep its experts on the SSD. Each layer reads only the experts its router picks, and recently used experts stay in a memory cache. The answers are identical to the fully loaded model's; it's slower.

Expert streaming on MLX (#85)

  • Same answers, far less memory. The model's shared part stays loaded, and the routed experts are read from the model's own files as each token needs them. Measured with Qwen3.6-35B-A3B in 4-bit (20 GB) on a 36 GB Apple Silicon Mac:

    How it runs Memory Speed
    Fully loaded 19.7 GB 53 tokens/s
    Streaming, every expert cached 20 GB 34 tokens/s
    Streaming, 4 GB cache, cold SSD 5.7 GB 9 tokens/s
    Streaming, smallest cache, cold SSD 1.8 GB 4 tokens/s
  • Off by default, per model. A model that fits in memory is always faster fully loaded. To turn streaming on:

    • in the catalog's install dialog;
    • in a model's settings on the Registries page;
    • through the API, with the registration params streamExperts and expertCacheGb.
  • Cache size. Expert cache (GB) defaults to half your memory, less the model's shared part.

  • Find MoE models in the catalog. The Hugging Face catalog marks mixture-of-experts models MoE, and a new MoE filter shows only those. Installing the MLX variant of one that won't fit offers Stream experts from disk, already ticked.

  • Loud refusals. A model the feature can't stream is refused when its engine starts, with the reason in the engine log. That covers a model without experts, one whose experts are a kind it doesn't read, and a checkpoint that wasn't converted for MLX.

Start with MoE models bigger than memory in the docs, or read how it works, with the numbers.

Also in this release

  • Catalog installs can carry registration params. They're checked before the download starts.
  • Edit registration. Models registered from a Hugging Face id get an Edit registration button to reach their settings.
  • New environment variable THEYGENT_MLX_PYTHON. It points expert streaming at the Python that mlx-lm is installed in. That Python is normally found from mlx_lm.server on its own.
  • Website.
    • The new blog post above (#85).
    • A dedicated AI Gateway page (#84).
    • An agent-templates gallery and an architecture blog post (#83).

Upgrading

No database migrations and no changes to agents or the IR. Expert streaming needs mlx-lm (make engines installs it on a Mac); it was verified with mlx-lm 0.31.3.

Known limits

  • Apple Silicon, MLX, chat only. Expert streaming is for mixture-of-experts chat models in MLX format, on Apple Silicon. llama.cpp and vLLM are not covered.
  • Verified architectures. Verified on Qwen3-MoE and Qwen3.6-MoE. Other architectures built on mlx-lm's standard expert block should work but are unverified.
  • Long prompts. The first reply to a long prompt is slow from a cold disk, because processing it touches nearly every expert.
  • Parallel requests make the cache less effective.

Full changelog: v1.0.2...v1.1.0