Skip to content

MoE-Direct v0.2

Choose a tag to compare

View all tags
@tmxkzm1925-max tmxkzm1925-max released this 31 Jul 06:59
· 62 commits to main since this release
v0.2
46532ec

MoE-Direct v0.2 - first public build

Download moe-direct-v0.2-win-x64.zip, not GitHub's auto-generated "Source code" archive - that one contains no binaries.

The first build you can actually run: a public preview for hands-on users, Windows only, unsigned. It serves Mixture-of-Experts models far larger than system RAM by keeping routed expert weights on the NVMe SSD and reading, per token, only the experts that token routes to.

What is in it

  • Direct-read serving - expert weights stay on disk in an aligned store and are read on demand into a fixed, model-aware RAM cache (unbuffered overlapped I/O, no OS page cache). The math, the routing, the weights and the sampling are untouched - only the storage path underneath the expert tensors changed.
  • A repacker that proves itself - the one-time repack rewrites expert tensors and SHA-256 compares every record against the source bytes, no sampling; launcher and engine both refuse to serve a repack that did not fully verify.
  • A launcher that shows its work - twelve steps from bundle check to clean shutdown; the repack plan is shown and confirmed before anything is written; every run ends in one machine-readable status= line, and failures add a plain-language hint.
  • A startup queue-depth sweep - the launcher measures your drive at QD 1/2/4/8 instead of assuming ours; your override always wins over the measurement.
  • Two independent gates - format (the repack is byte-exact and verified) and performance (a run passed the frozen release gate) are recorded separately, and neither implies the other.
  • Five model profiles with exact repos and revisions: Qwen3.5-122B (start here), gpt-oss-120b, Qwen3.5-35B, Qwen3.5-397B, Kimi K2.6 (1T-class). Weights are never redistributed.
  • Local by default - the server binds to loopback; no telemetry, no automatic uploads, no background updater. The README enumerates every file this project writes.

Numbers

Reference machine: Ryzen 7 7800X3D, 32 GB DDR5, RTX 5080, PCIe Gen5 NVMe, Windows 11.

  • Qwen3.5-122B sustained decode 5.59-5.69 tok/s, frozen release gate passed; the shipped binary re-checked in the anchor band at 6.32 tok/s average in a vicinity run.
  • gpt-oss-120b integrity anchor: 12 paired responses, token IDs identical between direct-read off and on.
  • Observations (measured, no gate): Kimi K2.6 at 1.03 tok/s with coherent output; a matched K2.6 direct-read vs plain-mmap pair at 3.95x; Qwen3.5-397B 1.99 tok/s; Qwen3.5-35B 4.40 tok/s CPU-only on a 16 GB laptop.

Good to know

  • "Read speed divided by miss bytes per token" is an I/O ceiling, not an expected tok/s - the launcher prints the ceiling and a calibrated estimate separately, and marks the fixed term [unmeasured] off the reference machine.
  • K2.6 at 1.03 tok/s is a demonstration of reach, not an interactive experience; its performance gate is unpassed.
  • The first-run repack costs minutes to roughly 18 minutes on the recorded machine and roughly the model's size again on disk, with no resume; the output is never deleted automatically.
  • One request at a time; an OpenAI-compatible subset; text only; MoE only; prefetch serves only on validated profiles, and overrides elsewhere are refused.

Details, full tables and every limitation: README. Security issues go through SECURITY.md, not public issues.

Verify the download

One paste, nothing to compare by eye - in Explorer, open the folder that holds both downloaded files, right-click empty space > Open in Terminal (PowerShell), then paste; it prints OK or MISMATCH:

$e=((Get-Content .\SHA256SUMS.txt -Raw).Trim() -split '\s+')[0];$a=(Get-FileHash .\moe-direct-v0.2-win-x64.zip -Algorithm SHA256).Hash;if($a -eq $e){'OK: hash matches'}else{'MISMATCH: download again'}

Then unblock the zip (right-click > Properties > Unblock) before extracting with Windows "Extract All" into a new, empty folder - unblocking after extraction does not clean up files already extracted. If you skip the paste, you are still protected: the launcher re-verifies every file inside the extracted bundle against its sealed manifest on every start, fail-closed. Only the paste establishes that the zip itself is the released one.

Built on llama.cpp (MIT), base commit 0bd0ec6 (b10057); upstream copyright, license and notices preserved. Built in an AI-assisted, measurement-first workflow; full credits in the README.

Assets 4
Loading