Project Maya v1.1.3

DFlash2 kept only where it is faster (a speed calibration on your PC), an expert profile that fills VRAM well from the first answer, an opt-in skip of the experts that would come from the SSD, and the EXL3 quant back in the Windows setup.

Update: the dashboard's About > Update, or git pull, then ./setup.sh (Windows: START-MAYA.bat). The engine recompiles; nothing is downloaded again. A setup that drafts with DFlash2 runs the speed calibration once at its first start (5-20 minutes, every step shown).

What's new

• The speed calibration (DFlash2). DFlash2 makes some PCs faster and others slower: it depends on how much of the model fits in the GPUs' memory, and on the CPU and the RAM. When the setup drafts with DFlash2 - and once, at the first start after this update, for a config that already does - Maya measures it on your PC against the MTP block (one GPU: against no drafting) and keeps the faster. It loads the model twice; each time it reads a long document, writes about 1,500 tokens of answers after a warm-up (sampled as the dashboard samples, the same questions both times) and compares the sustained speeds; DFlash2 stays only when it is more than 5% faster. ./maya.sh --speed-calibration measures again; --draft dflash2 --no-speed-calibration keeps DFlash2 without measuring.
• An expert profile for every GLM pack (#88 by @needmorevram). A start fills VRAM by a profile built from Maya-L's routing - code, math, eight languages, long documents and tool calls, with MTP drafting on - instead of expert order or one machine's usage file: on 9 GPUs a fresh start decodes 35.4 -> 40.7 tok/s. The draft layer keeps at least its card's equal share of slots. Learning across starts is opt-in (--expert-profile-save FILE), and the usage file is read only when STRATA_GLM_USAGE names it. An installed config gets the profile at its next start.
• Skip the experts that would come from the SSD (#109 by @Ajay9o9; opt-in, changes the answers). STRATA_GLM_DISK_SKIP_W=0.12: where the model does not fit in VRAM + RAM, a routed expert held by neither and with less than 12% of the route's weight is left out of the token instead of read from the SSD - an RTX 3090 with 60 GB of RAM decoded 62% faster, an RX 7900 XTX 54%. The answers move away from the full model's, more the less RAM there is: KL 0.12 with 60 GB of RAM (the top token the same 88% of the time), 0.19-0.23 with 24-40 GB (75-78%). Unset, nothing changes.
• The experimental EXL3 quant is offered on Windows again, as v1.1.0 released it; v1.1.1 had hidden it there after one crash report (#129), which we are still looking into with its reporter.
• The setup's DFlash2 choice says what it gives again - the same answers, faster - now that the speed calibration checks it on every PC.

Checked

• The speed calibration end to end on one and two Tesla V100s, where DFlash2 is slower: it kept no drafting and the MTP block there, wrote the result into the config, and a later start did not measure again; it leaves the learned profile, the usage file and the kept conversations untouched. Both outcomes are also tested with stand-in engines.
• The expert profile: on two V100s a fresh start decodes as fast as with that PC's own usage file on new topics, and 8% faster than with neither; on one V100, as fast as v1.1.2. It changes no token's computation: the one-token decode path gives the same output as v1.1.2's (KL 0) on NVIDIA and AMD. Its parity and profile tests pass.
• Skipping disk experts: unset, the same tokens as v1.1.2 on NVIDIA (one GPU and two) and AMD; set, on a V100 with part of the model on the SSD, decode 42-150% faster.
• The server end to end on two GPUs (answers, pictures, context reloads, a clean stop); the setup's tests; the engine builds on Linux, Windows and AMD; the GitHub checks pass.