Repository navigation
Hy4-Flash gets faster everywhere: drafting on small cards, prompts 2-2.5x faster on AMD / Halo, faster long prompts on NVIDIA, contexts up to 1M with the cache in RAM, pictures at full size on every card, and Windows + AMD as experimental.
- Drafting on small cards (DFly2). Besides Tencent's MTP block, Maya's sizes can now draft with a small DFly2 drafter that needs under 1 GB of VRAM, so 8-12 GB cards get drafting too (on 8 GB cards at shorter contexts: Maya loads the drafter only where it leaves the rest of the setup, and so the answers, unchanged). Where both fit, Maya measures both once on your PC and keeps the faster. As always, the model checks every drafted token: same answers, only faster. On our test PCs: an 8 GB RTX 3050 at a 4K context +29% on code and +3% on chat, a Gorgon Halo +40-85% with Maya-XS.
- AMD and Halo: much faster. Prompt reading is about 2-2.5x faster and drafting decode about 30% faster on a Gorgon Halo (Radeon 8065S) with Maya-XS. Also fixed: on AMD, Maya-L's answer with drafting could differ from the one without; they now match exactly.
- Faster long prompts on NVIDIA. About +8% at 16K tokens, +28% at 64K and +56% at 128K on three Tesla V100s; +7-14% on one card. Prompts up to 8K tokens read exactly as before.
- Faster decoding with drafts on several GPUs (#1056). About +9-12% on multi-GPU setups with drafting, and about +10% on the Halo; same answers. On newer GPUs (NVIDIA RTX 30 and newer, AMD on Linux) a second GPU stream adds more; older cards keep the path that is faster for them.
- Faster small cards. Where the CPU computes the experts that don't fit the card: about +3-18% decode depending on the CPU, same answers. On some CPUs Maya also keeps 4-bit copies of the busiest experts (exact results), measured once per PC and kept only where faster (+7-12% more on our RTX 3050 PC).
- Contexts up to 1M on ordinary cards. Where a long context's cache does not fit in VRAM, Maya keeps it in RAM and reads only the positions each token needs. The setup and the Models page now offer the contexts that fit this way: an 8 GB card reaches 256K, a 16 GB card 1M (with about 16 GB of RAM for the cache). Same answers as with the cache in VRAM, a few percent slower where it streams.
- Pictures take no VRAM of their own. A picture is read in buffers that are idle at that moment, so the VRAM v1.1.68 kept free for pictures now holds experts, and 8 GB cards read pictures at full size.
- Hy4-Flash on Windows + AMD (experimental). The setup no longer stops on a Windows PC with an AMD GPU (Radeon or Strix / Gorgon Halo): it builds and runs Hy4-Flash there and says it is experimental. We have no such PC to test on; if it misbehaves, please open an issue with the engine's log (how to fall back).
- Monitor fix. With two or more GPUs, "Where the experts live" showed no expert in VRAM although all of them were on the cards. It now counts every card; hover notes explain the other figures.
- Fixes. GLM on Windows with a Radeon card could fail to start ("warmed expert tables did not upload", #1035 / #924; a user is testing the fix). Hy4-Flash's engine could crash on AMD when Maya stopped it. A Hy4-Flash size too big for the card now says which size, and which smaller one fits.
- Opt-in: experts that follow the answers (
HY4_HOT=1). On cards that hold only part of the experts, the ones in VRAM follow what the answers use: about +22% decode with 16 GB of VRAM, +7% with 12 GB. Off by default; when on, wording can differ slightly. - Community fixes: #1015 (@Mqrius), #1013 (@pepuscz), #1053 (@paltian), #1014 (@TheArchitectit), #935 (@NektoDron), #894 (@Cayahuanca), #1041 (@Andrey0710), #1037 (@EncryptionLTD), #1047 (@zehm). Thank you all.
- Credits. Faster decoding with drafts on tensor parallel (#1056) is by @needmorevram. Faster long-prompt reading builds on ideas from @alesha-pro's Hy4-Flash engine work (the indexer's scores on the tensor cores and its picks split over the cards), and the idea behind
HY4_HOT, an adaptive expert cache. The rest of this release is by @mw00 (peasantsmith).
Full list: CHANGELOG.md.
Supported by @wjHunter, @os-groot, @davedgd, @jeanluchayes, @sugattor, @vikcious, @ThatSimpleTech, @router0mail and @davidfernengel.
Maya is free; if it helps you, you can sponsor it: https://github.com/sponsors/mw00