Skip to content

v1.1.73

Latest

Choose a tag to compare

@mw00 mw00 released this 22 Nov 01:27

Inkling 2 Mini now runs on far more PCs: NVIDIA cards from 8 GB, V100 and RTX 20 cards, and AMD on Linux. It also drafts where that is faster, with the same answers. And there are fixes for PCs that could not run Maya: the VRAM reserve, and Hy4-Flash on two GPUs.

  • Inkling 2 Mini on cards from 8 GB (NVIDIA, Linux). The most-used experts stay on the card and the rest in RAM; the GPU and the CPU share the missing ones, split by what your PC measures once. The answers are the same wherever an expert runs. With an 8 GB card it needs about 13 GB of RAM; the setup tells you what your PC needs.
  • Inkling 2 Mini on V100 and RTX 20 cards, with their own fast kernels.
  • Inkling 2 Mini on AMD (Linux): Radeon cards with 24 GB and the Strix / Gorgon Halo. Tested on a Radeon 8065S; Radeon RX 7000 / 9000 cards are untested, so please report how it goes.
  • Drafting for Inkling 2 Mini (--drafting auto). The official drafter and earlier text of the conversation propose tokens, checked so that the answers are the same as without drafting. Your PC measures once whether it is faster and keeps the faster choice.
  • Faster on every card: the looped layers' gate step, with the same answers.
  • Fix: --vram-reserve-mib is kept (#1026). It was ignored on a normal start, and DeepSeek used up the reserve after placing its experts, so requests could fail on 8 GB cards. DeepSeek now keeps the reserve free for the whole run; the first start after updating may restart the engine once by itself to measure it.
  • Fix: Hy4-Flash on two GPUs (#1100, #1084). Copies between the GPUs go through RAM where direct copies are not safe on the PC (the IOMMU), and a crash at the start on some two-GPU PCs is gone.
  • Credits. Inkling 2 Mini's engine is @alesha-pro's (with @mrsirg97-rgb, @OscarSheng and @cobra91), and the experts-in-RAM support builds on @OscarSheng's code. The rest of this release is by @mw00 (peasantsmith).

Full list: CHANGELOG.md.

Supported by @wjHunter, @os-groot, @davedgd, @jeanluchayes, @sugattor, @vikcious, @ThatSimpleTech, @router0mail and @davidfernengel.

Maya is free; if it helps you, you can sponsor it: https://github.com/sponsors/mw00