kinfer v0.2.3

MLX models draft with their MTP head, many replies at once go faster, and a load short of memory says which settings did it.

curl -fsSL https://raw.githubusercontent.com/LocalKinAI/kinfer/main/scripts/install.sh | sh

• MTP drafting on MLX. A Qwen3.5 or Qwen3.8 model that keeps its MTP head (Ollama's NVFP4 models, Hugging Face's releases) drafts three tokens a step while a reply writes alone, and one pass checks them. A model under 4 GB of weights does not draft unless -draft N says so, since its steps are too quick for drafts to pay.
• Many replies at once. From eight slots up, each step samples its slots on every core instead of one: Qwen2.5-0.5B at 128 at once went from 2,439 to 2,929 tok/s on an M3 Ultra, and to 3,836 on an M5 Max.
• Prompts where memory is left. The prompt pass grows past Ollama's rule when the context leaves room for it: ornith-1.5:35b read an 11,000-token prompt 21–24% faster on an M5 Max.
• Settings that leave no room. A load that leaves under 256 MiB of the GPU says so, naming the flags set by hand, and a first pass that does run out of memory says the same in its error. A model that will not load is now a 500, not a 404.
• -slots: use 8, 16, 64 or 128, never 17–48 — past sixteen sequences llama.cpp b11998's step time more than doubles.
• MLX also loads the mixtures of experts of Qwen3.5's architecture (Ollama's qwen3.5:35b-a3b-nvfp4, ornith-1.5:122b-nvfp4), checked so far only in parts.

measured	before	v0.2.3
qwen3.8:27b-mlx alone, M5 Max	30 tok/s	38–75 tok/s
Qwen2.5-0.5B, 128 at once, M3 Ultra (temperature 0.7 / 0)	2,439 / 2,597 tok/s	2,929 / 3,207 tok/s
ornith-1.5:35b, an 11,000-token prompt, M5 Max	2,526–2,635 tok/s	3,135–3,206 tok/s

With drafting, five of six MLX replies were the same word for word as without, and the sixth parted at a near tie. The patched libraries are unchanged (darwin_arm64_b11998k6). CHANGELOG.md

kinfer-v0.2.3-darwin-arm64.tar.gz sha256 a193c945ea2916c75bb76fe28572346cf29507487f800239ebba03d25226962e