Skip to content

AlphaLlama v2 (2.0.9)

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 12 Nov 17:14
v2.0.9
f864ef9

AlphaLlama v2 (2.0.9): fast local inference for large Mixture-of-Experts models on consumer GPUs.

Changed default: fast decode

On the AlphaInfer engine, decode mode fast is now the default (--decode-mode / ALPHA_DECODE_MODE):

Mode What it does Output
fast (default) multi-token prediction with fast verify close to, but not token-identical with, plain greedy decoding
exact multi-token prediction, exact verify token-identical to plain greedy decoding (the default of the exact rows before 2.0.9)
plain no multi-token prediction plain greedy decoding

Use --decode-mode exact when you need the same output as plain decoding. An explicit AI_MTP_MODE /
ALPHA_AI_MTP_K still wins, and ALPHA_SPEED_MODE=1 adds the row's best-speed tuning on top of fast.

Measured on an RTX 3090 with Qwen3.8-27B UD-Q4_K_M (streaming chat, thinking off, median of 3): writing 85 / 72 tok/s
(fast / exact), a whole-file code edit 121 / 97, everyday chat 78-89 / 68-76. Ollama 0.40.11 on the same card: 44.

What's new since 2.0.5

Fixes

  • Qwen3.8-27B, short prompts: a prefill pass could read a layer's input after its memory had been reused, so short
    prompts (tens of tokens) could get a wrong first word, stop early or give an off-topic reply, in every decode mode.
    Fixed; the 27B's next-token predictions now match llama.cpp's on the same tokens (top-1 100%).
  • 16 GB cards, 27B with CPU offload: deep decode rounds that did not fit the fixed workspace aborted the server.
    The engine now measures at start which rounds fit and narrows the ones that do not.
  • First start of a model without a workspace measurement: the engine served with a clamped context, so longer
    prompts failed. It now restarts with the full context before serving.
  • --decode-mode plain starts on the RTX 3090 27B row (it ran out of memory loading a draft head it never uses).
  • With thinking off, a model that writes <think> itself is no longer cut off mid-answer by the answer room.
  • A decode round never returns more tokens than the request has left.
  • 27B on 24 GB cards: exact and plain use the same KV cache type, so exact matches plain again at any context length.

Faster follow-up turns

  • Prompt reuse on Qwen3.6-35B / Qwen3.8-27B: a conversation's next turn resumes from its cached prompt instead of
    prefilling it again (RTX 3090, 30K-token prompt: next turn's first token 0.20 s on the 35B, was 5.5 s; 0.64 s on the
    27B, was 24.9 s). Each conversation goes back to the slot that holds it; two conversations on two slots both keep
    their cache.
  • Flash-Next: keeps a second conversation's cache in host memory (RTX 4080: 0.5 s instead of 6.6 s for a
    16.8K-token conversation's next turn), and an edited prompt resumes from the last saved point before the edit
    (30K-token prompt edited at 90%: first token 6.8 s vs 12.9 s).
  • 24 GB cards, Flash-Next prefill: faster first token by default, same output (median 0.21 / 1.13 / 1.26 s vs
    0.23 / 1.26 / 1.41 s on three test prompts).

API

  • usage.prompt_tokens_details.cached_tokens in every usage block (chat, streaming, /v1/completions): the prompt
    tokens served from cache. /v1/completions now reports prompt_tokens.
  • ALPHA_SPEED_MODE says when the speed drafter file for the model is missing (a log line and /health
    speed_drafter: "missing") and falls back to fast decoding.

Downloads

Package For Tested for this release
alphallama-v2-linux-x64-cuda12.tar.zst Linux x86-64, NVIDIA RTX 20 to 50 series (CUDA 12.8 runtime included). Includes AlphaInfer: the fastest path. Yes: RTX 3090 (24 GB), Qwen3.8-27B (fast / exact / speed mode), Qwen3.6-35B, Qwen3.8-Flash-Next
alphallama-v2-win-x64-cuda12.zip Windows x64, NVIDIA RTX 20 / 30 / 40 series, driver 551+ Smoke test (small model, laptop GPU)
alphallama-v2-win-x64-cuda13.zip Windows x64, NVIDIA including RTX 50 series, driver 580+ Smoke test (small model, laptop GPU)
alphallama-v2-win-x64-vulkan.zip Windows x64, any Vulkan GPU Smoke test (small model, laptop GPU)
alphallama-v2-win-x64-cpu.zip Windows x64, CPU only (AVX2) Smoke test (small model)
alphallama-v2-win-arm64-cpu.zip Windows on ARM (Snapdragon X), CPU Not run
alphallama-v2-macos-arm64.tar.gz macOS 12+, Apple Silicon (Metal) Not yet tested
alphallama-v2-macos-x64.tar.gz macOS 12+, Intel (CPU) Not yet tested

Each file has a .sha256 next to it; please check it before running (on Windows:
Get-FileHash <file> -Algorithm SHA256).

Quick start:

alphallama-serve -m model.gguf --port 8080      # OpenAI-compatible API on http://127.0.0.1:8080/v1

Notes

  • Peak speeds (the AlphaInfer engine, and the decode modes above) need the Linux package on an NVIDIA GPU. The
    Windows and macOS packages run the standard engine, with CUDA, Vulkan or Metal acceleration.
  • The Windows binaries are not code-signed yet. Some antivirus products quarantine new unsigned executables
    heuristically on first run (for example Norton "IDP.Generic"), and Windows Smart App Control blocks them; on a PC
    with Smart App Control on, the Windows packages will not start until they are signed.
  • The macOS packages are ad-hoc signed: run xattr -dr com.apple.quarantine <folder> after unpacking.

Source: commit f864ef9.

Assets 18
Loading