AlphaLlama v2 (2.0.9)

AlphaLlama v2 (2.0.9): fast local inference for large Mixture-of-Experts models on consumer GPUs.

Changed default: fast decode

On the AlphaInfer engine, decode mode fast is now the default (--decode-mode / ALPHA_DECODE_MODE):

Mode	What it does	Output
fast (default)	multi-token prediction with fast verify	close to, but not token-identical with, plain greedy decoding
exact	multi-token prediction, exact verify	token-identical to plain greedy decoding (the default of the exact rows before 2.0.9)
plain	no multi-token prediction	plain greedy decoding

Use --decode-mode exact when you need the same output as plain decoding. An explicit AI_MTP_MODE /
ALPHA_AI_MTP_K still wins, and ALPHA_SPEED_MODE=1 adds the row's best-speed tuning on top of fast.

Measured on an RTX 3090 with Qwen3.8-27B UD-Q4_K_M (streaming chat, thinking off, median of 3): writing 85 / 72 tok/s
(fast / exact), a whole-file code edit 121 / 97, everyday chat 78-89 / 68-76. Ollama 0.40.11 on the same card: 44.

What's new since 2.0.5

Fixes

• Qwen3.8-27B, short prompts: a prefill pass could read a layer's input after its memory had been reused, so short
  prompts (tens of tokens) could get a wrong first word, stop early or give an off-topic reply, in every decode mode.
  Fixed; the 27B's next-token predictions now match llama.cpp's on the same tokens (top-1 100%).
• 16 GB cards, 27B with CPU offload: deep decode rounds that did not fit the fixed workspace aborted the server.
  The engine now measures at start which rounds fit and narrows the ones that do not.
• First start of a model without a workspace measurement: the engine served with a clamped context, so longer
  prompts failed. It now restarts with the full context before serving.
• --decode-mode plain starts on the RTX 3090 27B row (it ran out of memory loading a draft head it never uses).
• With thinking off, a model that writes <think> itself is no longer cut off mid-answer by the answer room.
• A decode round never returns more tokens than the request has left.
• 27B on 24 GB cards: exact and plain use the same KV cache type, so exact matches plain again at any context length.

Faster follow-up turns

• Prompt reuse on Qwen3.6-35B / Qwen3.8-27B: a conversation's next turn resumes from its cached prompt instead of
  prefilling it again (RTX 3090, 30K-token prompt: next turn's first token 0.20 s on the 35B, was 5.5 s; 0.64 s on the
  27B, was 24.9 s). Each conversation goes back to the slot that holds it; two conversations on two slots both keep
  their cache.
• Flash-Next: keeps a second conversation's cache in host memory (RTX 4080: 0.5 s instead of 6.6 s for a
  16.8K-token conversation's next turn), and an edited prompt resumes from the last saved point before the edit
  (30K-token prompt edited at 90%: first token 6.8 s vs 12.9 s).
• 24 GB cards, Flash-Next prefill: faster first token by default, same output (median 0.21 / 1.13 / 1.26 s vs
  0.23 / 1.26 / 1.41 s on three test prompts).

API

• usage.prompt_tokens_details.cached_tokens in every usage block (chat, streaming, /v1/completions): the prompt
  tokens served from cache. /v1/completions now reports prompt_tokens.
• ALPHA_SPEED_MODE says when the speed drafter file for the model is missing (a log line and /health
speed_drafter: "missing") and falls back to fast decoding.

Downloads

Package	For	Tested for this release
alphallama-v2-linux-x64-cuda12.tar.zst	Linux x86-64, NVIDIA RTX 20 to 50 series (CUDA 12.8 runtime included). Includes AlphaInfer: the fastest path.	Yes: RTX 3090 (24 GB), Qwen3.8-27B (fast / exact / speed mode), Qwen3.6-35B, Qwen3.8-Flash-Next
alphallama-v2-win-x64-cuda12.zip	Windows x64, NVIDIA RTX 20 / 30 / 40 series, driver 551+	Smoke test (small model, laptop GPU)
alphallama-v2-win-x64-cuda13.zip	Windows x64, NVIDIA including RTX 50 series, driver 580+	Smoke test (small model, laptop GPU)
alphallama-v2-win-x64-vulkan.zip	Windows x64, any Vulkan GPU	Smoke test (small model, laptop GPU)
alphallama-v2-win-x64-cpu.zip	Windows x64, CPU only (AVX2)	Smoke test (small model)
alphallama-v2-win-arm64-cpu.zip	Windows on ARM (Snapdragon X), CPU	Not run
alphallama-v2-macos-arm64.tar.gz	macOS 12+, Apple Silicon (Metal)	Not yet tested
alphallama-v2-macos-x64.tar.gz	macOS 12+, Intel (CPU)	Not yet tested

Each file has a .sha256 next to it; please check it before running (on Windows:
Get-FileHash <file> -Algorithm SHA256).

Quick start:

alphallama-serve -m model.gguf --port 8080      # OpenAI-compatible API on http://127.0.0.1:8080/v1

Notes

• Peak speeds (the AlphaInfer engine, and the decode modes above) need the Linux package on an NVIDIA GPU. The
  Windows and macOS packages run the standard engine, with CUDA, Vulkan or Metal acceleration.
• The Windows binaries are not code-signed yet. Some antivirus products quarantine new unsigned executables
  heuristically on first run (for example Norton "IDP.Generic"), and Windows Smart App Control blocks them; on a PC
  with Smart App Control on, the Windows packages will not start until they are signed.
• The macOS packages are ad-hoc signed: run xattr -dr com.apple.quarantine <folder> after unpacking.

Source: commit f864ef9.

AlphaLlama v2 (2.0.5)

AlphaLlama v2 (2.0.5): fast local inference for large Mixture-of-Experts models on consumer GPUs.

What's new since 2.0.4

• Thinking at the model's own default effort, with room kept for the answer. A thinking request that names no
  reasoning_effort uses the chat template's default again (xhigh on Qwen3.8-Flash-Next), as in 2.0.3.
  ALPHA_DEFAULT_REASONING_EFFORT=low gives 2.0.4's shorter thinking and faster first answer.
• Answer room (on by default; ALPHA_ANSWER_ROOM=0 turns it off). max_tokens counts the thinking too, so a
  thinking request with no reasoning budget of its own and max_tokens of 1024 or more now closes its thinking early
  enough to leave the answer a quarter of max_tokens (at least 512 tokens, at most half). Before, a long think on a
  capped request could use up every token and return no answer. A request's own reasoning_budget still wins.
• max_completion_tokens is accepted when max_tokens is absent (OpenAI clients send it).
• The READY banner names the answer room and its switch.

Output with thinking off, and at temperature 0, is unchanged from 2.0.4 (25 of 25 test prompts identical).

Also in 2.0.4 (if you are coming from 2.0.3): 16 GB cards stay up at default settings, a startup hardware check and
READY banner, a faster first start of validated 35B / 27B models, a working reasoning_effort field, and a 5.9-11.7x
faster BM25 retrieval index by default. See the 2.0.4 release notes.

Downloads

Package	For	Tested for this release
alphallama-v2-linux-x64-cuda12.tar.zst	Linux x86-64, NVIDIA RTX 20 to 50 series (CUDA 12.8 runtime included). Includes AlphaInfer: the fastest path.	Yes: RTX 4080 (16 GB), Qwen3.8-Flash-Next, Qwen3.6-35B, 27B
alphallama-v2-win-x64-cuda12.zip	Windows x64, NVIDIA RTX 20 / 30 / 40 series, driver 551+	Not run
alphallama-v2-win-x64-cuda13.zip	Windows x64, NVIDIA including RTX 50 series, driver 580+	Not run
alphallama-v2-win-x64-vulkan.zip	Windows x64, any Vulkan GPU	Not run
alphallama-v2-win-x64-cpu.zip	Windows x64, CPU only (AVX2)	Not run
alphallama-v2-win-arm64-cpu.zip	Windows on ARM (Snapdragon X), CPU	Not run (see Notes)
alphallama-v2-macos-arm64.tar.gz	macOS 12+, Apple Silicon (Metal)	Not yet tested
alphallama-v2-macos-x64.tar.gz	macOS 12+, Intel (CPU)	Not yet tested

The Windows packages build from the same source as the tested Linux package; the 2.0.5 change is in the shared
server code.

Each file has a .sha256 next to it; please check it before running (on Windows:
Get-FileHash <file> -Algorithm SHA256).

Quick start:

alphallama-serve -m model.gguf --port 8080      # OpenAI-compatible API on http://127.0.0.1:8080/v1

Notes

• Peak speeds (the AlphaInfer engine) need the Linux package on an NVIDIA GPU. The Windows and macOS packages run
  the standard engine.
• The Windows binaries are not code-signed yet. Some antivirus products remove new unsigned executables heuristically
  (for example Norton "IDP.Generic"), and Windows Smart App Control blocks them; on a PC with Smart App Control on,
  the Windows packages will not start until they are signed.
• The macOS packages are ad-hoc signed: run xattr -dr com.apple.quarantine <folder> after unpacking.

Source: commit 2d68ed0.

AlphaLlama v2 (2.0.4)

AlphaLlama v2 (2.0.4): fast local inference for large Mixture-of-Experts models on consumer GPUs.

What's new since 2.0.3

• 16 GB cards stay up. On an RTX 4080-class card, 2.0.3 at its default settings could run out of VRAM within a
  few chat requests. One prefill scratch buffer (AI_Q4X_PF_STREAM2=0) and at most 8 captured-round profiles
  (AI_GRAPH_MAX=8) are now the defaults there. A 62-request soak (thinking on, long replies, 8K prompts) ran with no
  out-of-memory or decode errors.
• Startup says what is happening. A hardware check runs first: a CPU without AVX2/FMA stops with a clear message,
  and an older GPU gets a warning. The model size is shown before loading, and a READY banner shows the URL and how
  thinking is returned.
• Faster first start. A first start of a validated 35B / 27B model no longer hashes the whole file before serving
  (on an RTX 4080 it is ready in 17 s / 8 s instead of 72 s / 47 s); the check runs in the background.
• Faster first answer on Flash-Next. Qwen3.8-Flash-Next thinks at reasoning_effort low unless the request asks
  otherwise, so the first answer token arrives in about 1.3 s. ALPHA_DEFAULT_REASONING_EFFORT changes the effort
  server-wide, and ALPHA_DEFAULT_THINKING=0 turns thinking off.
• reasoning_effort works. String values in chat_template_kwargs no longer return 400, and the top-level OpenAI
  reasoning_effort field is accepted ("none" turns thinking off).
• Retrieval: the inverted BM25 index is on by default: queries are 5.9-11.7x faster with the same ranking quality
  (nDCG@10 unchanged on four BEIR corpora). ALPHA_BM25_INVERTED=0 keeps the forward scan. A corpus ingested over HTTP
  uses the index after POST /api/corpus/finalize; the server warns once if it was never finalized.

Note on exact output. The new 16 GB defaults can change the exact text of some long prompts at temperature: 0
compared with 2.0.3 (1 of 25 test prompts on an RTX 4080). The output is just as valid; with
AI_Q4X_PF_STREAM2=1 AI_GRAPH_MAX=16 it matches 2.0.3 again. 24 and 32 GB cards are unchanged.

Downloads

Package	For	Tested for this release
alphallama-v2-linux-x64-cuda12.tar.zst	Linux x86-64, NVIDIA RTX 20 to 50 series (CUDA 12.8 runtime included). Includes AlphaInfer: the fastest path.	Yes: RTX 4080 (16 GB), Qwen3.8-Flash-Next, Qwen3.6-35B, 27B
alphallama-v2-win-x64-cuda12.zip	Windows x64, NVIDIA RTX 20 / 30 / 40 series, driver 551+	Not run
alphallama-v2-win-x64-cuda13.zip	Windows x64, NVIDIA including RTX 50 series, driver 580+	Not run
alphallama-v2-win-x64-vulkan.zip	Windows x64, any Vulkan GPU	Not run
alphallama-v2-win-x64-cpu.zip	Windows x64, CPU only (AVX2)	Not run
alphallama-v2-win-arm64-cpu.zip	Windows on ARM (Snapdragon X), CPU	Not run (see Notes)
alphallama-v2-macos-arm64.tar.gz	macOS 12+, Apple Silicon (Metal)	Not yet tested
alphallama-v2-macos-x64.tar.gz	macOS 12+, Intel (CPU)	Not yet tested

The Windows packages build from the same source as the tested Linux package; their 2.0.4 changes are startup, API
and retrieval (the 16 GB defaults apply to the AlphaInfer path, which is Linux-only).

Each file has a .sha256 next to it; please check it before running (on Windows:
Get-FileHash <file> -Algorithm SHA256).

Quick start:

alphallama-serve -m model.gguf --port 8080      # OpenAI-compatible API on http://127.0.0.1:8080/v1

Notes

• Peak speeds (the AlphaInfer engine) need the Linux package on an NVIDIA GPU. The Windows and macOS packages run
  the standard engine.
• The Windows binaries are not code-signed yet. Some antivirus products remove new unsigned executables heuristically
  (for example Norton "IDP.Generic"), and Windows Smart App Control blocks them; on a PC with Smart App Control on,
  the Windows packages will not start until they are signed.
• The macOS packages are ad-hoc signed: run xattr -dr com.apple.quarantine <folder> after unpacking.

Source: commit 7c8a789.

AlphaLlama v2 (2.0.3)

AlphaLlama v2 (2.0.3): fast local inference for large Mixture-of-Experts models on consumer GPUs.

This release replaces 2.0.0. It contains every fix made since (2.0.1 and 2.0.2 were not published separately).

What's new since 2.0.0

• Better default sampling. A chat request that sends no temperature now uses the sampling recommended in the
  model file (general.sampling.*; Qwen3.8-Flash-Next: temperature 1.0, top_k 20, top_p 0.95) instead of greedy
  decoding, which could trap thinking models in repetition loops on long agent turns. Fields you send always win;
  ALPHA_DEFAULT_TEMP / _TOP_K / _TOP_P / _MIN_P change the defaults (ALPHA_DEFAULT_TEMP=0 = greedy).
  Requests with temperature: 0 behave exactly as before.
• Tool calls fixed. Tool-call arguments are accepted in any order. Before, a model that wrote them in a different
  order than the tool's schema got its whole reply returned as plain text with no tool_calls.
• Thinking in reasoning_content. The model's thinking now comes back in reasoning_content (streaming and
  non-streaming), also for chat templates that open <think> themselves; it no longer leaks into content.
  ALPHA_REASONING_FORMAT=none keeps it inline.
• 128K context on Flash-Next. -c 131072 now sizes the engine workspace automatically; prompts up to 125K tokens
  were recalled correctly on an RTX 5090 (before, a ~100K-token prompt stopped the server).
• Fixed: a server crash on prompts of 4096×n + 1..8 tokens.
• 16 GB cards: the prefill memory pool is now pre-grown at start, against an out-of-memory seen during prefill
  when little VRAM was left.
• The model id reported by /v1/models is the model file's name (ALPHA_MODEL_ID overrides it).

Downloads

Package	For	Tested for this release
alphallama-v2-linux-x64-cuda12.tar.zst	Linux x86-64, NVIDIA RTX 20 to 50 series (CUDA 12.8 runtime included). Includes AlphaInfer: the fastest path.	Yes: RTX 5090, Qwen3.8-Flash-Next at 128K context
alphallama-v2-win-x64-cuda12.zip	Windows x64, NVIDIA RTX 20 / 30 / 40 series, driver 551+	Not run (same source as CUDA 13)
alphallama-v2-win-x64-cuda13.zip	Windows x64, NVIDIA including RTX 50 series, driver 580+	Yes
alphallama-v2-win-x64-vulkan.zip	Windows x64, any Vulkan GPU	Yes
alphallama-v2-win-x64-cpu.zip	Windows x64, CPU only (AVX2)	Yes
alphallama-v2-win-arm64-cpu.zip	Windows on ARM (Snapdragon X), CPU	Not run (see Notes)
alphallama-v2-macos-arm64.tar.gz	macOS 12+, Apple Silicon (Metal)	Not yet tested
alphallama-v2-macos-x64.tar.gz	macOS 12+, Intel (CPU)	Not yet tested

Each file has a .sha256 next to it; please check it before running (on Windows:
Get-FileHash <file> -Algorithm SHA256).

Quick start:

alphallama-serve -m model.gguf --port 8080      # OpenAI-compatible API on http://127.0.0.1:8080/v1

Notes

• Peak speeds (the AlphaInfer engine) need the Linux package on an NVIDIA GPU. The Windows and macOS packages run
  the standard engine.
• The Windows binaries are not code-signed yet. Some antivirus products remove new unsigned executables heuristically
  (for example Norton "IDP.Generic"), and Windows Smart App Control blocks them; on a PC with Smart App Control on,
  the Windows packages will not start until they are signed. The .sha256 files let you confirm the download is the
  release file.
• The macOS packages are ad-hoc signed: run xattr -dr com.apple.quarantine <folder> after unpacking. Reports are
  welcome.

Source: commit b6cb52c.

AlphaLlama v2 (2.0.0)

AlphaLlama v2 (2.0.0): fast local inference for large Mixture-of-Experts models on consumer GPUs.

Highlights

• Qwen3.8-Flash-Next 1.08-1.18x faster than Strata 0.2.8 on RTX 4080 16 GB, RTX 3090 24 GB and RTX 5090 32 GB.
  Same model file, same prompts, same session, measured on InferArena.
• Bare -m picks the right setup. alphallama-serve -m model.gguf measures the machine once (PCIe and memory
  bandwidth, CPU caches) and applies the configuration measured best for the card's memory class (16 / 24 / 32 GB).
  Every value is logged, and an explicit environment variable always wins.
• Fixed: at temperature 0, Flash-Next could emit stray foreign-script tokens in a few percent of long answers.
  Now it emits none, and plain decoding repeats byte for byte.
• Qwen3.6-35B and Qwen3.8-27B: exact (MTP output identical to plain decoding) by default on validated cards;
  ALPHA_SPEED_MODE=1 selects the faster, non-identical speculative mode.
• License: Apache-2.0 (files from llama.cpp / ggml keep MIT).

Downloads

Package	For
alphallama-v2-linux-x64-cuda12.tar.zst	Linux x86-64, NVIDIA RTX 20 to 50 series (CUDA 12.8 runtime included). Includes AlphaInfer: the fastest path.
alphallama-v2-win-x64-cuda12.zip	Windows x64, NVIDIA RTX 20 / 30 / 40 series, driver 551+
alphallama-v2-win-x64-cuda13.zip	Windows x64, NVIDIA including RTX 50 series, driver 580+
alphallama-v2-win-x64-vulkan.zip	Windows x64, any Vulkan GPU
alphallama-v2-win-x64-cpu.zip	Windows x64, CPU only (AVX2)
alphallama-v2-win-arm64-cpu.zip	Windows on ARM (Snapdragon X), CPU
alphallama-v2-macos-arm64.tar.gz	macOS 12+, Apple Silicon (Metal)
alphallama-v2-macos-x64.tar.gz	macOS 12+, Intel (CPU)

Each file has a .sha256 next to it; please check it before running.

Quick start:

alphallama-serve -m model.gguf --port 8080      # OpenAI-compatible API on http://127.0.0.1:8080/v1

Notes

• Peak speeds (the AlphaInfer engine) need the Linux package on an NVIDIA GPU. The Windows and macOS packages run
  the standard engine.
• The Windows binaries are not code-signed yet. Some antivirus products flag new unsigned executables heuristically
  (for example Norton "IDP.Generic"). The .sha256 files let you confirm the download is the release file.
• The macOS packages are ad-hoc signed: run xattr -dr com.apple.quarantine <folder> after unpacking. The Intel
  package was tested on an Intel Mac (macOS 12.7.6); Apple Silicon is not yet tested; reports are welcome.
• Tested before release: the Linux package on an RTX 3090 (Flash-Next and Qwen3.6-35B), the Windows x64 CPU and
  ARM64 CPU packages, and the macOS Intel package.

Source: commit ffa934c.