AlphaLlama v2 (2.0.4)

AlphaLlama v2 (2.0.4): fast local inference for large Mixture-of-Experts models on consumer GPUs.

What's new since 2.0.3

• 16 GB cards stay up. On an RTX 4080-class card, 2.0.3 at its default settings could run out of VRAM within a
  few chat requests. One prefill scratch buffer (AI_Q4X_PF_STREAM2=0) and at most 8 captured-round profiles
  (AI_GRAPH_MAX=8) are now the defaults there. A 62-request soak (thinking on, long replies, 8K prompts) ran with no
  out-of-memory or decode errors.
• Startup says what is happening. A hardware check runs first: a CPU without AVX2/FMA stops with a clear message,
  and an older GPU gets a warning. The model size is shown before loading, and a READY banner shows the URL and how
  thinking is returned.
• Faster first start. A first start of a validated 35B / 27B model no longer hashes the whole file before serving
  (on an RTX 4080 it is ready in 17 s / 8 s instead of 72 s / 47 s); the check runs in the background.
• Faster first answer on Flash-Next. Qwen3.8-Flash-Next thinks at reasoning_effort low unless the request asks
  otherwise, so the first answer token arrives in about 1.3 s. ALPHA_DEFAULT_REASONING_EFFORT changes the effort
  server-wide, and ALPHA_DEFAULT_THINKING=0 turns thinking off.
• reasoning_effort works. String values in chat_template_kwargs no longer return 400, and the top-level OpenAI
  reasoning_effort field is accepted ("none" turns thinking off).
• Retrieval: the inverted BM25 index is on by default: queries are 5.9-11.7x faster with the same ranking quality
  (nDCG@10 unchanged on four BEIR corpora). ALPHA_BM25_INVERTED=0 keeps the forward scan. A corpus ingested over HTTP
  uses the index after POST /api/corpus/finalize; the server warns once if it was never finalized.

Note on exact output. The new 16 GB defaults can change the exact text of some long prompts at temperature: 0
compared with 2.0.3 (1 of 25 test prompts on an RTX 4080). The output is just as valid; with
AI_Q4X_PF_STREAM2=1 AI_GRAPH_MAX=16 it matches 2.0.3 again. 24 and 32 GB cards are unchanged.

Downloads

Package	For	Tested for this release
alphallama-v2-linux-x64-cuda12.tar.zst	Linux x86-64, NVIDIA RTX 20 to 50 series (CUDA 12.8 runtime included). Includes AlphaInfer: the fastest path.	Yes: RTX 4080 (16 GB), Qwen3.8-Flash-Next, Qwen3.6-35B, 27B
alphallama-v2-win-x64-cuda12.zip	Windows x64, NVIDIA RTX 20 / 30 / 40 series, driver 551+	Not run
alphallama-v2-win-x64-cuda13.zip	Windows x64, NVIDIA including RTX 50 series, driver 580+	Not run
alphallama-v2-win-x64-vulkan.zip	Windows x64, any Vulkan GPU	Not run
alphallama-v2-win-x64-cpu.zip	Windows x64, CPU only (AVX2)	Not run
alphallama-v2-win-arm64-cpu.zip	Windows on ARM (Snapdragon X), CPU	Not run (see Notes)
alphallama-v2-macos-arm64.tar.gz	macOS 12+, Apple Silicon (Metal)	Not yet tested
alphallama-v2-macos-x64.tar.gz	macOS 12+, Intel (CPU)	Not yet tested

The Windows packages build from the same source as the tested Linux package; their 2.0.4 changes are startup, API
and retrieval (the 16 GB defaults apply to the AlphaInfer path, which is Linux-only).

Each file has a .sha256 next to it; please check it before running (on Windows:
Get-FileHash <file> -Algorithm SHA256).

Quick start:

alphallama-serve -m model.gguf --port 8080      # OpenAI-compatible API on http://127.0.0.1:8080/v1

Notes

• Peak speeds (the AlphaInfer engine) need the Linux package on an NVIDIA GPU. The Windows and macOS packages run
  the standard engine.
• The Windows binaries are not code-signed yet. Some antivirus products remove new unsigned executables heuristically
  (for example Norton "IDP.Generic"), and Windows Smart App Control blocks them; on a PC with Smart App Control on,
  the Windows packages will not start until they are signed.
• The macOS packages are ad-hoc signed: run xattr -dr com.apple.quarantine <folder> after unpacking.

Source: commit 7c8a789.