Skip to content

AlphaLlama v2 (2.0.5)

Choose a tag to compare

View all tags
@github-actions github-actions released this 09 Nov 15:38
· 4 commits to main since this release
v2.0.5
2d68ed0

AlphaLlama v2 (2.0.5): fast local inference for large Mixture-of-Experts models on consumer GPUs.

What's new since 2.0.4

  • Thinking at the model's own default effort, with room kept for the answer. A thinking request that names no
    reasoning_effort uses the chat template's default again (xhigh on Qwen3.8-Flash-Next), as in 2.0.3.
    ALPHA_DEFAULT_REASONING_EFFORT=low gives 2.0.4's shorter thinking and faster first answer.
  • Answer room (on by default; ALPHA_ANSWER_ROOM=0 turns it off). max_tokens counts the thinking too, so a
    thinking request with no reasoning budget of its own and max_tokens of 1024 or more now closes its thinking early
    enough to leave the answer a quarter of max_tokens (at least 512 tokens, at most half). Before, a long think on a
    capped request could use up every token and return no answer. A request's own reasoning_budget still wins.
  • max_completion_tokens is accepted when max_tokens is absent (OpenAI clients send it).
  • The READY banner names the answer room and its switch.

Output with thinking off, and at temperature 0, is unchanged from 2.0.4 (25 of 25 test prompts identical).

Also in 2.0.4 (if you are coming from 2.0.3): 16 GB cards stay up at default settings, a startup hardware check and
READY banner, a faster first start of validated 35B / 27B models, a working reasoning_effort field, and a 5.9-11.7x
faster BM25 retrieval index by default. See the 2.0.4 release notes.

Downloads

Package For Tested for this release
alphallama-v2-linux-x64-cuda12.tar.zst Linux x86-64, NVIDIA RTX 20 to 50 series (CUDA 12.8 runtime included). Includes AlphaInfer: the fastest path. Yes: RTX 4080 (16 GB), Qwen3.8-Flash-Next, Qwen3.6-35B, 27B
alphallama-v2-win-x64-cuda12.zip Windows x64, NVIDIA RTX 20 / 30 / 40 series, driver 551+ Not run
alphallama-v2-win-x64-cuda13.zip Windows x64, NVIDIA including RTX 50 series, driver 580+ Not run
alphallama-v2-win-x64-vulkan.zip Windows x64, any Vulkan GPU Not run
alphallama-v2-win-x64-cpu.zip Windows x64, CPU only (AVX2) Not run
alphallama-v2-win-arm64-cpu.zip Windows on ARM (Snapdragon X), CPU Not run (see Notes)
alphallama-v2-macos-arm64.tar.gz macOS 12+, Apple Silicon (Metal) Not yet tested
alphallama-v2-macos-x64.tar.gz macOS 12+, Intel (CPU) Not yet tested

The Windows packages build from the same source as the tested Linux package; the 2.0.5 change is in the shared
server code.

Each file has a .sha256 next to it; please check it before running (on Windows:
Get-FileHash <file> -Algorithm SHA256).

Quick start:

alphallama-serve -m model.gguf --port 8080      # OpenAI-compatible API on http://127.0.0.1:8080/v1

Notes

  • Peak speeds (the AlphaInfer engine) need the Linux package on an NVIDIA GPU. The Windows and macOS packages run
    the standard engine.
  • The Windows binaries are not code-signed yet. Some antivirus products remove new unsigned executables heuristically
    (for example Norton "IDP.Generic"), and Windows Smart App Control blocks them; on a PC with Smart App Control on,
    the Windows packages will not start until they are signed.
  • The macOS packages are ad-hoc signed: run xattr -dr com.apple.quarantine <folder> after unpacking.

Source: commit 2d68ed0.

Assets 18
Loading