Skip to content

AlphaLlama v2 (2.0.3)

Choose a tag to compare

View all tags
@github-actions github-actions released this 03 Nov 15:44
· 6 commits to main since this release
v2.0.3
b6cb52c

AlphaLlama v2 (2.0.3): fast local inference for large Mixture-of-Experts models on consumer GPUs.

This release replaces 2.0.0. It contains every fix made since (2.0.1 and 2.0.2 were not published separately).

What's new since 2.0.0

  • Better default sampling. A chat request that sends no temperature now uses the sampling recommended in the
    model file (general.sampling.*; Qwen3.8-Flash-Next: temperature 1.0, top_k 20, top_p 0.95) instead of greedy
    decoding, which could trap thinking models in repetition loops on long agent turns. Fields you send always win;
    ALPHA_DEFAULT_TEMP / _TOP_K / _TOP_P / _MIN_P change the defaults (ALPHA_DEFAULT_TEMP=0 = greedy).
    Requests with temperature: 0 behave exactly as before.
  • Tool calls fixed. Tool-call arguments are accepted in any order. Before, a model that wrote them in a different
    order than the tool's schema got its whole reply returned as plain text with no tool_calls.
  • Thinking in reasoning_content. The model's thinking now comes back in reasoning_content (streaming and
    non-streaming), also for chat templates that open <think> themselves; it no longer leaks into content.
    ALPHA_REASONING_FORMAT=none keeps it inline.
  • 128K context on Flash-Next. -c 131072 now sizes the engine workspace automatically; prompts up to 125K tokens
    were recalled correctly on an RTX 5090 (before, a ~100K-token prompt stopped the server).
  • Fixed: a server crash on prompts of 4096×n + 1..8 tokens.
  • 16 GB cards: the prefill memory pool is now pre-grown at start, against an out-of-memory seen during prefill
    when little VRAM was left.
  • The model id reported by /v1/models is the model file's name (ALPHA_MODEL_ID overrides it).

Downloads

Package For Tested for this release
alphallama-v2-linux-x64-cuda12.tar.zst Linux x86-64, NVIDIA RTX 20 to 50 series (CUDA 12.8 runtime included). Includes AlphaInfer: the fastest path. Yes: RTX 5090, Qwen3.8-Flash-Next at 128K context
alphallama-v2-win-x64-cuda12.zip Windows x64, NVIDIA RTX 20 / 30 / 40 series, driver 551+ Not run (same source as CUDA 13)
alphallama-v2-win-x64-cuda13.zip Windows x64, NVIDIA including RTX 50 series, driver 580+ Yes
alphallama-v2-win-x64-vulkan.zip Windows x64, any Vulkan GPU Yes
alphallama-v2-win-x64-cpu.zip Windows x64, CPU only (AVX2) Yes
alphallama-v2-win-arm64-cpu.zip Windows on ARM (Snapdragon X), CPU Not run (see Notes)
alphallama-v2-macos-arm64.tar.gz macOS 12+, Apple Silicon (Metal) Not yet tested
alphallama-v2-macos-x64.tar.gz macOS 12+, Intel (CPU) Not yet tested

Each file has a .sha256 next to it; please check it before running (on Windows:
Get-FileHash <file> -Algorithm SHA256).

Quick start:

alphallama-serve -m model.gguf --port 8080      # OpenAI-compatible API on http://127.0.0.1:8080/v1

Notes

  • Peak speeds (the AlphaInfer engine) need the Linux package on an NVIDIA GPU. The Windows and macOS packages run
    the standard engine.
  • The Windows binaries are not code-signed yet. Some antivirus products remove new unsigned executables heuristically
    (for example Norton "IDP.Generic"), and Windows Smart App Control blocks them; on a PC with Smart App Control on,
    the Windows packages will not start until they are signed. The .sha256 files let you confirm the download is the
    release file.
  • The macOS packages are ad-hoc signed: run xattr -dr com.apple.quarantine <folder> after unpacking. Reports are
    welcome.

Source: commit b6cb52c.

Assets 18
Loading