Skip to content

AlphaLlama v2 (2.0.0)

Choose a tag to compare

View all tags
@AlphaChat-dotcom AlphaChat-dotcom released this 02 Nov 10:50
v2.0.0
ffa934c

AlphaLlama v2 (2.0.0): fast local inference for large Mixture-of-Experts models on consumer GPUs.

Highlights

  • Qwen3.8-Flash-Next 1.08-1.18x faster than Strata 0.2.8 on RTX 4080 16 GB, RTX 3090 24 GB and RTX 5090 32 GB.
    Same model file, same prompts, same session, measured on InferArena.
  • Bare -m picks the right setup. alphallama-serve -m model.gguf measures the machine once (PCIe and memory
    bandwidth, CPU caches) and applies the configuration measured best for the card's memory class (16 / 24 / 32 GB).
    Every value is logged, and an explicit environment variable always wins.
  • Fixed: at temperature 0, Flash-Next could emit stray foreign-script tokens in a few percent of long answers.
    Now it emits none, and plain decoding repeats byte for byte.
  • Qwen3.6-35B and Qwen3.8-27B: exact (MTP output identical to plain decoding) by default on validated cards;
    ALPHA_SPEED_MODE=1 selects the faster, non-identical speculative mode.
  • License: Apache-2.0 (files from llama.cpp / ggml keep MIT).

Downloads

Package For
alphallama-v2-linux-x64-cuda12.tar.zst Linux x86-64, NVIDIA RTX 20 to 50 series (CUDA 12.8 runtime included). Includes AlphaInfer: the fastest path.
alphallama-v2-win-x64-cuda12.zip Windows x64, NVIDIA RTX 20 / 30 / 40 series, driver 551+
alphallama-v2-win-x64-cuda13.zip Windows x64, NVIDIA including RTX 50 series, driver 580+
alphallama-v2-win-x64-vulkan.zip Windows x64, any Vulkan GPU
alphallama-v2-win-x64-cpu.zip Windows x64, CPU only (AVX2)
alphallama-v2-win-arm64-cpu.zip Windows on ARM (Snapdragon X), CPU
alphallama-v2-macos-arm64.tar.gz macOS 12+, Apple Silicon (Metal)
alphallama-v2-macos-x64.tar.gz macOS 12+, Intel (CPU)

Each file has a .sha256 next to it; please check it before running.

Quick start:

alphallama-serve -m model.gguf --port 8080      # OpenAI-compatible API on http://127.0.0.1:8080/v1

Notes

  • Peak speeds (the AlphaInfer engine) need the Linux package on an NVIDIA GPU. The Windows and macOS packages run
    the standard engine.
  • The Windows binaries are not code-signed yet. Some antivirus products flag new unsigned executables heuristically
    (for example Norton "IDP.Generic"). The .sha256 files let you confirm the download is the release file.
  • The macOS packages are ad-hoc signed: run xattr -dr com.apple.quarantine <folder> after unpacking. The Intel
    package was tested on an Intel Mac (macOS 12.7.6); Apple Silicon is not yet tested; reports are welcome.
  • Tested before release: the Linux package on an RTX 3090 (Flash-Next and Qwen3.6-35B), the Windows x64 CPU and
    ARM64 CPU packages, and the macOS Intel package.

Source: commit ffa934c.

Assets 18
Loading