Skip to content

v0.5.0

Latest

Choose a tag to compare

@github-actions github-actions released this 18 Nov 16:40
Changelog
  • Updated the llama.cpp base from d327bf1f0 to 3c809f443 and ggml to 0.28.0. Inherited changes include DeepSeek V4.1, LiquidAI LFM3/d1 multimodal models, Limite 1B, Mistral Large 4 and MiniCPM-OCR-Jev; FP8 weights, batched tensor transfers, no-allocation memory measurement, realtime conversations/tool calls, deferred tools, router memory scheduling, conversation compaction and search. Also includes CUDA fusion, CDNA sparse attention, SYCL quantized-cache FlashAttention, OpenCL quant/indexer support, Metal/Hexagon updates, parser/DRY limits, and conversion/server/Web UI fixes.
  • Added opt-in SINQ preconditioning for original floating-point Qwen3.5 models through llama-quantize --sinq-config FILE, using existing IQ/Q codecs and CPU/CUDA kernels. Per-tensor controls cover strip widths, iterations, strength, tensor/layer/role overrides, imatrix compensation and JSON diagnostics. Direct encoding also offers optional scale refinement and imatrix-exponent controls, defaulting to off and 2 respectively. Embeddings, output heads and auxiliary projections retain ordinary quantization. See the SINQ guide for supported codecs, widths and settings.
  • Added an offline SINQ optimizer for mixed stock/SINQ/AWQ candidates, with verified activation export, locked assignments, local/blockwise/Fisher scoring and bounded groupwise KLD selection. It supports CPU/CUDA evaluation, source-exact FP32/BF16 advanced scoring, artifact/input locks, generation/selection resume and explicit resource limits. Reconstruction and selection scores do not guarantee held-out quality gains.
  • Added validated SINQ GGUF metadata and FP16 companions for CPU and qualified CUDA inference, including tensor parallelism and owned Qwen3.5 MTP drafts. Cached scales and scaled inputs reduce runtime overhead; LoRA retains original inputs. Unsupported backends, row splits, Hadamard combinations and conditioned-source requantization/copying are rejected. These GGUFs require BeeLlama: other engines may ignore companions and produce incorrect output.
  • Added llama-perplexity --ppl-full and --kld-token-output to score all N-1 corpus targets, including remainders/context boundaries, with per-context resets and per-token NLL/top-1 TSV. BF16 comparisons measure full-vocabulary KL(P_BF16 || P_candidate); scoring overlaps decoding and honors thread limits. Measure the compact-reference floor with a matched BF16 self-comparison, keeping context, batch, ubatch and runtime settings matched. See the argument reference.
  • Integrated FP8 GGUF support without changing Bee's low-bit cache/state or Prism IDs: FP8 uses wire ID 43 and Q6_0 uses private wire ID 144. Fixed CUDA Q4_1-by-Q8_1 scale products to use F32 instead of F16-rounded multiplication, reducing cancellation-amplified error. Added format, ABI and roundtrip regressions.
  • Adapted KVarN and precision-tail allocation/accounting and exact fitting to upstream's model-owned no-allocation mode, including ISWA subcaches. Host checkpoints batch irregular slices while preserving strided reads, serialized layout and transactional restore semantics.
  • Fixed Qwen4Exp multi-stream QSA masks, pooled-indexer shapes and shared MTP weight borrowing with KVarN/exact tails. Dense legacy drafts skip unused indexer uploads, and conversion accepts model.mtp-prefixed HyperConnection mixers while preserving standalone-head names and PLE filtering. Added decode, conversion and draft-replay regressions.
  • Improved KVarN backend dispatch. CUDA selects split/vector geometry from device resources and workload rather than fixed context thresholds, retaining explicit overrides. HIP/ROCm gains RDNA3-compatible headwide stores, aligned-word unpacking, bounded windowed prefill for qualified batches, earlier causal tile skipping and compatible graph replay (PR #197). Vulkan shares topology between attention/scratch planning, selects parallel stores and WHT from available work/device capabilities, and preserves serial ordering for duplicate destinations. Vulkan MoE also avoids unnecessary skipped-row rescans. See the GPU dispatch guide for policies and validation limits.

macOS:

Linux:

Windows:

Docker:

  • CPU: docker pull ghcr.io/anbeeld/beellama.cpp:server-cpu-v0.5.0
  • CUDA: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda-v0.5.0
  • CUDA 12: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda12-v0.5.0
  • CUDA 13.3: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.5.0
  • ROCm: docker pull ghcr.io/anbeeld/beellama.cpp:server-rocm-v0.5.0
  • Vulkan: docker pull ghcr.io/anbeeld/beellama.cpp:server-vulkan-v0.5.0
  • SYCL: docker pull ghcr.io/anbeeld/beellama.cpp:server-sycl-v0.5.0

Browse all container images