Repository navigation
Changelog
- Updated the llama.cpp base from
d327bf1f0to3c809f443and ggml to 0.28.0. Inherited changes include DeepSeek V4.1, LiquidAI LFM3/d1 multimodal models, Limite 1B, Mistral Large 4 and MiniCPM-OCR-Jev; FP8 weights, batched tensor transfers, no-allocation memory measurement, realtime conversations/tool calls, deferred tools, router memory scheduling, conversation compaction and search. Also includes CUDA fusion, CDNA sparse attention, SYCL quantized-cache FlashAttention, OpenCL quant/indexer support, Metal/Hexagon updates, parser/DRY limits, and conversion/server/Web UI fixes. - Added opt-in SINQ preconditioning for original floating-point Qwen3.5 models through
llama-quantize --sinq-config FILE, using existing IQ/Q codecs and CPU/CUDA kernels. Per-tensor controls cover strip widths, iterations, strength, tensor/layer/role overrides, imatrix compensation and JSON diagnostics. Direct encoding also offers optional scale refinement and imatrix-exponent controls, defaulting to off and 2 respectively. Embeddings, output heads and auxiliary projections retain ordinary quantization. See the SINQ guide for supported codecs, widths and settings. - Added an offline SINQ optimizer for mixed stock/SINQ/AWQ candidates, with verified activation export, locked assignments, local/blockwise/Fisher scoring and bounded groupwise KLD selection. It supports CPU/CUDA evaluation, source-exact FP32/BF16 advanced scoring, artifact/input locks, generation/selection resume and explicit resource limits. Reconstruction and selection scores do not guarantee held-out quality gains.
- Added validated SINQ GGUF metadata and FP16 companions for CPU and qualified CUDA inference, including tensor parallelism and owned Qwen3.5 MTP drafts. Cached scales and scaled inputs reduce runtime overhead; LoRA retains original inputs. Unsupported backends, row splits, Hadamard combinations and conditioned-source requantization/copying are rejected. These GGUFs require BeeLlama: other engines may ignore companions and produce incorrect output.
- Added
llama-perplexity --ppl-fulland--kld-token-outputto score all N-1 corpus targets, including remainders/context boundaries, with per-context resets and per-token NLL/top-1 TSV. BF16 comparisons measure full-vocabularyKL(P_BF16 || P_candidate); scoring overlaps decoding and honors thread limits. Measure the compact-reference floor with a matched BF16 self-comparison, keeping context, batch, ubatch and runtime settings matched. See the argument reference. - Integrated FP8 GGUF support without changing Bee's low-bit cache/state or Prism IDs: FP8 uses wire ID 43 and Q6_0 uses private wire ID 144. Fixed CUDA Q4_1-by-Q8_1 scale products to use F32 instead of F16-rounded multiplication, reducing cancellation-amplified error. Added format, ABI and roundtrip regressions.
- Adapted KVarN and precision-tail allocation/accounting and exact fitting to upstream's model-owned no-allocation mode, including ISWA subcaches. Host checkpoints batch irregular slices while preserving strided reads, serialized layout and transactional restore semantics.
- Fixed Qwen4Exp multi-stream QSA masks, pooled-indexer shapes and shared MTP weight borrowing with KVarN/exact tails. Dense legacy drafts skip unused indexer uploads, and conversion accepts
model.mtp-prefixed HyperConnection mixers while preserving standalone-head names and PLE filtering. Added decode, conversion and draft-replay regressions. - Improved KVarN backend dispatch. CUDA selects split/vector geometry from device resources and workload rather than fixed context thresholds, retaining explicit overrides. HIP/ROCm gains RDNA3-compatible headwide stores, aligned-word unpacking, bounded windowed prefill for qualified batches, earlier causal tile skipping and compatible graph replay (PR #197). Vulkan shares topology between attention/scratch planning, selects parallel stores and WHT from available work/device capabilities, and preserves serial ordering for duplicate destinations. Vulkan MoE also avoids unnecessary skipped-row rescans. See the GPU dispatch guide for policies and validation limits.
macOS:
Linux:
- Ubuntu x64 CPU
- Ubuntu arm64 CPU
- Ubuntu x64 CUDA 12.4
- Ubuntu x64 CUDA 13.3
- Ubuntu x64 Vulkan
- Ubuntu x64 ROCm 7.2
- Ubuntu x64 SYCL
Windows:
- Windows x64 CPU
- Windows x64 Vulkan
- Windows x64 SYCL
- Windows x64 CUDA 12.4 - DLLs
- Windows x64 CUDA 13.3 - DLLs
- Windows x64 HIP
Docker:
- CPU:
docker pull ghcr.io/anbeeld/beellama.cpp:server-cpu-v0.5.0 - CUDA:
docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda-v0.5.0 - CUDA 12:
docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda12-v0.5.0 - CUDA 13.3:
docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.5.0 - ROCm:
docker pull ghcr.io/anbeeld/beellama.cpp:server-rocm-v0.5.0 - Vulkan:
docker pull ghcr.io/anbeeld/beellama.cpp:server-vulkan-v0.5.0 - SYCL:
docker pull ghcr.io/anbeeld/beellama.cpp:server-sycl-v0.5.0