llamAmpere v0.5.5

New: --eval-cache DIR: an opt-in, read-only prompt cache on disk for eval and benchmark prompts. Long prompts are read once, saved, and reloaded in seconds after that.

• About 6-7x faster long-context evals with everything on. Eval cache + MTP drafting + parallel slots stack. Approximate typical speed-up per 100K eval prompt with eval cache + MTP + --parallel 2: about 6-7x for a 2,000-token answer, about 17x for a 500-token answer, about 3x for a 10,000-token reasoning answer. This is arithmetic from measured parts, not one end-to-end run; see the release notes.
• About 64x less prefill at 100K. Swift 1.5 (Qwen3.8 27B) IQ4_XS-M, SJ-KVaRN 4/4, ~102K-token prompts, RTX 3090 Ti at 350 W: 1.6-1.7 s to load vs 104.6-105.1 s to read fresh. Time to first answer drops from 105.7 s to 4.4 s.
• Hours back per eval run. About 101 s saved per 100K prompt is about 2.8 hours per 100-prompt pass, and every re-run saves it again.
• Same answers. At temperature 0 the output text is byte-for-byte identical to a fresh prefill.
• Ready-made folders (AA-LCR, LongCodeQA, MRCR, RULER, LongBench v2): https://huggingface.co/datasets/jakeatx/llamampere-evalcache
• Read-only at eval time (no SSD wear). A folder that doesn't match the model or cache settings is refused at startup.
• Bug fixes.

Full details: docs/llamampere-v0.5.5/RELEASE_NOTES.md

llamAmpere v0.5.4

A crash-fix release on top of v0.5.3. No new features or flags, and no speed change. If you run the server with --parallel 3 or more, update to this one.

• The server no longer stops with failed to remove sequence N when it serves several requests at once with the MTP drafter on (--parallel 3 or more, Qwen3.8 models such as Swift and ATX). It happened when the server moved a waiting request to another memory slot and dropped the history the drafter needs to undo a rejected guess. The prompt cache was not the cause, so --cache-ram 0 was not a workaround. In our test with Swift 1.5 at --parallel 4 and --parallel 8, v0.5.3 aborted in 4 of 4 runs; v0.5.4 finished all 32 requests (5,120 tokens each) with no abort, peak VRAM 22,850 MiB.
• With --parallel 1, output is unchanged: at temperature 0, Swift 1.5 with tq5_0 K / turbo4 V writes the same 5,000 tokens as v0.5.3. No GPU kernels changed.
• No speed change: at --parallel 8, runs in the order v0.5.3, v0.5.4, v0.5.3 gave 19.42, 19.32 and 19.36 tok/s per request. Details in the release notes.
• The v0.5.3 known issue (--parallel 8 stopping at startup) is fixed by this release.

Build and run commands: README.

llamAmpere v0.5.3

A bug-fix release on top of v0.5.2. No new features or flags. If you build from source, update to this one.

• SJ-KVaRN: a conversation resumed from the prompt cache (RAM, or disk after a restart) now gives the same answer as reading it fresh. In our restore test at temperature 0, differing turns went from 6 of 24 to 0 of 24 at 4/4 and from 4 of 24 to 0 of 24 at 3/3.
• The disk prompt cache now works when --mmproj is loaded (issue #35). Text-only prompts are saved and restored; prompts with images stay in RAM only.
• The disk cache now says when it stops writing because the disk is too full, and --cache-disk-min-free also accepts NG and N%.
• Startup no longer prints a false "draft vocabulary shortlist not applied" warning; the shortlist was always used (issue #23).
• Builds for RTX 50-series (-DCMAKE_CUDA_ARCHITECTURES=120, CUDA 12.8 or newer). The RTX 3090 build is unchanged. 40- and 50-series cards are not tuned for.
• With tq5_0 K / turbo4 V, output and 100K decode speed are unchanged from v0.5.2. With SJ-KVaRN, a fresh read now splits long prompts at 1,024-token boundaries, so temperature-0 answers are not token-for-token the same as v0.5.2's; the measured quality difference was not significant. Details in the release notes.
• Known issue at release, fixed in v0.5.4: with --parallel 3 or more and the MTP drafter on, the server could stop with failed to remove sequence N.

Build and run commands: README.

llamAmpere v0.5.2

A small point release on top of v0.5.1. If you build from source, update to this one.

• Several prefill and SJ-KVaRN fixes were added, improving reliability and speed.
• The RAM prompt cache keeps the newest 4 checkpoints plus one per doubling; --cache-ram-checkpoints 0 keeps all.
• Optional per-model SJ-KVaRN transform files (--sjkvarn-transform FILE|URL), off by default. Files for ATX-4-XS, Swift 1.5 IQ4_XS-M and the 12 GB Swift 1.5 model are on Hugging Face next to each model. The measured KL per model and an example command are in the release notes.
• --parallel 1 output and 100K decode speed are unchanged from v0.5.1.

Build and run commands: README.

llamAmpere v0.5.1

SJ-KVaRN with parallel slots

v0.5 refused to start SJ-KVaRN with more than one sequence ("SJ-KVaRN rejects n_seq_max = 2"). This also hit the
server's default --parallel (auto), which resolves to 4 slots.

In v0.5.1:

• Each slot gets its own SJ-KVaRN stream: sink, staging ring and sealed body.
• --kv-unified shares one paged record pool between slots, so any slot can use the whole context.
• Slots stay isolated. 2- and 4-slot runs differ from single-slot runs only as much as plain-cache batching does
  (per-position KL about 1e-4 to 1e-3, the same as the plain-cache control).
• Prompt caching and slot save/restore work across slots, including restoring a saved slot into another slot.
• When the shared pool is full, the server returns an HTTP error instead of crashing.
• --parallel 1 gives token-identical output to v0.5 at unchanged speed: 91.14 vs 91.59 tok/s at 100K depth,
  within noise.

Tested

Swift 1.5 IQ4_XS-M with the MTP drafter, at SJ-KVaRN 4/4 and 3/3t, on one RTX 3090 Ti:

Setup	Context	Peak VRAM
4 slots, 3/3t	196,608	23,034 MiB
4 slots, 4/4 --kv-unified	131,072	22,750 MiB

Two slots decode about 110-119 tok/s together, against 103-107 tok/s for one slot alone (temperature 1, MTP).

Recommended settings

For a single user, keep --parallel 1 as in the v0.5 recommended commands:
each extra slot costs one staging ring of VRAM. For several concurrent users, use --parallel N, and add
--kv-unified when slots should share the context.

llamAmpere v0.5

llamAmpere is a llama.cpp fork tuned for the RTX 3090 / 3090 Ti (Ampere, SM86), running Qwen3.8-27B with its MTP draft
head.

What's new in v0.5

• SJ-KVaRN, a new compressed KV cache, in 4/4, 3/3t and 3/2t configurations. It builds on the KVarN method
  (huawei-csl/KVarN).
  • At 4/4 it has 41% lower KL than v0.4's tq5_0/turbo4 cache for 2% more cache memory.
  • At 262K full depth, 4/4 peaks 287 MiB lower than v0.4's tq5_0/turbo4 (21,924 MiB), and 3/3t peaks at 20,941 MiB.
• Faster decode at 100K. On our task benchmark at 100K depth:
  • v0.5 tq5_0/turbo4: +8.41% ± 5.76% over v0.4;
  • v0.5 SJ-KVaRN 4/4: +6.77% ± 2.68% over v0.4;
  • weighted coding/agentic/RAG, 2 seeds.
• Prompt cache on by default. It has a RAM tier sized from host memory and a 16 GiB disk tier, and works with every
  KV type, SJ-KVaRN included. A resumed conversation skips the full re-prefill.
• 12 GB cards (RTX 3060 12 GB).
  • A 2.3 bpw Swift 1.5 model with SJ-KVaRN 3/3t and the MTP drafter.
  • 204,800 tokens of context in about 11 GB.
  • 77.71% on LiveCodeBench v6, which is 86% of BF16.
• Adaptive draft depth 3-4 is now the default.
• Faster MTP verify decode on EXL3 models.
• A catch-up to upstream llama.cpp and TheTom's TurboQuant fork.
• Coming in v0.5.1: a Swift 1.5 EXL3 4.0 bpw model.
• Read more: the SJ-KVaRN paper and the standalone codec, JakeATX/sj-kvarn-codec.

Full notes, including what we tried and did not ship: docs/llamampere-v0.5/RELEASE_NOTES.md

Recommended settings for 24 GB cards (RTX 3090 / 3090 Ti)

Needs Linux, an NVIDIA card with 24 GB, the CUDA toolkit (tested with 12.4), CMake, git and a C++ compiler. The model
download is 15.6 GB.

git clone -b v0.5 https://github.com/JakeATX/llamAmpere.git
cd llamAmpere
cmake -S . -B build-sm86 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DLLAMA_BUILD_BORINGSSL=ON
cmake --build build-sm86 -j8 --target llama-server

Fastest decode (tq5_0 K / turbo4 V):

./build-sm86/bin/llama-server -hf jakeatx/ATX-Swift-1.5-Qwen3.8-27B-Uncensored-IQ4_XS-M-GGUF \
  -c 262144 -ngl 99 -fa on -ctk tq5_0 -ctv turbo4 -b 4096 -ub 1024 --parallel 1

Lowest KL at about the same speed (SJ-KVaRN 4/4):

./build-sm86/bin/llama-server -hf jakeatx/ATX-Swift-1.5-Qwen3.8-27B-Uncensored-IQ4_XS-M-GGUF \
  -c 262144 -ngl 99 -fa on -ctk sjkvarn4 -ctv sjkvarn4 -b 4096 -ub 1024 --parallel 1

For a smaller cache, use -ctk sjkvarn3 -ctv sjkvarn3 --sjkvarn-body-type auto (3/3t) or
-ctk sjkvarn3 -ctv sjkvarn2 --sjkvarn-body-type auto (3/2t).

The server listens on http://127.0.0.1:8080 (OpenAI-compatible API). The MTP drafter and the prompt cache are on by
default.

llamAmpere v0.4

What's new in v0.4

Speed

• Verify kernels for 5 to 8 tokens per pass, plus an Ampere routing table that picks the best kernel per weight type
  and width.
• MTP draft depth 4 by default. Adaptive depth 3-4 is available as an option.
• The MTP drafter turns on by itself for Qwen3.8 GGUFs that carry the MTP head, so no drafter flags are needed.
• The drafter's KV cache now uses the same cache types as the main model.
• A built-in draft vocabulary shortlist (--spec-draft-vocab-map auto), now on EXL3 too.
• The fused attention tile for q8_0 keys now covers 6 to 8 tokens per pass.
• GDN recurrent-state copies now run as one copy kernel instead of memcpy nodes in the CUDA graph.
• The GDN kernel reads the recurrent state in place instead of gathering a copy every round.
• Fewer copies per decode round: an activation cache for the matrix-vector kernels, a strided copy folded into the
  next kernel, a skipped identity gather and logits pad, and a better launch shape for short concatenations.
• The IQ3 and IQ2 codebook grids are staged in shared memory by default on SM86.
• EXL3 weight-major tensor-core kernels and a fused FFN bridge, both on by default.
• New prefill tiles and a verify-width kernel for PTQ1_0 ternary weights (Ternary Bonsai 2).

KV cache

• A new 5-bit key cache type, turbo5 (tq5_0), with a sign-magnitude encoding.
• Fused attention for turbo4 values paired with turbo5, turbo6 or q8_0 keys, with staged key and value tiles.
• One flat loader for turbo4 keys and values.
• One naming family by bit width, turbo2 to turbo6, with the tq spellings accepted in every cache-type flag.
• turbo2 is enforced as values-only.
• Unsupported cache types on Metal, Vulkan and SYCL now fail with a clear error message.

Long context and memory

• A 262,144-token context on one 24 GB card, with IQ4_XS and with EXL3 4.0 bpw.
• The prefill attention workspace is capped at 256 MiB by default, and prefill runs in groups.
• The attention scratch buffer is sized for the kernel that actually runs.
• -fit accounting fixes: the recurrent state is no longer counted twice, and the adaptive drafter is now budgeted. A
  launch with only -m now gets a much larger context.
• An n-gram drafter for cards where the MTP head does not fit, built on a new recurrent-state snapshot ring
  (--spec-n-rs-seq).

Fixes and maintenance

• tq4_1s weights: fixed NaN from views of weights converted at load time, and the copy abort.
• EXL3: output widths that are not a multiple of 128 no longer abort, and the half-precision accumulation bound is
  corrected.
• HIP: fixed tq5_0 sign packing on 64-wide AMD waves.
• A stress check of the server's cache paths: prompt reuse, branching, restore, cancel, regenerate and slot recycling.
• A catch-up to upstream llama.cpp.

Recommended settings for 24 GB cards (RTX 3090 / 3090 Ti)

Needs Linux, an NVIDIA card with 24 GB (RTX 3090 / 3090 Ti), the CUDA toolkit (tested with 12.4), CMake, git and a C++
compiler. Paste into a terminal; the model download is 15.6 GB.

git clone -b v0.4 https://github.com/JakeATX/llamAmpere.git
cd llamAmpere
cmake -S . -B build-sm86 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build-sm86 -j8 --target llama-server
./build-sm86/bin/llama-server \
  --hf-repo jakeatx/ATX-Swift-1.5-Qwen3.8-27B-Uncensored-IQ4_XS-M-GGUF \
  --hf-file ATX-Swift-1.5-Qwen3.8-27B-Uncensored-IQ4_XS-M.gguf -c 262144 \
  -ngl 99 -fa on -ctk turbo5 -ctv turbo4 -b 4096 -ub 1024 --parallel 1

The server listens on http://127.0.0.1:8080 (OpenAI-compatible API). The MTP drafter (draft depth 4), the vocabulary
shortlist and the drafter's cache types are on by default, so no drafter flags are needed. Tested exactly as written from
a fresh clone of v0.4: 262,144-token context, 67.8 tok/s after a 250,000-token prompt (5,120 generated), peak
22,346 MiB on an RTX 3090 Ti.