llama.cpp single 3090 — Qwen3.6-27B + MTP at 60 t/s code, matches vLLM dual aider on half the hardware (v0.8.3)

[noonghunna]

📢 Update (2026-05-22) — superseded. Single-card llama.cpp now ships a 200K max-safe default (262K boots but walls ~125K under fill — see #197; the compose tree also collapsed 3 → 2), and we since ran a matched ik_llama.cpp comparison. Current numbers + methodology are in the new post: What chasing single-GPU efficiency with ik_llama taught us about our own llama.cpp stack.

---

llama.cpp single 3090 — Qwen3.6-27B + MTP at 60 t/s code, matches vLLM dual aider on half the hardware (v0.8.3)

TL;DR: v0.8.3 shipped two new llama.cpp single-card profiles that close a long-standing gap on this stack: getting near-vLLM throughput on a single RTX 3090 without losing context-length to fit. The big surprise: llamacpp/mtp matches the dual-3090 vLLM bf16 aider score (17/30 = 56.7%) exactly — half the hardware, same coding outcome — and its boundary stress matrix walks past the previously-documented "Cliff 2" at 50–60K cleanly. Numbers below.

Substrate

• llama.cpp mainline build 9235 (d14ce3dab) — the merge commit for MTP PR #22673
• Single RTX 3090, sm_86, PCIe-only, 230W
• Qwen3.6-27B Q4_K_M from unsloth/Qwen3.6-27B-MTP-GGUF (the MTP-bundled GGUF)
• All legs run via scripts/rebench-full.sh (bench + verify-stress + quality-full + soak + aider-polyglot-30); summary in BENCHMARKS.md

The three profiles

Profile	Context	Vision	MTP	When to use
llamacpp/default	262K	✓	—	Long-context max — RAG, document pipelines, anything that needs the full ctx
llamacpp/mtp	131K	—	✓	IDE-agent / coding / chat — best throughput, MTP-accelerated
llamacpp/mtp-vision	49K	✓	✓	Multimodal coding / chat — first single-card MTP + vision combo on this stack

All three run on the local llama-cpp:local image (built by scripts/setup.sh, not pulled from Docker Hub) — they're immune to the vLLM nightly Docker Hub purges that hit some users this week (#165, #167).

Headline numbers

Profile	narr TPS	code TPS	quality 8-pack	aider-polyglot-30	verify-stress
llamacpp/mtp (Q4_K_M, 131K, MTP n=2, q4_0 KV)	51.28	59.72	102 / 150 (68 %)	17 / 30 (56.7 %)	7 / 7
llamacpp/mtp-vision (Q4_K_M, 49K, MTP+vision)	56.52	66.17	(capability claim — bench + multimodal probe ✓)	(n/a — Aider doesn't use vision)	7 / 7
llamacpp/default (Q3_K_XL, 262K)	21.22	20.79	(cliff-immune long-context anchor)	(n/a — chat path, not coding)	7 / 7

Comparison: llamacpp/mtp vs the vLLM dual-3090 configs (#119):

Compose	Hardware	Quality 8-pack	Aider-polyglot-30
Qwen vLLM bf16 dual	2× 3090	86 / 150 (57 %)	17 / 30 (56.7 %) ← matches
Qwen vLLM INT8 dual	2× 3090	96 / 150 (64 %)	14 / 30 (46.7 %)
Qwen vLLM TQ3 dual + Genesis	2× 3090	88 / 150 (59 %)	13 / 30 (43.3 %)
llamacpp/mtp (us)	1× 3090	102 / 150 (68 %)	17 / 30 (56.7 %)

The quality 8-pack is +6 to +16 pp ahead of every vLLM dual config we've benched, and aider-polyglot ties the best bf16-dual leg. We did not expect Q4_K_M GGUF to lead quality across the matrix — it does.

Cliff 2 was config-driven, not architectural

The single-card stack has had a documented "Cliff 2" at ~50–60K accumulated context — a DeltaNet GDN forward path that OOMs prefill on Qwen3.6-27B. The framing in our own docs/CLIFFS.md had been "architectural, single-card Qwen3.6-27B can't go past ~50K reliably." That turned out to be too strong.

At -ub 1024 + 131K KV + MTP n=2 + --reasoning off, the cliff goes away:

• verify-stress's long-context rungs include independent secret recall at 58K and 91K — both clean on a single 3090
• The per-pass activation peak at -ub 2048 was the actual bound; halving it (-ub 1024) + the larger 131K KV pool walks past it
• This doesn't fix the vLLM-side Cliff 2 (different code path, different attention impl) — CLIFFS.md updated 2026-05-19 to clarify the finding is llama.cpp-specific

Hat-tip to @ampersandru (#152) — their cross-engine ik_llama.cpp gauntlet was the forcing function that pushed us to retest at higher ctx + smaller -ub. Their full config decomposition is also queued as a mainline MTP retune A/B (n=4 + draft-p-min 0.0) on the queue for the next bench cycle.

How to run

# default — 262K + vision, cliff-immune
bash scripts/switch.sh llamacpp/default

# MTP — IDE-agent / coding / chat, ~60 t/s code
bash scripts/switch.sh llamacpp/mtp

# MTP + vision — single-card multimodal
bash scripts/switch.sh llamacpp/mtp-vision

All three serve on the OpenAI-compatible endpoint at http://localhost:<port>/v1/chat/completions and accept all standard sampling / tool-use parameters. Built locally via scripts/setup.sh; no Docker Hub dependency.

Updates since posting

2026-05-20 — speed vs context for llamacpp/mtp-vision (@JensJN's question): the shipped 49K context ceiling on mtp-vision is the speed-optimal point, not the only safe point. Smaller -ub (512 instead of the shipped 1024) trades ~10% TPS for ~4× more context on a single 3090, with full cliff coverage (verify-stress 7/7 incl. 91K needle):

Config	ctx	narr TPS	verify-stress
shipped ub=1024	49K	56.5	✓
override ub=512 CTX=196608	192K	50.9	✓ 7/7

Use it via:

UBATCH_SIZE=512 CTX_SIZE=196608 bash scripts/switch.sh llamacpp/mtp-vision

Documented in models/qwen3.6-27b/llama-cpp/README.md (commit 1b2a76c). Defaults unchanged; the lever is yours to dial.

2026-05-20 — llama-cpp:local image bug fix (@zemaphore's report): v0.8.3 composes referenced an orphan image that didn't exist for fresh-clone users. Fixed in commits c3e7c7e → 4a53eda → 78556f8 — composes now use the rolling upstream ghcr.io/ggml-org/llama.cpp:server-cuda image (no patches, no pin), and report.sh captures image digest + OCI labels for reproducible bug reports.

What's next

• MTP retune (queued) — A/B n=4 + --spec-draft-p-min 0.0 vs the shipped n=2 defaults; ampersandru's data suggests headroom here, likely lands within the week
• ik_llama.cpp eval (exploratory) — bench --merge-qkv + --merge-up-gate-experts + -khad + q8_0 KV combo to scope fork-only kernel-fusion residual gains
• Cliff 2 vLLM-side — still real on vLLM; tracked separately in docs/CLIFFS.md. The llama.cpp finding doesn't transfer

If you're running Qwen3.6-27B on a single 3090 — especially with IDE-agent / coding workloads — try llamacpp/mtp and report back. Particularly interested in cross-rig confirmations of the >60K context behavior (the verify-stress numbers are from one rig; a second confirmation would harden the "config-driven, not architectural" finding).

[zemaphore]

Hey noonghunna,

First up a big thank you for the relentless pursuit of 3090 greatness, correctness and speed. I wanted to look into your tuning of llama.cpp and see the performance for myself. Sadly your mentioned method of building the container still seems to be work in progress? I was not able to find a code-path in scripts/setup.sh to get a local docker image build going. I'll try and write my own Dockerfile for it and solve it, but just in case you want to clean up that part of your documentation.

Regards
zema

[noonghunna]

Hey @zemaphore — you found a real release-blocking bug. Thanks for the careful report; three fixes landed on master.

What was actually broken

v0.8.3 shipped all three llama.cpp composes (llamacpp/default, llamacpp/mtp, llamacpp/mtp-vision) referencing image: llama-cpp:local — a custom image that existed only on the maintainer's rig. There was no Dockerfile, no build script, no setup.sh hook to produce it for users. Anyone running bash scripts/switch.sh llamacpp/mtp on a fresh clone would hit "image not found" and die at boot. The README also lied — claimed "no custom build needed" while the composes pointed at the orphan image.

The custom image was a v0.8.3-dev artifact from when MTP PR #22673 was still bleeding-edge in upstream. The official upstream ghcr.io/ggml-org/llama.cpp:server-cuda now has it merged — so the custom image was just stale, not necessary.

The fixes (landed on master)

1. c3e7c7e — replace llama-cpp:local with upstream ghcr.io/ggml-org/llama.cpp:server-cuda across all 3 composes + the engine yaml + README.
2. 4a53eda — switch from a pinned build tag to the rolling :server-cuda tag. We have no club-3090 patches on llama.cpp (unlike our vLLM track which pins to specific nightlies for Genesis-anchor coupling) — pinning here was cargo-culted from the vLLM pattern. Users now auto-pick up MTP improvements, EAGLE3 fixes, kernel updates from upstream. Override with LLAMACPP_IMAGE=ghcr.io/.../server-cuda-bXXXX if a future upstream build regresses and you need to pin reactively.
3. 78556f8 — report.sh now captures the image digest + OCI labels (build tag, upstream commit, source URL) in addition to the plain image tag. This was load-bearing once we went rolling: the tag alone is ambiguous; the digest pins down the exact bytes you're running. If you re-run report.sh --full after pulling, your report will now carry e.g. b9246 + 871b0b70f..., so we can know exactly which build to repro on.

Verified end-to-end on a fully clean rig (deleted the orphan image from local docker, re-ran bash scripts/switch.sh llamacpp/mtp as a fresh user would): boots in 8s, single-card, image-confirmed-upstream, 50-51 t/s narrative — matches the shipped 51.28 within noise. So git pull && bash scripts/switch.sh llamacpp/mtp should now work for you out of the box.

Re: your perf gap (32/36 vs ours 51/60)

MTP is firing for you (~1.5× the AR baseline of ~21 t/s; ours gets ~2.5×). Two suspects in your report.sh worth investigating:

PCIe link downgrade — worth checking under load

Your lspci snapshot shows:

LnkCap: Speed 16GT/s, Width x16   ← supports Gen 4
LnkSta: Speed 2.5GT/s (downgraded), Width x16   ← actually running Gen 1

That's PCIe Gen 1 (~1 GB/s/lane vs Gen 4's ~2 GB/s/lane). For pure decode this shouldn't matter much once weights are resident, but for MTP specifically the draft head's per-pass activation buffers + any DMA work go through that link. The lspci might be capturing ASPM idle state (Gen 1 when nothing's pending — normal power saving). Worth checking under load:

# While the bench is mid-flight:
sudo lspci -vvv -s 0000:01:00.0 | grep LnkSta

If it stays at 2.5 GT/s during bench → real bottleneck, root-cause in your hypervisor (KVM passthrough config: pcie-root-port settings, IOMMU group, ACS). If it pops to 16 GT/s under load → benign ASPM.

KVM virtualization tax

Environment: kvm (virtualized) in your report — expect 5-15 % overhead vs bare metal regardless of PCIe (hypervisor context switches, IOMMU translation). Combined with potential PCIe downgrade, the ~40 % gap becomes more plausible.

Other quick checks

1. Confirm MTP draft is actually loading. In docker logs <container>, look for loaded draft model or MTP head: enabled at boot. Your 32 t/s vs AR's 21 t/s implies MTP IS engaged, but worth verifying explicitly.
2. For a clean A/B: try --temp 0 (greedy) on the bench. Eliminates sampling-rejection variance and isolates whether your engine's MTP acceptance is below ours.
3. MTP n-max sweep. Your --spec-draft-n-max 2 matches our default. --spec-draft-n-max 4 --spec-draft-p-min 0.0 is queued as an A/B on our side (Llama.cpp merged MTP - now the most reliable for single 3090 for Qwen3.6-27B with high context + vision #152 has cross-engine evidence of headroom there).

If you re-test on the rolling tag and your numbers shift (because upstream landed MTP improvements between our build b9235 and whatever's current), I'd love to add your data to the discussion. Same if the PCIe-under-load check turns up something. And again — genuinely good catch, this would have been hours of frustration for the next person.

[zemaphore]

Resolved!

The performance bottleneck is purely due to my 3090 set power limit of 230W. Are you sure - as indicated under substrate in your post - that you're running a 230W power budget too @noonghunna ?

Performance with 350W budget on my end:

========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
=== warmups (3) ===
  warm-1     wall= 18.94s  ttft=   208ms  toks=1000  wall_TPS= 52.79  decode_TPS= 53.38
  warm-2     wall= 19.19s  ttft=   173ms  toks=1000  wall_TPS= 52.10  decode_TPS= 52.57
  warm-3     wall= 19.26s  ttft=   173ms  toks=1000  wall_TPS= 51.91  decode_TPS= 52.38

=== measured (5) ===
  run-1      wall= 19.22s  ttft=   172ms  toks=1000  wall_TPS= 52.04  decode_TPS= 52.51
  run-2      wall= 19.95s  ttft=   173ms  toks=1000  wall_TPS= 50.13  decode_TPS= 50.57
  run-3      wall= 19.40s  ttft=   172ms  toks=1000  wall_TPS= 51.55  decode_TPS= 52.01
  run-4      wall= 18.82s  ttft=   173ms  toks=1000  wall_TPS= 53.13  decode_TPS= 53.62
  run-5      wall= 19.40s  ttft=   171ms  toks=1000  wall_TPS= 51.54  decode_TPS= 52.00

=== summary [narrative] (n=5) ===
  wall_TPS       mean=  51.68   std=  1.08   CV= 2.1%   min=50.13   max=53.13
  decode_TPS     mean=  52.14   std=  1.10   CV= 2.1%   min=50.57   max=53.62
  TTFT          mean=   172ms  std=    1ms  min=171ms  max=173ms
  PP tok/s       n/a (long-prompt fallback below)

========== CODE (prompt=78 chars, max_tokens=800) ==========
=== warmups (3) ===
  warm-1     wall=  9.46s  ttft=   172ms  toks= 565  wall_TPS= 59.74  decode_TPS= 60.85
  warm-2     wall= 13.40s  ttft=   169ms  toks= 800  wall_TPS= 59.70  decode_TPS= 60.46
  warm-3     wall= 13.67s  ttft=   172ms  toks= 800  wall_TPS= 58.54  decode_TPS= 59.28

=== measured (5) ===
  run-1      wall= 13.00s  ttft=   176ms  toks= 800  wall_TPS= 61.53  decode_TPS= 62.37
  run-2      wall=  8.43s  ttft=   171ms  toks= 488  wall_TPS= 57.87  decode_TPS= 59.06
  run-3      wall=  8.21s  ttft=   167ms  toks= 490  wall_TPS= 59.68  decode_TPS= 60.92
  run-4      wall=  6.79s  ttft=   169ms  toks= 402  wall_TPS= 59.17  decode_TPS= 60.67
  run-5      wall= 13.12s  ttft=   167ms  toks= 800  wall_TPS= 60.97  decode_TPS= 61.76

=== summary [code] (n=5) ===
  wall_TPS       mean=  59.84   std=  1.46   CV= 2.4%   min=57.87   max=61.53
  decode_TPS     mean=  60.96   std=  1.26   CV= 2.1%   min=59.06   max=62.37
  TTFT          mean=   170ms  std=    4ms  min=167ms  max=176ms
  PP tok/s       n/a (long-prompt fallback below)

========== PROMPT-PROCESSING (fallback target=10000 prompt tokens, max_tokens=16) ==========
=== measured (1) ===
  run-1      wall= 11.88s  ttft= 11596ms  prompt_toks= 13179  PP_tok/s=1136.53

=== summary [prompt-processing] (n=1) ===
  PP tok/s       mean=1136.53   std=  0.00   CV= 0.0%   min=1136.53   max=1136.53
  TTFT          mean= 11596ms  std=    0ms  min=11596ms  max=11596ms

=== GPU state ===
0, 100 %, 21023 MiB, 24576 MiB, 348.79 W, 82

=== Last 3 SpecDecoding metrics ===

P.S.: Link speed is 16x full on load... just scales down due to power management on idle.

[noonghunna]

happy days! 230-250W is okay in practise, however i've started using max power for the tests i release.

[zemaphore]

For whoever is interested... for 27B dense and MTP the compute scale meaningful with power budget.

TPS (narrative)	Increase Narrative TPS (%)	TPS (code)	Increase Code TPS (%)	3090 Power Budget (Watt)	Increase Power (%)
22.7	0%	26.9	0%	200	0%
27.8	22%	33.0	23%	220	10%
30.3	34%	36.1	34%	230	15%
43.0	89%	50.0	86%	280	40%
51.4	126%	60.5	125%	350	75%

[zemaphore]

Sadly I can't quite reach your numbers with a build from llama.cpp-master (

========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
=== warmups (3) ===
  warm-1     wall= 30.38s  ttft=   301ms  toks= 986  wall_TPS= 32.45  decode_TPS= 32.78
  warm-2     wall= 30.18s  ttft=   208ms  toks=1000  wall_TPS= 33.14  decode_TPS= 33.37
  warm-3     wall= 30.86s  ttft=   211ms  toks=1000  wall_TPS= 32.41  decode_TPS= 32.63

=== measured (5) ===
  run-1      wall= 31.35s  ttft=   207ms  toks=1000  wall_TPS= 31.90  decode_TPS= 32.11
  run-2      wall= 30.47s  ttft=   211ms  toks=1000  wall_TPS= 32.82  decode_TPS= 33.04
  run-3      wall= 30.61s  ttft=   209ms  toks= 951  wall_TPS= 31.07  decode_TPS= 31.28
  run-4      wall= 31.75s  ttft=   212ms  toks=1000  wall_TPS= 31.49  decode_TPS= 31.70
  run-5      wall= 31.01s  ttft=   209ms  toks=1000  wall_TPS= 32.25  decode_TPS= 32.46

=== summary [narrative] (n=5) ===
  wall_TPS       mean=  31.90   std=  0.67   CV= 2.1%   min=31.07   max=32.82
  decode_TPS     mean=  32.12   std=  0.68   CV= 2.1%   min=31.28   max=33.04
  TTFT          mean=   210ms  std=    2ms  min=207ms  max=212ms
  PP tok/s       n/a (vLLM log scrape unavailable; use PP=1 for long-prompt fallback)

========== CODE (prompt=78 chars, max_tokens=800) ==========
=== warmups (3) ===
  warm-1     wall= 11.04s  ttft=   221ms  toks= 383  wall_TPS= 34.70  decode_TPS= 35.41
  warm-2     wall= 12.07s  ttft=   217ms  toks= 437  wall_TPS= 36.21  decode_TPS= 36.87
  warm-3     wall= 21.73s  ttft=   205ms  toks= 800  wall_TPS= 36.81  decode_TPS= 37.16

=== measured (5) ===
  run-1      wall= 11.30s  ttft=   224ms  toks= 400  wall_TPS= 35.41  decode_TPS= 36.13
  run-2      wall= 16.98s  ttft=   207ms  toks= 615  wall_TPS= 36.21  decode_TPS= 36.66
  run-3      wall= 11.52s  ttft=   210ms  toks= 408  wall_TPS= 35.41  decode_TPS= 36.06
  run-4      wall= 21.78s  ttft=   205ms  toks= 800  wall_TPS= 36.72  decode_TPS= 37.07
  run-5      wall= 11.53s  ttft=   217ms  toks= 399  wall_TPS= 34.61  decode_TPS= 35.27

=== summary [code] (n=5) ===
  wall_TPS       mean=  35.67   std=  0.82   CV= 2.3%   min=34.61   max=36.72
  decode_TPS     mean=  36.24   std=  0.68   CV= 1.9%   min=35.27   max=37.07
  TTFT          mean=   213ms  std=    8ms  min=205ms  max=224ms
  PP tok/s       n/a (vLLM log scrape unavailable; use PP=1 for long-prompt fallback)

=== GPU state ===
0, 98 %, 20931 MiB, 24576 MiB, 229.84 W, 73

cat compose.yaml
services:
  llama:
    image: local/llama.cpp:server-cuda-${LLAMA_PR:-master}
    build:
      context: https://github.com/ggml-org/llama.cpp.git#master
      dockerfile: .devops/cuda.Dockerfile
      target: server
      args:
        CUDA_VERSION: "12.8.1"
        CUDA_DOCKER_ARCH: "86"   # RTX 3090 = Ampere sm_86
    container_name: llama-cpp-build
    networks:
      - stacks_default
    env_file: .env
    environment:
      - TZ=Europe/Berlin
      - LLAMA_CACHE=/models
      - HF_HOME=/models
      - HF_TOKEN=${HF_TOKEN}
    volumes:
      - /mnt/llama_models:/models
    command: >
      --models-max 1
      --sleep-idle-seconds 600
      -np 1
      --port 8080
      --host 0.0.0.0
      -m /models/models--unsloth--Qwen3.6-27B-MTP-GGUF/snapshots/0da008d9b980b76520a952bb3aa4abbaa0955647/Qwen3.6-27B-Q4_K_M.gguf
      -c ${CTX_SIZE:-131072}
      -b ${BATCH_SIZE:-4096}
      -ub ${UBATCH_SIZE:-1024}
      -ngl 99
      -fa on
      --cache-type-k ${KV_TYPE:-q4_0}
      --cache-type-v ${KV_TYPE:-q4_0}
      -np 1
      --spec-type draft-mtp
      --spec-draft-n-max ${MTP_DRAFT_N_MAX:-2}
      --jinja
      --reasoning ${REASONING:-off}
      --reasoning-format ${REASONING_FORMAT:-deepseek}

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu
    restart: unless-stopped

club-3090 rig report

Generated: 2026-05-20 07:31:45 UTC

Redacted output (paths, host, user, tokens). Re-run with --no-redact for full data.

System

• OS: Ubuntu 24.04.4 LTS
• Kernel: 6.8.0-117-generic
• Environment: kvm (virtualized)
• Locale: C.UTF-8
• Timezone: UTC
• Uptime: up 2 hours, 26 minutes

CPU + RAM

• CPU: QEMU Virtual CPU version 2.5+ (8 threads)
• RAM: 62Gi total, 56Gi available
• Swap: 4.0Gi

Disk

• /var/lib/docker: 81G available, ext4 filesystem

GPU hardware

• GPU 0: NVIDIA GeForce RTX 3090 | 24576 MiB | driver 580.159.03 | VBIOS 94.02.59.00.F2 | persistence=Enabled
  • Power: limit=230.00 W (default=350.00 W, max=350.00 W) | current_draw=19.59 W ⚠ user-capped below default
  • PCIe: x16 lanes negotiated (GPU max x16, Gen up to 4) | bus 00000000:01:00.0
• CUDA Runtime (per driver): 13.0
• ECC mode: [N/A] (3090s don't have ECC; expect N/A)

NVLink

No NVLink detected (PCIe-only)

Topology

PCIe / GPU topology matrix

        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-7     0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

PCIe / P2P detail (lspci)

lspci PCIe/P2P detail (LnkSta / ACS / topology)

# lspci -t  (PCIe topology tree)
-[0000:00]-+-00.0
           +-01.0
           +-1a.0
           +-1a.1
           +-1a.2
           +-1a.7
           +-1b.0
           +-1c.0-[01]--+-00.0
           |            \-00.1
           +-1c.1-[02]--
           +-1c.2-[03]--
           +-1c.3-[04]--
           +-1d.0
           +-1d.1
           +-1d.2
           +-1d.7
           +-1e.0-[05-09]--+-01.0-[06]--+-03.0
           |               |            +-08.0
           |               |            \-12.0
           |               +-02.0-[07]--
           |               +-03.0-[08]--
           |               \-04.0-[09]--+-01.0
           |                            +-02.0
           |                            \-03.0
           +-1f.0
           +-1f.2
           \-1f.3

# lspci -vvv -s 0000:01:00.0  (GPU function: LnkCap/LnkSta/ACSCap/ACSCtl)
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
                LnkSta: Speed 2.5GT/s (downgraded), Width x16

# lspci -vvv -s 0000:00:1c.0  (upstream bridge of 0000:01:00.0: LnkCap/LnkSta/ACSCap/ACSCtl)
                LnkCap: Port #1, Speed 16GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns
                LnkSta: Speed 2.5GT/s, Width x16
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

# lspci -nnk | grep -A3 -i nvidia  (driver binding + device IDs)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] GA102 [GeForce RTX 3090] [1462:3881]
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] GA102 High Definition Audio Controller [1462:3881]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

Full nvidia-smi

Full nvidia-smi output

Wed May 20 07:31:45 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        On  |   00000000:01:00.0 Off |                  N/A |
| 44%   47C    P8             19W /  230W |   20931MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A           39249      C   /app/llama-server                     20920MiB |
+-----------------------------------------------------------------------------------------+

Display / desktop state

• $DISPLAY: unset (headless)
• Display processes running: none detected
• GPU 0 idle VRAM: 20931 MiB ⚠ something is using this GPU (display, browser, container)

Container runtime

• Docker: 29.5.1
• docker compose (v2): 5.1.3
• NVIDIA Container Toolkit: 1.19.0

Stack version

• club-3090: v0.8.3-4-gdb9c5e1 (branch: master, SHA db9c5e1)
• GENESIS_PIN default: 7b9fd319 (per scripts/setup.sh)

Profile state

• Profile schema version: 1
• Profile counts: 9 hardware, 4 models, 5 workloads, 7 engines, 6 drafters
• Compose registry: 43 entries
• Canonical scenarios: 9
• Calibration:
  • gemma-4-26b-a4b: 2 rows
  • gemma-4-31b: 7 rows
  • qwen3.6-27b: 11 rows
  • qwen3.6-35b-a3b: 2 rows
• Active estate: none (~/.club3090/estate.yml not found)

KV math calibration

• Overall: 22/22 (100%)
• No FAIL rows. kv-calc projections should agree with measured VRAM within the ±1.5 GB error band.

Full kv-calc --calibration output

========================================================================================
Calibration — predicted per-card VRAM vs measured BENCHMARKS rows
========================================================================================

  Predicted = weights + activation + overhead + drafter + (KV capped at available).
  Budget = mem_util × VRAM. Measured = nvidia-smi peak during bench (target ≈ budget).
  Verdict ✓ iff PASS/TIGHT and measured < VRAM (boot OK).

== qwen3.6-27b ==
  compose                     predicted    budget   measured  verdict
  ─────────────────────────   ─────────  ────────  ─────────  ───────
  dual                          19.91 GB   22.80 GB    23.60 GB    PASS ✓
  dual-turbo                    18.84 GB   22.80 GB    19.80 GB    PASS ✓
  dual-dflash                   18.04 GB   22.80 GB    23.60 GB    PASS ✓
  dual-dflash-noviz             22.80 GB   22.80 GB    23.80 GB   TIGHT ✓
  dual4                         15.73 GB   22.80 GB    23.50 GB    PASS ✓
  dual4-dflash                  16.18 GB   22.80 GB    22.00 GB    PASS ✓
  dual-dflash-noviz@175K        22.80 GB   22.80 GB    21.80 GB   TIGHT ✓
  long-text                     22.32 GB   22.32 GB    22.30 GB   TIGHT ✓
  long-vision                   22.12 GB   22.80 GB    23.00 GB    PASS ✓
  bounded-thinking              22.80 GB   22.80 GB    21.70 GB    PASS ✓
  minimal@64K                   21.60 GB   21.60 GB    22.40 GB   TIGHT ✓

  Verdict accuracy: 11/11 (100%)

== qwen3.6-35b-a3b ==
  compose                     predicted    budget   measured  verdict
  ─────────────────────────   ─────────  ────────  ─────────  ───────
  qwen-a3b-preview              22.03 GB   22.08 GB    21.94 GB    PASS ✓
  qwen-a3b-preview-mtp          22.08 GB   22.08 GB    22.72 GB    PASS ✓

  Verdict accuracy: 2/2 (100%)

== gemma-4-31b ==
  compose                     predicted    budget   measured  verdict
  ─────────────────────────   ─────────  ────────  ─────────  ───────
  gemma-dual-int8               22.80 GB   22.80 GB    22.20 GB   TIGHT ✓
  gemma-dual-int8-262k          22.80 GB   22.80 GB    22.10 GB    PASS ✓
  gemma-dual-dflash             22.08 GB   22.08 GB    22.70 GB   TIGHT ✓
  gemma-dual-dflash             22.08 GB   22.08 GB    22.30 GB   TIGHT ✓
  gemma-dual-awq                20.40 GB   20.40 GB    19.80 GB   TIGHT ✓
  gemma-dual                    22.08 GB   22.08 GB    22.50 GB   TIGHT ✓
  gemma-dual-int8@256K          22.80 GB   22.80 GB    22.50 GB   TIGHT ✓

  Verdict accuracy: 7/7 (100%)

== gemma-4-26b-a4b ==
  compose                     predicted    budget   measured  verdict
  ─────────────────────────   ─────────  ────────  ─────────  ───────
  gemma-a4b-awq                 22.08 GB   22.08 GB    23.45 GB   TIGHT ✓
  gemma-a4b-awq-mtp             22.08 GB   22.08 GB    23.50 GB   TIGHT ✓

  Verdict accuracy: 2/2 (100%)

Overall: 22/22 (100%)

Notes:
  - This is a directional estimator (±1.5 GB error band on the breakdown).
  - vLLM's `gpu_worker.py` boot log is the authoritative source.
  - If predicted PASS but measured > budget, file an issue with `scripts/report.sh --bench`.

Active container

• Name: llama-cpp-build
• Engine: llamacpp
• Status: Up 16 minutes (healthy)
• Ports: unknown
• Image: local/llama.cpp:server-cuda-master

Container engine state (llama.cpp)

Model load + KV cache init:

0.00.288.608 I srv    load_model: loading model '/models/models--unsloth--Qwen3.6-27B-MTP-GGUF/snapshots/0da008d9b980b76520a952bb3aa4abbaa0955647/Qwen3.6-27B-Q4_K_M.gguf'
0.26.301.975 I srv    load_model: creating MTP draft context against the target model '/models/models--unsloth--Qwen3.6-27B-MTP-GGUF/snapshots/0da008d9b980b76520a952bb3aa4abbaa0955647/Qwen3.6-27B-Q4_K_M.gguf'
0.26.437.033 I srv    load_model: initializing slots, n_slots = 1
0.26.486.262 I srv    load_model: speculative decoding context initialized
0.26.486.265 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 131072
0.26.486.366 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
0.26.486.367 I srv    load_model: use `--cache-ram 0` to disable the prompt cache
0.26.486.367 I srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391

Recent warnings/errors (last 5):

warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host

Full boot log (first 200 lines)

First 200 lines of docker logs

warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
0.00.186.188 W DEPRECATED: argument '-np' specified multiple times, use comma-separated values instead (only last value will be used)
0.00.186.399 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.186.402 I device_info:
0.00.287.101 I   - CUDA0   : NVIDIA GeForce RTX 3090 (24124 MiB, 23859 MiB free)
0.00.287.114 I   - CPU     : QEMU Virtual CPU version 2.5+ (64303 MiB, 64303 MiB free)
0.00.287.187 I system_info: n_threads = 8 (n_threads_batch = 8) / 8 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.287.215 I srv          init: running without SSL
0.00.287.243 I srv          init: using 7 threads for HTTP server
0.00.287.325 I srv         start: binding port with default address family
0.00.288.593 I srv          main: loading model
0.00.288.608 I srv    load_model: loading model '/models/models--unsloth--Qwen3.6-27B-MTP-GGUF/snapshots/0da008d9b980b76520a952bb3aa4abbaa0955647/Qwen3.6-27B-Q4_K_M.gguf'
0.00.288.651 I common_init_result: fitting params to device memory ...
0.00.288.654 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.26.026.320 W llama_context: n_ctx_seq (131072) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.26.183.404 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.26.301.975 I srv    load_model: creating MTP draft context against the target model '/models/models--unsloth--Qwen3.6-27B-MTP-GGUF/snapshots/0da008d9b980b76520a952bb3aa4abbaa0955647/Qwen3.6-27B-Q4_K_M.gguf'
0.26.302.010 W llama_context: n_ctx_seq (131072) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.26.437.033 I srv    load_model: initializing slots, n_slots = 1
0.26.453.594 I common_context_can_seq_rm: the context supports bounded partial sequence removal
0.26.485.921 I common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
0.26.485.927 I common_speculative_impl_draft_mtp: - n_max=2, n_min=0, p_min=0.00, n_embd=5120
0.26.485.929 I common_speculative_impl_draft_mtp: - gpu_layers=-1, cache_k=f16, cache_v=f16, ctx_tgt=yes, ctx_dft=yes, devices=[default]
0.26.486.262 I srv    load_model: speculative decoding context initialized
0.26.486.265 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 131072
0.26.486.366 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
0.26.486.367 I srv    load_model: use `--cache-ram 0` to disable the prompt cache
0.26.486.367 I srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
0.26.486.392 W srv          init: --cache-idle-slots requires --kv-unified, disabling
0.26.497.125 I init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
<think>

</think>

'
0.26.505.072 I srv          init: init: chat template, thinking = 0
0.26.505.087 I srv          main: model loaded
0.26.505.089 I srv          main: server is listening on http://0.0.0.0:8080
0.26.505.090 I srv  update_slots: all slots are idle
0.59.517.273 I srv  params_from_: Chat format: peg-native
0.59.517.421 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.59.517.425 I srv  get_availabl: updating prompt cache
0.59.517.429 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.59.517.431 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 131072 tokens, 8589934592 est)
0.59.517.431 I srv  get_availabl: prompt cache update took 0.01 ms
0.59.517.475 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
1.00.076.894 I slot print_timing: id  0 | task 0 | prompt eval time =     186.97 ms /    13 tokens (   14.38 ms per token,    69.53 tokens per second)
1.00.076.899 I slot print_timing: id  0 | task 0 |        eval time =     372.30 ms /    13 tokens (   28.64 ms per token,    34.92 tokens per second)
1.00.076.899 I slot print_timing: id  0 | task 0 |       total time =     559.27 ms /    26 tokens
1.00.076.902 I slot print_timing: id  0 | task 0 |    graphs reused =          5
1.00.076.903 I slot print_timing: id  0 | task 0 | draft acceptance = 0.80000 (    8 accepted /    10 generated)
1.00.076.920 I statistics        draft-mtp: #calls(b,g,a) =    1      5      5, #gen drafts =      5, #acc drafts =     4, #gen tokens =     10, #acc tokens =     8, dur(b,g,a) = 0.005, 46.052, 0.013 ms
1.00.076.934 I slot      release: id  0 | task 0 | stop processing: n_tokens = 26, truncated = 0
1.00.076.943 I srv  update_slots: all slots are idle
1.06.429.234 I srv  params_from_: Chat format: peg-native
1.06.429.407 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.231 (> 0.100 thold), f_keep = 0.346
1.06.429.411 I srv  get_availabl: updating prompt cache
1.06.429.591 W srv   prompt_save:  - saving prompt with length 26, total state size = 150.113 MiB (draft: 0.029 MiB)
1.06.495.352 I srv          load:  - looking for better prompt, base f_keep = 0.346, sim = 0.231
1.06.495.360 I srv        update:  - cache state: 1 prompts, 150.113 MiB (limits: 8192.000 MiB, 131072 tokens, 131072 est)
1.06.495.360 I srv        update:    - prompt 0x575df8c2d040:      26 tokens, checkpoints:  0,   150.113 MiB
1.06.495.361 I srv  get_availabl: prompt cache update took 65.95 ms
1.06.495.433 I slot launch_slot_: id  0 | task 8 | processing task, is_child = 0
1.06.495.439 W slot update_slots: id  0 | task 8 | n_past = 9, slot.prompt.tokens.size() = 26, seq_id = 0, pos_min = 25, n_swa = 0
1.06.495.440 W slot update_slots: id  0 | task 8 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
1.08.699.470 I slot print_timing: id  0 | task 8 | prompt eval time =     131.28 ms /    39 tokens (    3.37 ms per token,   297.08 tokens per second)
1.08.699.474 I slot print_timing: id  0 | task 8 |        eval time =    2072.63 ms /    73 tokens (   28.39 ms per token,    35.22 tokens per second)
1.08.699.474 I slot print_timing: id  0 | task 8 |       total time =    2203.91 ms /   112 tokens
1.08.699.475 I slot print_timing: id  0 | task 8 |    graphs reused =         33
1.08.699.475 I slot print_timing: id  0 | task 8 | draft acceptance = 0.77586 (   45 accepted /    58 generated)
1.08.699.482 I statistics        draft-mtp: #calls(b,g,a) =    2     34     34, #gen drafts =     34, #acc drafts =    30, #gen tokens =     68, #acc tokens =    53, dur(b,g,a) = 0.006, 291.353, 0.056 ms
1.08.699.504 I slot      release: id  0 | task 8 | stop processing: n_tokens = 113, truncated = 0
1.08.699.513 I srv  update_slots: all slots are idle
1.42.502.109 I srv  params_from_: Chat format: peg-native
1.42.502.344 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.345
1.42.502.349 I srv  get_availabl: updating prompt cache
1.42.502.518 W srv   prompt_save:  - saving prompt with length 113, total state size = 151.741 MiB (draft: 0.126 MiB)
1.42.588.186 I srv          load:  - looking for better prompt, base f_keep = 0.345, sim = 1.000
1.42.588.193 I srv        update:  - cache state: 2 prompts, 301.854 MiB (limits: 8192.000 MiB, 131072 tokens, 131072 est)
1.42.588.193 I srv        update:    - prompt 0x575df8c2d040:      26 tokens, checkpoints:  0,   150.113 MiB
1.42.588.194 I srv        update:    - prompt 0x575df7186030:     113 tokens, checkpoints:  0,   151.741 MiB
1.42.588.194 I srv  get_availabl: prompt cache update took 85.85 ms
1.42.588.270 I slot launch_slot_: id  0 | task 40 | processing task, is_child = 0
1.42.588.276 W slot update_slots: id  0 | task 40 | n_past = 39, slot.prompt.tokens.size() = 113, seq_id = 0, pos_min = 112, n_swa = 0
1.42.588.276 W slot update_slots: id  0 | task 40 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
1.44.489.927 I slot print_timing: id  0 | task 40 | prompt eval time =     123.47 ms /    39 tokens (    3.17 ms per token,   315.85 tokens per second)
1.44.489.931 I slot print_timing: id  0 | task 40 |        eval time =    1778.06 ms /    65 tokens (   27.35 ms per token,    36.56 tokens per second)
1.44.489.932 I slot print_timing: id  0 | task 40 |       total time =    1901.53 ms /   104 tokens
1.44.489.932 I slot print_timing: id  0 | task 40 |    graphs reused =         59
1.44.489.933 I slot print_timing: id  0 | task 40 | draft acceptance = 0.70370 (   38 accepted /    54 generated)
1.44.489.942 I statistics        draft-mtp: #calls(b,g,a) =    3     61     61, #gen drafts =     61, #acc drafts =    52, #gen tokens =    122, #acc tokens =    91, dur(b,g,a) = 0.007, 503.537, 0.092 ms
1.44.489.954 I slot      release: id  0 | task 40 | stop processing: n_tokens = 104, truncated = 0
1.44.489.976 I srv  update_slots: all slots are idle
1.53.238.573 I srv  params_from_: Chat format: peg-native
1.53.238.714 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.299 (> 0.100 thold), f_keep = 0.337
1.53.238.724 I srv  get_availabl: updating prompt cache
1.53.238.914 W srv   prompt_save:  - saving prompt with length 104, total state size = 151.573 MiB (draft: 0.116 MiB)
1.53.302.975 I srv          load:  - looking for better prompt, base f_keep = 0.337, sim = 0.299
1.53.302.984 I srv        update:  - cache state: 3 prompts, 453.427 MiB (limits: 8192.000 MiB, 131072 tokens, 131072 est)
1.53.302.985 I srv        update:    - prompt 0x575df8c2d040:      26 tokens, checkpoints:  0,   150.113 MiB
1.53.302.986 I srv        update:    - prompt 0x575df7186030:     113 tokens, checkpoints:  0,   151.741 MiB
1.53.302.986 I srv        update:    - prompt 0x72cf100105e0:     104 tokens, checkpoints:  0,   151.573 MiB
1.53.302.986 I srv  get_availabl: prompt cache update took 64.26 ms
1.53.303.051 I slot launch_slot_: id  0 | task 70 | processing task, is_child = 0
1.53.303.058 W slot update_slots: id  0 | task 70 | n_past = 35, slot.prompt.tokens.size() = 104, seq_id = 0, pos_min = 103, n_swa = 0
1.53.303.063 W slot update_slots: id  0 | task 70 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
1.53.500.658 I slot create_check: id  0 | task 70 | created context checkpoint 1 of 32 (pos_min = 112, pos_max = 112, n_tokens = 113, size = 149.752 MiB)
1.56.551.776 I slot print_timing: id  0 | task 70 | n_decoded =    102, tg =  34.00 t/s
1.59.571.568 I slot print_timing: id  0 | task 70 | n_decoded =    211, tg =  35.05 t/s
2.02.607.926 I slot print_timing: id  0 | task 70 | n_decoded =    328, tg =  36.22 t/s
2.04.371.215 I slot print_timing: id  0 | task 70 | prompt eval time =     248.72 ms /   117 tokens (    2.13 ms per token,   470.41 tokens per second)
2.04.371.219 I slot print_timing: id  0 | task 70 |        eval time =   10819.30 ms /   386 tokens (   28.03 ms per token,    35.68 tokens per second)
2.04.371.219 I slot print_timing: id  0 | task 70 |       total time =   11068.02 ms /   503 tokens
2.04.371.220 I slot print_timing: id  0 | task 70 |    graphs reused =        218
2.04.371.220 I slot print_timing: id  0 | task 70 | draft acceptance = 0.69876 (  225 accepted /   322 generated)
2.04.371.230 I statistics        draft-mtp: #calls(b,g,a) =    4    222    222, #gen drafts =    222, #acc drafts =   185, #gen tokens =    444, #acc tokens =   316, dur(b,g,a) = 0.008, 1798.055, 0.331 ms
2.04.371.247 I slot      release: id  0 | task 70 | stop processing: n_tokens = 503, truncated = 0
2.04.371.258 I srv  update_slots: all slots are idle
6.20.767.724 I srv  params_from_: Chat format: peg-native
6.20.767.861 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.120 (> 0.100 thold), f_keep = 0.006
6.20.767.867 I srv  get_availabl: updating prompt cache
6.20.768.135 W srv   prompt_save:  - saving prompt with length 503, total state size = 159.040 MiB (draft: 0.562 MiB)
6.20.917.137 I srv          load:  - looking for better prompt, base f_keep = 0.006, sim = 0.120
6.20.917.145 I srv        update:  - cache state: 4 prompts, 762.220 MiB (limits: 8192.000 MiB, 131072 tokens, 131072 est)
6.20.917.146 I srv        update:    - prompt 0x575df8c2d040:      26 tokens, checkpoints:  0,   150.113 MiB
6.20.917.147 I srv        update:    - prompt 0x575df7186030:     113 tokens, checkpoints:  0,   151.741 MiB
6.20.917.147 I srv        update:    - prompt 0x72cf100105e0:     104 tokens, checkpoints:  0,   151.573 MiB
6.20.917.147 I srv        update:    - prompt 0x575df9cb9620:     503 tokens, checkpoints:  1,   308.793 MiB
6.20.917.148 I srv  get_availabl: prompt cache update took 149.28 ms
6.20.917.220 I slot launch_slot_: id  0 | task 234 | processing task, is_child = 0
6.20.917.229 W slot update_slots: id  0 | task 234 | n_past = 3, slot.prompt.tokens.size() = 503, seq_id = 0, pos_min = 502, n_swa = 0
6.20.917.230 I slot update_slots: id  0 | task 234 | Checking checkpoint with [112, 112] against 2...
6.20.917.231 W slot update_slots: id  0 | task 234 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
6.20.917.232 W slot update_slots: id  0 | task 234 | erased invalidated context checkpoint (pos_min = 112, pos_max = 112, n_tokens = 113, n_swa = 0, pos_next = 0, size = 149.752 MiB)
6.24.146.040 I slot print_timing: id  0 | task 234 | n_decoded =    100, tg =  32.15 t/s
6.27.198.291 I slot print_timing: id  0 | task 234 | n_decoded =    210, tg =  34.08 t/s
6.30.206.008 I slot print_timing: id  0 | task 234 | n_decoded =    307, tg =  33.48 t/s
6.33.228.887 I slot print_timing: id  0 | task 234 | n_decoded =    409, tg =  33.54 t/s
6.36.269.385 I slot print_timing: id  0 | task 234 | n_decoded =    517, tg =  33.94 t/s
6.39.322.651 I slot print_timing: id  0 | task 234 | n_decoded =    613, tg =  33.52 t/s
6.42.327.435 I slot print_timing: id  0 | task 234 | n_decoded =    708, tg =  33.25 t/s
6.45.332.626 I slot print_timing: id  0 | task 234 | n_decoded =    800, tg =  32.93 t/s
6.48.384.534 I slot print_timing: id  0 | task 234 | n_decoded =    910, tg =  33.27 t/s
6.51.117.336 I slot print_timing: id  0 | task 234 | prompt eval time =     118.07 ms /    25 tokens (    4.72 ms per token,   211.74 tokens per second)
6.51.117.344 I slot print_timing: id  0 | task 234 |        eval time =   30081.91 ms /   986 tokens (   30.51 ms per token,    32.78 tokens per second)
6.51.117.345 I slot print_timing: id  0 | task 234 |       total time =   30199.98 ms /  1011 tokens
6.51.117.345 I slot print_timing: id  0 | task 234 |    graphs reused =        660
6.51.117.346 I slot print_timing: id  0 | task 234 | draft acceptance = 0.60538 (  540 accepted /   892 generated)
6.51.117.359 I statistics        draft-mtp: #calls(b,g,a) =    5    668    668, #gen drafts =    668, #acc drafts =   513, #gen tokens =   1336, #acc tokens =   856, dur(b,g,a) = 0.009, 5411.463, 1.009 ms
6.51.117.372 I slot      release: id  0 | task 234 | stop processing: n_tokens = 1011, truncated = 0
6.51.117.387 I srv  update_slots: all slots are idle
6.51.135.924 I srv  params_from_: Chat format: peg-native
6.51.136.053 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.025
6.51.136.057 I srv  get_availabl: updating prompt cache
6.51.136.258 W srv   prompt_save:  - saving prompt with length 1011, total state size = 168.547 MiB (draft: 1.130 MiB)
6.51.208.484 I srv          load:  - looking for better prompt, base f_keep = 0.025, sim = 1.000
6.51.208.492 I srv        update:  - cache state: 5 prompts, 930.767 MiB (limits: 8192.000 MiB, 131072 tokens, 131072 est)
6.51.208.493 I srv        update:    - prompt 0x575df8c2d040:      26 tokens, checkpoints:  0,   150.113 MiB
6.51.208.493 I srv        update:    - prompt 0x575df7186030:     113 tokens, checkpoints:  0,   151.741 MiB
6.51.208.494 I srv        update:    - prompt 0x72cf100105e0:     104 tokens, checkpoints:  0,   151.573 MiB
6.51.208.494 I srv        update:    - prompt 0x575df9cb9620:     503 tokens, checkpoints:  1,   308.793 MiB
6.51.208.494 I srv        update:    - prompt 0x575dfa021d20:    1011 tokens, checkpoints:  0,   168.547 MiB
6.51.208.495 I srv  get_availabl: prompt cache update took 72.44 ms
6.51.208.572 I slot launch_slot_: id  0 | task 683 | processing task, is_child = 0
6.51.208.580 W slot update_slots: id  0 | task 683 | n_past = 25, slot.prompt.tokens.size() = 1011, seq_id = 0, pos_min = 1010, n_swa = 0
6.51.208.581 W slot update_slots: id  0 | task 683 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
6.54.045.775 I slot print_timing: id  0 | task 683 | n_decoded =    100, tg =  36.76 t/s
6.57.062.201 I slot print_timing: id  0 | task 683 | n_decoded =    209, tg =  36.43 t/s
7.00.092.762 I slot print_timing: id  0 | task 683 | n_decoded =    305, tg =  34.79 t/s
7.03.157.502 I slot print_timing: id  0 | task 683 | n_decoded =    400, tg =  33.81 t/s
7.06.212.745 I slot print_timing: id  0 | task 683 | n_decoded =    505, tg =  33.92 t/s
7.09.280.388 I slot print_timing: id  0 | task 683 | n_decoded =    599, tg =  33.36 t/s
7.12.288.583 I slot print_timing: id  0 | task 683 | n_decoded =    703, tg =  33.53 t/s
7.15.296.006 I slot print_timing: id  0 | task 683 | n_decoded =    804, tg =  33.54 t/s
7.18.305.879 I slot print_timing: id  0 | task 683 | n_decoded =    896, tg =  33.21 t/s
7.21.295.672 I slot print_timing: id  0 | task 683 | prompt eval time =     116.36 ms /    25 tokens (    4.65 ms per token,   214.85 tokens per second)
7.21.295.676 I slot print_timing: id  0 | task 683 |        eval time =   29970.71 ms /  1000 tokens (   29.97 ms per token,    33.37 tokens per second)
7.21.295.677 I slot print_timing: id  0 | task 683 |       total time =   30087.07 ms /  1025 tokens
7.21.295.677 I slot print_timing: id  0 | task 683 |    graphs reused =       1087
7.21.295.678 I slot print_timing: id  0 | task 683 | draft acceptance = 0.65777 (  567 accepted /   862 generated)
7.21.295.687 I statistics        draft-mtp: #calls(b,g,a) =    6   1099   1099, #gen drafts =   1099, #acc drafts =   858, #gen tokens =   2198, #acc tokens =  1423, dur(b,g,a) = 0.010, 8995.224, 1.661 ms
7.21.295.700 I slot      release: id  0 | task 683 | stop processing: n_tokens = 1024, truncated = 0
7.21.295.726 I srv  update_slots: all slots are idle
7.21.312.338 I srv  params_from_: Chat format: peg-native
7.21.312.520 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.024
7.21.312.524 I srv  get_availabl: updating prompt cache
7.21.312.711 W srv   prompt_save:  - saving prompt with length 1024, total state size = 168.791 MiB (draft: 1.145 MiB)
7.21.384.158 I srv          load:  - looking for better prompt, base f_keep = 0.024, sim = 1.000
7.21.384.166 I srv        update:  - cache state: 6 prompts, 1099.558 MiB (limits: 8192.000 MiB, 131072 tokens, 131072 est)
7.21.384.168 I srv        update:    - prompt 0x575df8c2d040:      26 tokens, checkpoints:  0,   150.113 MiB
7.21.384.168 I srv        update:    - prompt 0x575df7186030:     113 tokens, checkpoints:  0,   151.741 MiB
7.21.384.168 I srv        update:    - prompt 0x72cf100105e0:     104 tokens, checkpoints:  0,   151.573 MiB

Recent failed boot attempts

No recently-exited vLLM or llama.cpp containers found.

---

Generated by bash scripts/report.sh. Flags: --verify (verify-full), --stress (verify-stress 7/7 incl. Cliff 2 needles), --soak (SOAK_MODE=continuous, catches Cliff 2b), --bench (canonical TPS), --full (all four, ~35 min). Use --no-redact to disable redaction (internal sharing only).

[JensJN]

Hi @noonghunna - thank you so for much for doing all this great work!

Quick question:
I note that you increased -ub from the default 512 to 1024 (vs. the previously shipped 2048).
Given higher ubatch sizes cause VRAM pressure, might it be possible to increase context for the llamacpp/mtp-vision profile by lowering ubatch back to the default 512?
49K context is unfortunately too low for a lot of agentic workloads where vision is needed (e.g. navigating or debugging UIs for example), so it might be a worthwhile trade-off.

Disregard if already included in your sweep and thanks again!

[noonghunna]

Great suggestion, @JensJN — you nailed it. Tested 2026-05-20 and the trade-off is strictly favorable for the agentic-vision use case you described.

Validated numbers (single 3090, llamacpp/mtp-vision, verify-stress 7/7 incl. 60K + 91K needle):

Config	ctx	VRAM	narr TPS	verify-stress
shipped ub=1024	49K	22.0 GB	56.5	✓ (shipped)
override ub=512 CTX=131072	131K	21.0 GB	50.0	✓ 7/7
override ub=512 CTX=196608	192K	22.5 GB	50.9	✓ 7/7
256K + ub=512 (untested ceiling)	256K	24.1 GB	—	TIGHT, not safe

~10% TPS hit buys ~4× more context. And the 91K-needle recall holds cleanly at both 131K and 192K — so the cliff-2-territory coverage we shipped at 49K stays intact when you push ctx.

Use it now (no rebuild needed)

UBATCH_SIZE=512 CTX_SIZE=196608 bash scripts/switch.sh llamacpp/mtp-vision

The lever is documented in models/qwen3.6-27b/llama-cpp/README.md (commit 1b2a76c).

Why we're not changing the default

Two reasons:

1. Shipped TPS numbers in this thread + # Head-to-Head, Gemma 4 31B vs Qwen3.6 27B — 6 configs on dual 3090 (2026-05-11) #119 are reference points for users comparing configs. Changing the default 49K → 192K under everyone would invalidate those without warning.
2. The trade is workload-dependent. Pure-chat / short-prompt users get less win from the extra context. Agentic-vision (your case) gets a lot. Better to give both groups the lever + clear guidance than to pick for them.

Underlying lesson

-ub is doing two jobs at once:

• Capping per-pass activation peak (cliff-survival for tool prefill — why we landed 1024 in v0.8.3)
• Eating into the VRAM budget that could otherwise be KV cache

When mmproj F16 (~0.8 GB) is loaded, that second job becomes load-bearing — there's less headroom for everything. Smaller -ub (512) gives back ~1 GB to the KV pool, enabling 4× ctx for a 10% TPS cost. For llamacpp/mtp (no vision) the same lever works but with smaller margins; for llamacpp/default already at the model's training-max 262K, there's no ctx upside (only TPS cost).

Thanks again for the smart suggestion — this is exactly the kind of "we shipped one safe default but the lever is configuration-conditional" insight we want the community surfacing. Folded into our compose-optimizer design notes as evidence that -ub needs to be solved by the optimizer per-(ctx, vision, headroom) tuple, not picked as a static knob.

[JensJN]

Great and makes a lot of sense, thanks for checking so quickly!

[laurimyllari]

@noonghunna Any tips for reproducing the 102 / 150 (68 %) quality numbers for llamacpp/mtp?

I started with master and no modifications, then added the froggeric template v19. After that I started looking into parallel tool calls. llama.cpp doesn't have the command line option and needs "parallel_tool_calls": true in the request, but the benchmark already supplies that. (Maybe --parallel-tool-calls is not needed with ik-llama either?)

I'm getting only 95/150 and am wondering if I'm missing a setting. I'd like to try to reproduce your numbers so I could compare my other experiments against it.

Quality results

llamacpp/mtp defaults

• git commit fe86b72
• no local modifications

Pack	Pass / Total	Score	p50 latency	p95 latency	Status
toolcall-15 (v1.0.1)	9 / 15	60%	1.13s	1.73s	ok
instructfollow-15 (v1.0.0)	12 / 15	80%	0.67s	1.33s	ok
structoutput-15 (v1.0.0)	8 / 15	53%	1.20s	2.25s	ok
dataextract-15 (v1.0.0)	0 / 15	0%	2.39s	3.79s	ok
reasonmath-15 (v1.0.0)	6 / 15	40%	4.15s	9.21s	ok
bugfind-15 (v1.0.0)	14 / 15	93%	3.64s	6.29s	ok
hermesagent-20 (v1.0.0)	10 / 20	50%	14.38s	29.14s	ok
cli-40 (v1.0.2)	20 / 40	50%	1.29s	12.27s	ok
TOTAL	79 / 150	53%

llamacpp/mtp with template fix

• added --chat-template-file /models/qwen3.6-27b-gguf/chat_template-v19.jinja

Pack	Pass / Total	Score	p50 latency	p95 latency	Status
toolcall-15 (v1.0.1)	9 / 15	60%	1.14s	1.80s	ok
instructfollow-15 (v1.0.0)	13 / 15	87%	0.63s	1.03s	ok
structoutput-15 (v1.0.0)	13 / 15	87%	1.24s	2.65s	ok
dataextract-15 (v1.0.0)	15 / 15	100%	2.47s	3.99s	ok
reasonmath-15 (v1.0.0)	6 / 15	40%	4.49s	11.75s	ok
bugfind-15 (v1.0.0)	12 / 15	80%	3.48s	7.51s	ok
hermesagent-20 (v1.0.0)	9 / 20	45%	18.94s	36.42s	ok
cli-40 (v1.0.2)	18 / 40	45%	1.47s	16.85s	ok
TOTAL	95 / 150	63%

[noonghunna]

will be updating the composes shortly along with revised results for both llama.cpp and ik_llama.cpp
while testing with ik_llama, i realised there was more context available on llama than what i had previously tested.
Also updated localbench-cli repo today with few fixes to in the test harness that was causing a few tests to fail.
Announcement should go out shortly.

[tomByrer]

Update-update: Context is only 200k; still good, but not as good as pinned update up top.

[noonghunna]

You're right, good catch. The shipped runtime default is 200K, not 262K — that pinned line predates the #199→#201 ceiling re-measurement and is too generous. 262K still boots (CTX_SIZE=262144), but the flash-attention scratch buffer walls it at ~125K once the window actually fills — that's the OOM a couple of folks hit (e.g. #197). So 200K is the max-safe default: it fills to ~183K with ~1 GB headroom on a 24 GB card, which is why verify-stress passes there and not at 262K. I'll tighten the pinned wording to 200K. 🙏

[tomByrer]

FYI, EAGLE3 for Qwen3.6 has been merged into llama.cpp mainline
https://github.com/ggml-org/llama.cpp/releases/tag/b9723