Show and tell: Qwen3.8-27B coding agent on 16GB VRAM with CPU FFN offload

[arczhi]

Repository, scripts, and full benchmark notes:

https://github.com/arczhi/5060ti-qwen3.8-27b

I have been testing whether a 27B coding model can complete real repository work on a 16GB GPU.

Hardware:

• CPU: AMD Ryzen 9 9950X, 16 cores / 32 threads
• GPU: NVIDIA RTX 5060 Ti, 16,311 MiB VRAM
• System RAM: 32GB DDR5-6000
• NVIDIA driver: 595.71.05
• CUDA: 13.2
• GPU power limit: 180W
• OS: Linux x86_64
• Runtime: Docker + llama.cpp CUDA server

During one serving snapshot, the process used about 15,767 MiB of VRAM and GPU utilization was around 51%.

Model and runtime configuration:

• Model: Qwen3.8-27B GGUF
• Quantization: UD-IQ4_XS
• Context: 80K
• MTP: MTP-1
• KV cache: Q4_0
• Flash Attention: enabled
• Parallel slots: 1
• Reasoning: disabled
• CPU FFN offload: 4 layers

As I understand it, --n-cpu-ffn N keeps the dense FFN weights of the first N layers in system RAM and runs those layers on the CPU. The goal is to reduce GPU memory pressure. It does not move the MTP draft model.

I used the same Pi agent session, repository, design document, skill, and implementation task for all variants.

The task was to implement a feature in a medium-sized Go repository, add tests, and run the scoped verification commands.

Results:

Configuration: No --n-cpu-ffn

• Agent time: 14m20s
• Server requests: 14
• Prefill: 22.18-326.36 tok/s
• Decode: 11.06-12.95 tok/s
• Server generation time: 7.89-184.60s per request
• Result: completed

Configuration: --n-cpu-ffn 4

• Agent time: 14m04s
• Server requests: 53
• Weighted prefill: 656.86 tok/s
• Weighted decode: 19.40 tok/s
• Average server request time: 15.16s
• Result: completed

Configuration: --n-cpu-ffn 8

• Agent time: 25m30s before stopping
• Server requests: 112
• Weighted prefill: 565.51 tok/s
• Weighted decode: 18.36 tok/s
• Average server request time: 12.78s
• Result: no clean final report

The FFN4 run modified the required code, added five tests, and passed the scoped build, vet, and gofmt checks.

The no-offload run also modified seven files and added five tests. Two failures were confirmed to be pre-existing baseline failures.

MTP acceptance ranged from 84.9% to 100% in the reference run.

The FFN4 configuration was the best balance in this real coding-agent task. It improved the observed prefill and decode behavior compared with the historical no-offload run.

FFN8 did not improve decode speed further. The agent made substantial progress, but entered a repeated host-side test-environment loop and did not reach a clean final report.

The main lesson for me is that CPU resources can matter even when the workload is GPU inference. A small amount of CPU FFN offload appears useful for reducing memory pressure, but more offload is not automatically better.

I would appreciate feedback on these questions:

1. Is this behavior consistent with the expected implementation of --n-cpu-ffn?
2. Are there recommended settings for keeping the MTP draft fast while offloading selected FFN layers?
3. Is there a better way to measure decode speed when prompt-cache reuse and agent tool calls are involved?
4. Are there other llama.cpp parameters worth testing for long-context coding agents on 16GB GPUs?

This is a single-machine benchmark. The goal is to make the setup reproducible and compare real task completion, not to claim universal performance.