Literature Review Hardware And Serving

Literature Review: Hardware & Serving

Part 4 of the Literature Review: ROCm versus CUDA, Ollama/llama.cpp tuning, and the Qwen3 think suppression investigation.

Why a structural fix beats a textual warning for recurring citation fabrication

Prompted by two live incidents (2026-07-24, 2026-07-26): a Searcher cites a wrong URL, the grounding check flags it inline in the finding's own text, and a fresh dispatch is handed that exact warning and cites the bad URL anyway, 7 times in a row in the worse case. The salvage path inherits the same poison since it reads raw evidence directly.

Negation is genuinely fragile in transformers, not folklore: several papers (arXiv:2511.12381, arXiv:2601.21433, arXiv:2605.28639) show suppression instructions can prime reproduction of the forbidden content instead of avoiding it, and a broader finding (arXiv:2605.01771) shows agents often acknowledge a constraint without executing it. Checked directly against CRAG (arXiv:2401.15884) and Self-RAG (arXiv:2310.11511): both use a trained scorer to structurally exclude bad content before the generator sees it, never a warning left in context. CiteGuard (arXiv:2510.17853) adds a persistent cross iteration exclusion set. No paper directly A/B tests "removal versus warning" and declares removal superior, that's a strong inference from this literature, not a cited head to head result.

Recommendation, implemented as a Python layer fix, not a prompt change: when the grounding check flags a citation mismatch, structurally strip that finding from the evidence FindingsWriter sees (mirroring CRAG/Self-RAG), keep a run level blocklist of confirmed bad claim/URL pairs filtered against every later evidence assembly point including the salvage path (mirroring CiteGuard), and keep the textual warning only as cheap defense in depth, not the primary mechanism.

Was ROCm the cause of a 9 candidate vLLM disqualification streak?

Confirmed ROCm/hardware specific: this project's vLLM required a bitsandbytes version on ROCm with no real production mileage, community trackers show bitsandbytes-on-ROCm was reportedly broken as recently as 2024 to 2025, and vLLM's own blog confirms its optimized attention backend targets only data center CDNA3, consumer Radeon gets a baseline path.

Not ROCm specific, and this covers most of that day's actual disqualifications: the thin coverage non convergence pattern reproduces identically on Ollama and vLLM, and narrate instead of call and citation fabrication are model behavior, not backend, failures. Verdict: ROCm's bitsandbytes immaturity explains the crashy, malformed JSON tail, but not most of the actual verdicts.

Did the community build something better for consumer AMD? Yes: llama.cpp's ROCm/HIP backend (what Ollama already uses) is repeatedly described as mature for RDNA consumer cards with no correctness complaints found, versus vLLM/bitsandbytes' broken history. A specific claim that Vulkan beats ROCm by 20% on RDNA4 came from an unverifiable personal blog and is contradicted by a more thorough multi contributor GitHub discussion showing ROCm and Vulkan trading wins by workload; rejected, don't cite it. Net result, acted on the same session: reverted to Ollama as the permanent backend, an informed tradeoff, the two known Ollama bugs that motivated the earlier vLLM move are still real and still apply to the candidates they affect.

Ollama and llama.cpp tuning for this hardware (RX 9060 XT, gfx1200, RDNA4, 17GB, ROCm 7.2.4)

Flash attention and OLLAMA_KV_CACHE_TYPE are the two real free levers per Ollama's own docs; KV cache quantization only takes effect once flash attention is on. q8_0 roughly halves KV cache memory with negligible quality loss; it's gated by an architecture allowlist that includes gemma3, gptoss, mistral3, and the qwen3 family, other architectures silently fall back to f16 with no error. The project's default gptoss architecture is on the list.

A real crash mode matches this exact card: ggml-org/llama.cpp#21376 documents the RX 9060 XT hard crashing (OOM to segfault) when KV cache allocation doesn't fit remaining VRAM after model weights load, unresolved at research time. Vulkan spills to system RAM instead of crashing on the same config, a genuinely different claim from the rejected "20% faster" one. rocWMMA (the flash attention accelerator) does support gfx1200, correcting an outdated "RDNA3 only" claim, but whether Ollama's prebuilt binaries ship the compile flag for it was unconfirmed. Implication, not yet acted on: turn on flash attention plus q8_0 KV cache for gpt-oss with an explicit GPU overhead reserved first, given the crash mode above; whether to trial Vulkan as the primary backend is a separate, unbenchmarked decision.

Qwen3 think suppression: fixable template gap, or unfixable serving bug?

A Reddit lead claimed the community fix is a numeric reasoning token budget rather than a boolean disable. Checked against Ollama's own docs: no numeric budget lever exists on either Ollama endpoint, that control is real only for raw llama.cpp server deployments. Live tested anyway: sending the correct native think: false field to qwen3:4b still produced a full reasoning block inline in content, ruling out a caller side parameter mistake.

A second lead pointed at a community patched Qwen3.6 chat template that forces an empty think block before the assistant turn to stop premature stalling, a different bug shape from this project's own symptom (open ended reasoning, not early stalling), but it confirmed template correctness genuinely governs think behavior. Investigating further: Ollama has no dedicated plain qwen3 renderer, only qwen3.5/qwen3coder/qwen3vl, and the qwen3.5 renderer has no architecture gate, so it can be forced onto any GGUF. Building a throwaway tag with that renderer against the project's real qwen3:4b weights: the renderer loaded and worked correctly with thinking enabled, but with thinking disabled it still generated a full reasoning block right after its own forced empty think prefix, and hallucinated a repeated tool call copied from its own instructions.

Verdict: four independent combinations (OpenAI style flag, native field on two endpoints, and the battle tested qwen3.5 renderer) all failed to suppress thinking on this specific checkpoint. The common factor is the model weights themselves, not the serving mechanism, consistent with the community fix's own explicit "3.5 and 3.6 only" scoping. No further serving or template lever identified; the next lead, if pursued, is on the model or training side (whether the LoRA's training data ever included nothink examples), not anything Ollama configurable.