Abliterated Qwen3.8-27B loops and falls apart on multi-step agentic tasks — is this an intrinsic abliteration side effect, or is there a config fix?

[syedabrar211065]

I'm serving an abliterated Qwen3.8-27B (OBLITERATED V3, Q8_0 GGUF) through llama.cpp's server (--jinja, MTP speculative draft) to an agentic coding CLI over the Anthropic-style /v1/messages route. Single-turn chat is fine. Multi-step agentic sessions are not.

Symptom : After a handful of good turns, the model degrades: it starts repeating tool calls, hallucinating, and stops following instructions. Early on the repeats are byte-identical (same ls/find over and over); later they mutate into near-identical variants — e.g. it hunts for a file by trying dozens of typo'd directory names (Ai_Agent_Works → Ai_Agent_Wocks → Ai.Agent.Works → Ai.Agent.Wokr …), each command slightly different, making zero progress.

Key data point: the clean (non-abliterated) weights of the same model, same quant, same server config, do not do this. They stop and recover correctly. So it tracks with abliteration, not the backend.

What I've tried (server side):

DRY sampler, --dry-multiplier up to 1.0, --dry-base 1.75, --dry-allowed-length 8, --dry-penalty-last-n 4096
--presence-penalty 1.5 with --repeat-last-n 4096 (so the window spans prior turns, not just the default 64)
Output length capped, plus a watchdog to kill wedged generations
These help with identical repeats but do nothing for the varied ones — which makes sense, since repetition penalties only act on matching token spans.

My working theory (would love confirmation):

In an agent loop, tool calls are grammar-constrained structured output, and sampler penalties barely reach grammar-forced tokens — so DRY/presence can't bite the looping tool calls.
More fundamentally, abliteration seems to damage multi-step recovery: once the model makes a small mistake it can't back out, it just flails with variations.
Questions:

1. Is degraded agentic/multi-step behavior a known, expected cost of abliteration, separate from the benchmark hit? Anyone measured it?

2. Is there any sampler or client-side config that actually helps here, or is this intrinsic to the weights (i.e. only non-abliterated weights fix it)?

3. For those running abliterated models as agents (not just chat): what's your setup, and does it hold a plan across many turns?

Setup / repro:

1.Backend: llama.cpp server (server-cuda), single 48 GB GPU. Qwen3.8-27B-OBLITERATED Q8_0 + its MTP draft head, --jinja, --ctx-size 131072, q8_0 K/V cache, flash-attn on, 2 server slots over a unified KV pool.

2.Clients, both reproduce it:

• Claude Code (VSCode extension) — via the Anthropic /v1/messages route.
• DeepSeek Harness (dsh) — via the OpenAI /v1/chat/completions route. Identical failure in both, so it's protocol- and client-independent.

3.Client sampling: Claude Code sends temperature 1.0, top_p 0.95, and no penalty/DRY/top_k fields (confirmed by reading /slots mid-request), so the server-side sampler defaults above are what actually apply.

4. Not the draft: MTP acceptance measured fine (~80% at 20K ctx), so speculative decoding isn't the cause.

[demirciberk]

Confirmed (from source), and a few pieces of external context — no upstream llama.cpp doc or benchmark directly answers "is this a known abliteration cost," so that part stays a hypothesis.

1. How llama.cpp actually orders grammar vs. penalties (confirmed from source)

common_sampler_sample() in common/sampling.cpp does, by default (grammar_first=false, which is what the server uses for chat/tool-call routes):

1. apply the reasoning-budget sampler,
2. run the full sampler chain (DRY, penalties, top_k/top_p, temp, …) on the full vocab logits, pick a token,
3. check the picked token against the grammar; only if it's invalid does it resample with grmr applied before the chain.

So the claim "sampler penalties barely reach grammar-forced tokens" isn't quite accurate for llama.cpp as implemented — DRY/penalty do see the full distribution first, every step. Grammar only intervenes as a rejection filter, and even the rejection-sampling path re-applies your penalty chain afterward.

2. Why DRY/presence-penalty don't touch the mutating repeats — this is expected, not a bug

DRY and repeat/presence penalties act on exact token-sequence overlap within their window. Ai_Agent_Works → Ai_Agent_Wocks → Ai.Agent.Works are different token sequences; none of these samplers are designed to detect near-duplicate/edit-distance-close continuations, only literal repeats. So "DRY doesn't stop the typo-hunting" is a designed limitation of every penalty sampler llama.cpp ships, independent of grammar or abliteration.

This exact failure mode (loop inside a grammar-constrained JSON string, not stopped by default repetition penalty) has been diagnosed upstream in another engine: vLLM's root-cause writeup for Gemma-4 tool-call loops (vllm-project/vllm#40099) found that inside a JSON string value the grammar allows thousands of legal tokens, but the base logit distribution heavily favors the repeat, and default repetition_penalty=1.0 does nothing about it — so they added a dedicated n-gram RepetitionDetectionParams loop-breaker (auto-enabled when structured output is active), separate from ordinary penalties. llama.cpp has no equivalent semantic/n-gram loop-breaker today — DRY is the closest tool it has, and by design it won't catch non-identical variants. There's also a related llama.cpp-specific bug class where an unbounded zero_or_more grammar rule lets a model spin forever inside a tool call (#21375) — worth ruling out separately from the abliteration question, since it reproduces on non-abliterated models too.

3. Is agentic degradation a known/measured cost of abliteration specifically?

Not confirmed. There's no llama.cpp-side documentation or benchmark on this, and I found no credible measured study isolating "abliteration hurts multi-step self-correction" as a distinct axis from the general capability hit abliteration is already known to cause. Treat the following as hypothesis only: abliteration removes a "refusal direction" from the residual stream (typically projected out uniformly across many/all layers), and that projection can also damage nearby features that aren't purely refusal-related — self-correction/backtracking behavior is a plausible casualty of that, but I have no citation confirming it's specifically abliteration and not just quantization/model-specific brittleness. Your own control (clean weights, same quant/config, don't loop) is the strongest evidence in the thread, but it's a single anecdotal data point, not a benchmark.

4. What to actually try next

To separate "grammar/tool-loop interaction" from "abliterated weights lose multi-step coherence generally":

• Reproduce the failure without any grammar/tool calling — give the abliterated model a long, plain-text multi-turn task that requires backtracking (e.g., a multi-step logic puzzle where an early wrong branch must be abandoned) at the same --ctx-size/quant. If it degrades the same way with zero grammar involved, the grammar/tool-call framing is not the cause and the theory in your post is disproved for this case.
• Watch the server log (-v) for context shift messages. At --ctx-size 131072 with a long agentic session, if the KV cache ever triggers context-shifting/prompt truncation, your --repeat-last-n/--dry-penalty-last-n windows can silently desync from what the model is actually looking at, which looks exactly like "penalties stopped working after N turns."
• Since llama.cpp lacks a vLLM-style semantic loop-breaker, consider adding one client-side in your agent harness: track edit-distance between consecutive tool-call arguments, and after N near-duplicate attempts, hard-abort the turn and inject a corrective message rather than letting the model keep sampling. That's effectively what vLLM's RepetitionDetectionParams does at the sampler level, and it's easy to replicate at the orchestrator level without server changes.
• Try a materially lower temperature specifically for tool-call turns (your report shows the client sends temperature 1.0, top_p 0.95 with no other constraints) — this won't fix an underlying weight-level coherence loss, but it shrinks the combinatorial space the model can wander through for "creative" typo variants.
• If you can reproduce a minimal repro (single tool, small grammar, non-identical repeated variants), that's worth filing as a llama.cpp feature request for an n-gram/loop-detection sampler analogous to vLLM's, referencing [Bugfix] Auto-enable repetition detection for grammar-constrained structured output vllm-project/vllm#40099 as prior art — nothing like it currently exists in common/sampling.cpp.

I don't have evidence either confirming or ruling out "only non-abliterated weights fix it" — that would need a controlled A/B with several abliteration techniques/strengths on the same base model, which I haven't seen published.

[demirciberk]

Glad it helped! If you're set, marking this comment as the answer helps others find it faster.

[syedabrar211065]

@demirciberk
Thanks again—your suggestions drove this whole investigation. It turns out my original framing was slightly off. While abliteration does cause some intrinsic degradation, the catastrophic looping was actually caused by a combination of configuration issues masking each other.

Here is the breakdown of what was actually happening, what failed, and the final verdict.

The Real Culprits (Configuration)

The looping wasn't primarily a grammar constraint issue. It was caused by three overlapping config amplifiers:

• Chat template forcing maximum reasoning: The template defaulted reasoning_effort to xhigh. The abliterated weights obeyed this literally and ruminated in the <think> stream forever without concluding. Changing this to medium (which maps to an empty instruction) resulted in correct, terminated answers.
• Mismatched MTP draft head: I was using the base model's MTP draft head for speculative decoding on the abliterated target. Because abliteration modifies hidden states, the draft head was running on activations it wasn't trained on, corrupting generation enough to prevent termination.
• Client retries masking as hangs: The chat template was set to raise an exception for unhandled reasoning_effort values (like high). This caused an HTTP 500, which the client quietly retried with backoff. To me, this presented as a multi-minute generation hang rather than a crash.

Because these issues were independently sufficient to cause failure, single-variable testing was a trap. Turning off the draft head changed nothing while xhigh was still active, which initially made me rule out speculative decoding as the cause.

What Failed (Testing your suggestions)

Your suggestions were spot on for diagnosing this, and led to two very interesting negative results:

• Repetition penalties make it worse: Adding DRY and presence penalties didn't stop the loop; it turned repetition into hallucination. When the model is penalized for outputting the correct exact string (like a valid file path or shell command), it is forced to generate a near-duplicate typo instead. Penalties don't fix agentic loops; they manufacture them.
• Client-side prompt injection is ignored: I built the edit-distance loop breaker you suggested. It detected the near-duplicate tool calls perfectly. However, injecting a "stop repeating yourself" notice into the system prompt did nothing. An abliterated model's core damage is instruction-following—you can't fix that by feeding it more instructions. Any server-side loop breaker must hard-abort generation, not down-weight tokens or inject prompts.

The Final Verdict on Abliteration

Even after fixing the template defaults, removing the draft head, and turning off <think> blocks by default, there is still a residual defect. On a small fraction of seeds, the abliterated model will still loop. Abliteration genuinely damages multi-step coherence and instruction following over long contexts.

Sadly, this means I can't use the abliterated model as I originally needed for my agent workflows. I’ve switched back to the non-abliterated weights for now, as they terminate properly, follow instructions, and actually handle dual-use coding tasks just fine in a proper context.

This leaves me really curious about the broader community's experience: how are other people using these abliterated models? Is anyone successfully running them in deep multi-turn agent loops, and if so, do you face this exact same degradation issue? I would love to know what solutions, orchestration-level workarounds, or workflow tricks you rely on that I haven't tried yet to resolve this.