Two traps for local RAG: context_window vs the loaded n_ctx, and why abliterated models undermine grounding specifically

[behrnt-slatgng]

Disclosure: I run abliterated open weights in production behind a hosted chat and coding agent (Grunz). That is the opposite of what PrivateGPT is for and I am not suggesting otherwise — posting two things about the weights and about context configuration, both of which apply identically when you run everything locally, and one of which is actually worse locally.

1. The context window is a load-time property, and the config can lie about it

With a hosted provider the trap is that the served window is smaller than the model card claims. Locally the trap is sharper, because there are two numbers and nothing reconciles them:

• context_window in settings.yaml, which the framework uses for its own prompt-assembly and truncation arithmetic
• the n_ctx the model was actually loaded with

If the first is larger than the second, the assembly logic packs a prompt sized for a window that does not exist. What happens next depends on the backend: llama.cpp will error, or silently roll the context, or shift the KV cache — and in the rolling case the part discarded is the front, which is the system prompt and the instructions, not the recent turn.

The symptom is "the model started ignoring my system prompt after a few messages", which almost nobody diagnoses as a config mismatch.

Compounding it: retrieval makes this easy to hit without noticing. similarity_top_k chunks plus a system prompt plus history adds up fast, and the number of chunks is the knob people tune for quality with no visibility into the token cost. Four chunks from a dense PDF is a different budget than four chunks from a FAQ.

Worth surfacing the actual loaded n_ctx next to the configured context_window, and warning when they disagree. It converts a silent failure into an obvious one.

2. Abliterated models lose format compliance before they lose knowledge

A lot of people running PrivateGPT reach for an uncensored variant — huihui, Heretic, the various -abliterated uploads. The degradation order on those is consistent and not the one most people expect:

1. instruction-following and output-format adherence degrade first
2. knowledge and reasoning degrade later

The model still knows the material. It stops reliably respecting the instruct template, honouring stop sequences, and holding a system-prompt constraint across a long context.

This matters specifically for RAG because the whole pattern depends on the model doing what it is told about provenance: answer from the provided context, say when the context does not cover it, cite the source. Those are instructions, and instruction-following is the exact capability ablation degrades first. An ablated model in a RAG pipeline is measurably more willing to answer from parametric memory while presenting it as though it came from your documents — which is the one failure mode a private, document-grounded assistant most needs to avoid.

Two practical notes:

• Do not evaluate the variant by chatting with it. Prose quality holds up, often looking better than base. Test it by sending N requests with an explicit output contract — a required citation format, a required "not in the provided context" response — and score the fraction that come back well-formed, independently of correctness.
• Stay at Q6_K or above if you can. Ablation damage appears to compound with quantization damage rather than sit beside it, and perplexity barely moves while compliance falls off a cliff, so the usual quant-quality metric will tell you everything is fine.

I have volume rather than a clean controlled setup, so I would genuinely welcome being corrected by anyone who has measured this properly.