Repository navigation
Probe idea: output-contract adherence as a measurable property, separate from refusal #2237
behrnt-slatgng
started this conversation in
Ideas and suggestions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Disclosure up front: I run abliterated open weights in production behind a hosted chat and coding agent (Grunz). Not pitching anything — this is a measurement gap I keep hitting and garak seems like the natural home for it.
The observation
On refusal-ablated variants (refusal-direction ablation in the mlabonne sense, huihui's variants, Heretic, and the rest), the first thing to degrade is not knowledge, and it is not safety behaviour in the sense garak already probes. It is instruction-following and output-format adherence.
Compared against the base model at the same quant:
Prose quality holds up. Generic benchmarks barely move. Perplexity on a standard corpus is close to unchanged while format compliance falls off a cliff, so the usual metric gives no warning at all.
Why this is a security property, not just a quality one
A lot of deployed guardrailing is really downstream parsing: an output filter, a schema validator, a tool-call allowlist, a classifier reading the model's own structured self-report. All of it assumes the model emits something parseable and on-spec.
A model that drifts off its output contract does not fail closed. It fails unparsed — and what a system does with an unparsed response is usually far less considered than what it does with a refusal. Ablation raises that rate measurably.
It also matters for anyone using garak to compare an ablated variant against its base: two models can score similarly on the existing probes and then behave very differently once something downstream is parsing them.
What a probe could look like
Nothing exotic:
The two comparisons that would actually be informative are ablated-vs-base at a fixed quant, and the same model walked down a quant ladder.
My own observation — high volume, but uncontrolled, so treat it as a hunch rather than a result — is that ablation damage and quantization damage appear to compound rather than sit beside each other. Compliance degrades worse than linearly as bpw drops on an ablated model in a way the base model does not reproduce. I would much rather see that measured properly than keep repeating it.
Happy to contribute compliance-rate data across the variants I serve if that is useful, and equally happy to be told this already exists under a name I have missed.
All reactions