Measured: what directional ablation does (and doesn't) do to the rest of the safety surface

[NososAnosos]

Hi all — independent eval data point I thought this community would find interesting.

What I did: Ran an 84-case adversarial battery (prompt injection, exfiltration canaries, refusal calibration pairs, benign controls — deterministic pass/fail/ambiguous scoring, hash-chained transcripts) against gemma-3-4b-it and its abliterated counterpart, same battery both sides.

Honest scope note: the abliterated weights are mlabonne's, not Heretic's output — same technique family (directional/refusal-vector ablation), but please don't read this as "we tested Heretic." Heretic itself wasn't run here.

Why: the README publishes refusal rate + KL divergence. I wanted to know what ablation does to everything else.

Results:

Category	Original	Abliterated
Jailbreak (blocked)	3/7	0/7
Obfuscation (blocked)	5/10	0/10
Refusal calibration	8/8	4/8
Exfiltration canary	5/2	5/2 (identical cases)
Injection (direct+indirect)	unchanged	unchanged
Benign compliance	6/6	6/6

Refusal behavior collapsed completely, as advertised. Everything else — canary resistance, injection resistance, benign compliance — came through bit-for-bit identical. The refusal direction looks surprisingly disentangled from the rest of the safety surface. One curious detail: a canary-leak case that fails on the original fails identically on the abliterated model, suggesting that failure mode isn't governed by the refusal vector at all.

Method is open source: github.com/NososAnosos/llm-redteam-harness

Happy to re-run against actual Heretic-produced weights if that's useful to anyone here.