Patterns are output contracts, and abliterated models quietly stop honouring them

[behrnt-slatgng]

Disclosure first: I build Grunz, a chat + coding agent running abliterated open-weight models. Nothing to sell — fabric's whole design depends on a model property that breaks in a specific, predictable way on certain models, and it seemed worth flagging.

Patterns are output contracts, and some models have quietly stopped honouring contracts

A fabric pattern is an instruction with an expected output shape. That works because the model obeys the shape. Everything downstream — piping between patterns, parsing, scripting around it — assumes obedience.

If you run abliterated / decensored models (and a lot of people using open weights do), the first thing that degrades is not knowledge. It is instruction-following and output-format adherence.

The model still knows the material. What gets measurably worse:

• respecting the instruct template and prefill
• honouring stop sequences
• staying inside a structured-output contract
• holding a system-prompt constraint across a long context

So a pattern that produces clean markdown on the base model produces almost clean markdown on the ablated one — an extra preamble, a stray "Here's the summary:", headings that drift after the third section. Individually trivial. Fatal if you are piping one pattern into the next.

Why it is easy to miss

The normal way people evaluate an ablated variant is to chat with it, confirm it does not refuse, and check the prose is good. That test cannot detect this. The model looks fine, often better than fine, because the refusal behaviour that used to interrupt you is gone.

Perplexity does not catch it either. On a generic corpus it barely moves while format compliance falls off a cliff. The metric says the model is fine and the pipeline says it is not.

What I think this implies for fabric specifically

Pattern output is worth validating, not just producing. Even a crude shape check — does it start with the expected heading, is there a preamble before the first token of real output — would surface the difference between "the model was wrong" and "the model was malformed". Those have different fixes and right now they look identical.

Chained patterns are where it compounds. A 3% format-drift rate per pattern is invisible in isolation and a coin flip across a chain of five.

Grammar-constrained decoding is doing more load-bearing work on these models than on clean ones, because it forecloses malformed output at the token level rather than relying on the model to want to follow the instruction.

Questions

1. Has anyone here compared the same pattern across a base model and its abliterated counterpart, scoring format compliance rather than output quality? I have the symptom at volume but no clean controlled comparison.

2. Is there interest in a pattern-level output assertion — something a pattern can declare about its own expected shape, checkable without an LLM?

Happy to be told this is a user problem rather than a fabric problem. It just took me a long time to work out that the model had not got dumber, it had stopped doing what it was told.

[ksylvan]

Interesting issue. Let me think about how we can approach it...

[behrnt-slatgng]

Thanks for looking at it. If it helps narrow things down, the smallest useful version is probably not a full validator, just a check for a preamble: does the output start with the first line the pattern asks for, or is there a "Here's the summary:" style line in front of it. That one catches most of what I see in practice, and it separates "the model was wrong" from "the model was malformed" without Fabric having to understand each pattern's full shape.

A pattern could optionally declare its expected first line or heading, and the check only runs where that is declared, so nothing changes for existing patterns.

I can post a few concrete before/after outputs from the same pattern on a base model and its abliterated variant if that would be useful for deciding.