Optimisers absorb format non-compliance and the cost is invisible in the final score

[behrnt-slatgng]

Disclosure up front: I build Grunz, a hosted chat + coding agent on abliterated open-weight models. Nothing to sell — this is about a variable I think optimisation currently absorbs without naming.

Signatures are output contracts, and some models are worse at honouring contracts than their quality suggests

A DSPy signature is a promise about output shape. Adapters parse against it. So compliance with that shape is load-bearing for the whole programme, separately from whether the content is any good.

For abliterated / decensored checkpoints — an increasingly large share of open-weight deployment — the thing that degrades first is not knowledge. It is instruction-following and output-format adherence. The model still knows the material; it gets worse at emitting it in the shape you asked for.

Why that is interesting for DSPy specifically

Optimisers will paper over this, and that is both a feature and a problem.

The feature: a compiled programme on a badly-behaved model will find prompts and demos that coax compliance out of it. That is real value and it is exactly what optimisation is for.

The problem: the cost of that coaxing is invisible. The optimiser spends its budget buying back format compliance instead of buying better reasoning, and the final metric does not distinguish the two. Two checkpoints that compile to similar scores may have got there very differently — one had headroom to improve the task, the other spent everything learning to fill in the fields.

And it means a compiled programme can be more brittle than its score implies. If the prompt is doing heavy lifting to enforce shape, small changes — a different adapter, a longer input, a model update — knock it over in a way a genuinely-compliant model would not.

What I think would be worth measuring

Parse-success rate as a metric alongside task score, reported per module during compilation. Two numbers rather than one: how often did the output parse, and how good was it when it did. A model at 0.72 task score with a 99% parse rate is in a very different position from one at 0.72 with an 80% parse rate that the optimiser rescued.

That would also make model selection legible. Right now the choice between checkpoints is made on a score that silently bundles compliance and capability.

Questions

1. Does anything already surface parse-failure rate during compilation? If the adapters track retries or failures somewhere accessible I would like to be pointed at it.

2. Has anyone compared optimiser budget-to-quality on a base model versus its abliterated counterpart? My prediction is that the ablated one needs meaningfully more budget to reach the same score, and that the difference is almost entirely format compliance rather than reasoning. That seems like a publishable result if true.

3. Is constrained decoding a first-class option for the parsing step? For a damaged checkpoint, grammar-level constraint removes the model's willingness from the equation, which seems strictly better than spending optimisation budget on it.

Happy to be told the adapters already handle this robustly enough that it does not matter. That would itself be useful to know.