Repository navigation
Metric proposal: format compliance as its own axis, orthogonal to correctness #3370
Replies: 1 comment
|
Separating contract compliance from correctness seems useful. I would also distinguish execution completeness: a missing response is different from a response that violates the schema, and both differ from a schema-valid but incorrect judgment. For comparisons, I would retain the original intended case count and report completion rate, contract-valid yield, and end-to-end correct yield against that denominator. Conditional rates, such as correctness among valid outputs, can still be useful, but should show their denominators so failed cases do not silently disappear. A small fault-injection test matrix could make the distinction concrete: no response, malformed JSON, schema-invalid JSON, and schema-valid but semantically wrong output. I would also report first-attempt and post-repair outcomes separately, since retries change the system being evaluated. Would execution status belong in the surrounding evaluation report rather than the compliance metric itself? That would preserve the metric’s narrower meaning while making model-selection comparisons easier to interpret. |
Uh oh!
There was an error while loading. Please reload this page.
Disclosure: I run abliterated open weights in production behind a hosted chat and coding agent (Grunz). Posting a metric proposal rather than a product — this is a gap I keep hitting and deepeval is the natural place for it.
Format compliance deserves to be a first-class metric, separate from correctness
Most LLM evaluation, deepeval's included, is organised around whether the answer is right: relevancy, faithfulness, hallucination, correctness against a ground truth. That is the important axis for most applications.
But for anything agentic, there is a second axis that determines whether the system works at all, and it is almost never scored separately: does the output conform to the contract it was asked to conform to?
An agent loop does not care whether an unparseable response was correct. It cannot read it either way. And the two properties genuinely come apart: a model can be more accurate and less compliant than its alternative, and picking on accuracy alone will make your agent worse.
Why this is not hypothetical
The cleanest demonstration is refusal-ablated open weights — the huihui, Heretic and mlabonne-style
-abliteratedvariants that a lot of people are now running.On these, the degradation order is consistently:
Prose quality holds. Generic benchmarks barely move. Perplexity on a standard corpus is close to unchanged while format compliance falls off a cliff. So every metric people normally reach for says the variant is fine, and then the parse failure rate in production is materially higher than the base model at the same quant.
The same pattern shows up, less dramatically, across quantization levels and across small instruct-tuned models generally. Ablation just makes it easy to see.
What the metric would look like
Deliberately simple, because the value is in having it at all:
Two comparisons make it immediately useful: ablated-vs-base at fixed quant, and one model walked down a quant ladder. Both are things people currently decide by feel.
deepeval already has the harness shape for this, and
JsonCorrectnessMetricis adjacent — but as I read it, it is scoped to schema validity rather than to contract adherence generally, and it is not framed as a model-selection signal. The reframing is the useful part: this is the axis that predicts whether an agent built on this model will work, and it is orthogonal to every quality metric in the list.Happy to contribute compliance-rate data across the variants I serve if that would help calibrate it, and happy to be pointed at this already existing under a name I have missed.
All reactions