Repository navigation
Is there a task for measuring output-format compliance independently of answer correctness? #4233
Replies: 3 comments 3 replies
|
Treat format compliance as a separate contract test, not as a prerequisite hidden inside answer grading. For each row, I would keep three independent outcomes:
That lets a well-formed wrong answer pass the format test, and a correct answer wrapped in extra prose fail it. The fixtures should pin the output schema, required/forbidden keys, enum values, tool name and argument shape, plus any stop/prefix rule. Keep the validator deterministic and publish the invalid reason ( For the base-versus-abliterated and quant comparison, use the identical prompt/template/decoding setup and report the paired per-item change, stratified by contract type and context length. Repeated samples matter too: one malformed JSON completion in a stochastic setting should not get confused with a systematic loss of adherence. A paired bootstrap over items would give you an interval without turning this into an answer-quality benchmark. I would start with a small task that has synthetic payloads, so correctness can be deliberately varied while the output contract stays fixed. That makes the separation testable before adding domain tasks. |
|
There is already a primitive version of this idea in the harness through GSM8K's For a proper JSON/tool-call evaluation, though, I think it makes sense to explicitly separate contract compliance from answer correctness. The harness seems capable of supporting that through a custom filter or I don't know of prior work measuring format compliance systematically across quantization levels, though, and I haven't looked outside the harness. Whether this belongs in the harness itself is ultimately a maintainer call. |
|
Partly yes for (1). Metric definitions: lm-evaluation-harness/lm_eval/tasks/jsonschema_bench/metrics.py Lines 116 to 157 in d6de816 Task config: README: https://github.com/EleutherAI/lm-evaluation-harness/blob/d6de81643928d653435c431bae19945d41d32520/lm_eval/tasks/jsonschema_bench/README.md Example run (needs Caveats before you rely on it:
Related, For (2), a deterministic validator task with its own metrics is a normal fit for the harness, since For (3), I found nothing in the repo that holds format compliance fixed while varying quantization. I can't speak to work outside the repo. The mechanics are available, though. The (I looked this up in the code with help from an AI assistant; the links are to the lines I checked.) |
Uh oh!
There was an error while loading. Please reload this page.
Disclosure up front: I run open weights in production behind a hosted chat + coding agent (Grunz). Not pitching anything — this is an evaluation methodology question I keep running into and can't resolve from the existing task list.
The problem
There are now thousands of "abliterated" / decensored variants on the Hub — models where a refusal direction has been identified in activation space and projected out of the weights. People pick between them, and between quants of them, largely on vibes, because the standard metrics don't capture what actually breaks.
What I observe consistently, across variants and across families, is this: abliteration degrades instruction-following and output-format adherence well before it degrades knowledge or reasoning.
The model still knows the material. What gets worse is:
The consequence for evaluation is that perplexity and most knowledge/reasoning benchmarks barely move while the thing that actually breaks in production falls off a cliff. A variant can score within noise of its base on MMLU and have a materially higher structured-output failure rate. Anyone selecting a model on the published numbers will pick wrong.
What I'm looking for
A task (or a pattern for writing one) that measures format compliance as a first-class metric, separately from answer quality. Roughly:
The key property is that it must be independent of whether the answer is right. A model that returns a well-formed wrong answer should score 1.0 here; a model that returns a correct answer wrapped in prose that breaks the schema should score 0.
The actual questions
Is there already a task in the harness that does this, that I've missed? I've looked at the generative tasks and at the structured-output adjacent ones, but everything I found scores content correctness with format as an implicit precondition rather than the measured variable.
If not, is this the right place for it? It's not a capability benchmark in the usual sense — it's closer to a robustness or reliability probe. I don't know whether that belongs in the harness or in a separate tool.
Is there prior art on holding format compliance constant while varying quantization? My working hunch is that abliteration damage and quantization damage compound rather than sit side by side, so an abliterated model degrades faster down a quant ladder than its base does. That's a claim that should be measurable and I'd rather measure it than keep repeating it as folklore.
Happy to do the implementation work and submit it if the answer to (1) is no and the answer to (2) is yes. I'd rather ask first than arrive with a PR that duplicates something or lands in the wrong repo.
All reactions