fix(metrics): make inst_level_* a micro-average over instructions

[arkrolin]

Related to #1671

Summary

This PR makes instruction-level accuracy pool every instruction in the dataset, matching the official IFEval definition, instead of averaging per-prompt ratios.

• inst_level_strict / inst_level_loose are now a micro-average: a prompt carrying 3 instructions weighs 3x one carrying a single instruction.
• Adds a weighted_mean aggregator that honours per-metric weights declared in Score.metadata.
• Applies it to the three benchmarks that shared the defect: ifeval, ifbench, multi_if.
• prompt_level_strict / prompt_level_loose keep their unweighted mean; weighted and unweighted metrics coexist under one aggregator.
• A prompt with an empty instruction list now drops out of the instruction-level metric instead of contributing a spurious 0.

Root Cause

The official implementation (google-research/instruction_following_eval/evaluation_lib.py, print_report at :170) accumulates across the whole dataset and divides once — accumulation at :192-193, output at :211:

instruction_total   += len(instruction_id_list)
instruction_correct += sum(follow_instruction_list)
...
print(f"instruction-level: {instruction_correct / instruction_total}")

evalscope reduced twice instead:

1. evalscope/benchmarks/ifeval/utils.py:131-133 — agg_inst_level_acc collapsed each sample's instruction list into a ratio via sum(items) / len(items).
2. evalscope/metrics/aggregators/aggregators.py — Mean then averaged those ratios with no weights.

The composition is a macro-average over prompts, so a 1-instruction sample and a 3-instruction sample carried equal weight in the final score.

The bias is not a constant offset. It inflates models that do better on lightly-constrained prompts and deflates the opposite, so it can reorder models rather than shifting all of them together. IFEval's per-prompt instruction counts are unevenly distributed between 1 and 3, so the difference does not cancel out.

Worth noting: the correct micro logic already existed in the repository. ifbench/evaluation_lib.py carries the upstream print_report verbatim, but it has no callers anywhere in the package — the code path that actually ran was the agg_inst_level_acc one.

Changes

New aggregator (evalscope/metrics/aggregators/aggregators.py)

• WeightedMean, registered as weighted_mean, reads {metric_name: weight} from Score.metadata[METRIC_WEIGHTS_KEY].
• A metric is weighted only where a weight was declared. Declared-ness — not whether the numbers happen to differ from 1.0 — decides, so a dataset whose prompts all carry exactly one instruction is still handled as a weighted metric.
• Samples that declare no weight for a metric get a neutral 1.0 rather than dropping out of that metric's denominator.
• For a weighted metric, AggScore.num reports the unit total instead of the sample count. This keeps the report layer's micro_mean rollup a true micro-average across subsets, which matters for multi_if (7 language subsets).
• metadata carries weighted, samples, and total_weight so the distinction stays inspectable in a report.
• Malformed weight metadata and all-zero weights fall back to the unweighted mean rather than raising or dividing by zero — a presentation detail must not abort a finished evaluation.

Benchmark wiring

• ifeval, ifbench: aggregation='weighted_mean', weight = len(instruction_id_list), taken from the doc.
• multi_if: each turn_{step}_inst_level_* is weighted by that turn's instruction count, read from the checker's own outputs_strict['instruction_id_list']. parse_result([outputs]) on a single turn had collapsed to that turn's own ratio, which the framework then macro-averaged.

Docs

• docs/{en,zh}/benchmarks/ifeval.md and the adapter description they are generated from now state the micro-average semantics.

weighted_mean was already declared in KNOWN_AGGREGATIONS (api/metric/semantics.py:36) and poly_math already emits this identity, so the identity axis is unchanged and no semantics catalog entry is needed.

Reproduction

The two reduction paths, using the shipped agg_inst_level_acc and the Mean behaviour:

def agg_inst_level_acc(items):            # evalscope/benchmarks/ifeval/utils.py:131
    return sum(items) / len(items) if items else 0
def mean(v): return sum(v) / len(v)       # Mean aggregator over samples

def macro(follow): return mean([agg_inst_level_acc(f) for f in follow])
def micro(follow): return sum(sum(f) for f in follow) / sum(len(f) for f in follow)

minimal: [1 inst ok] + [3 inst, 1 ok]    macro=0.6667  micro=0.5000  delta=+0.1667
model good on 1-inst, bad on 3-inst      macro=0.7500  micro=0.5000  delta=+0.2500
model bad on 1-inst, good on 3-inst      macro=0.2500  micro=0.5000  delta=-0.2500

The sign flips with the model's profile, which is why this can change relative ranking.

End to end through the real ifeval checker, with a 1-instruction prompt and a 3-instruction prompt where 1 of the 4 total instructions is followed:

inst_level_strict    micro (this PR) = 0.2500  num=4
                     macro (before)  = 0.1667  num=2
prompt_level_strict  unchanged       = 0.0000  num=2

0.2500 is the correct value: 1 followed instruction out of 4. num becomes the instruction total, and the prompt-level metric is untouched.

Validation

python -m pytest tests/metrics/aggregators/ -q
# 31 passed  (12 new, 19 existing)

New coverage in tests/metrics/aggregators/test_weighted_mean.py:

• pooling units instead of averaging samples, and explicitly asserting the macro value does not leak through
• num reporting the unit total so the cross-subset rollup stays micro
• unweighted metrics keeping a plain mean and a sample count
• weighted and unweighted metrics coexisting in one Score
• no declared weights, malformed weights, and all-zero weights each falling back
• a checker failure emptying Score.value dropping that sample rather than scoring it 0
• an empty instruction list being excluded rather than contributing a spurious 0
• the emitted identity carrying aggregation='weighted_mean'

ruff check passes on all touched paths. ruff format --check reports whole-file rewrites for any file in a core.autocrlf=true checkout because the configured line-ending is lf; verified clean by re-checking the touched files with LF endings.

Open questions

1. Should the previous macro value be retained as an additional metric (e.g. inst_level_strict_macro) so historical results stay comparable? This PR changes the existing metric in place and adds nothing, on the assumption that matching the official definition is the intent.

[Yunnglin]

LGTM