fix(metrics): correct answer parsing and text scoring

[git-jxj]

Summary

This PR fixes three scoring issues that can make correct model outputs receive incorrect or inconsistent evaluation results:

• Multi-select answers written with connectors such as A and B, A/B, or case variants like A AND B were truncated to the first label.
• Per-sample Rouge helpers overwrote each prediction/reference pair and retained only the final pair's score.
• ANLS computed edit distance on normalized strings but divided it by the original, unnormalized string lengths.

Root Cause

The multi-choice parser stopped its label prefix at connector words and did not recognize slash or ideographic-comma separators. Connector normalization was also case-sensitive.

The Rouge helpers assigned every pair's metrics to the same result keys inside the scoring loop, so later pairs replaced earlier ones.

ANLS normalized whitespace and case before computing Levenshtein distance, but used the raw reference and prediction lengths as the denominator. Extra whitespace could therefore change the score without changing the normalized answer.

Changes

• Parse and / or connectors case-insensitively when they occur between valid choice labels.
• Support slash and ideographic-comma separators in multi-select answers.
• Preserve guard behavior for explanatory text such as B, not C.
• Average Rouge-1, Rouge-2, and Rouge-L recall, precision, and F1 values across all successfully scored pairs.
• Return the complete Rouge metric key set with zero values when no pair can be scored.
• Normalize ANLS distance and denominator with the same processed strings.
• Add regression coverage for the affected behavior and boundary cases.

Reproduction

On the unmodified main branch:

ANSWER: A and B -> {'A'}
ANSWER: A/B     -> {'A'}
Rouge-1-F       -> 0.0   # identical first pair, disjoint final pair
ANLS padded     -> 0.875
ANLS plain      -> 0.6667

After this change:

ANSWER: A and B -> {'A', 'B'}
ANSWER: A/B     -> {'A', 'B'}
Rouge-1-F       -> 0.5
ANLS padded     -> 0.6667
ANLS plain      -> 0.6667

The multi-choice guard case remains unchanged:

ANSWER: B, not C -> {'B'}

Validation

python -m pytest \
  tests/test_multi_choices.py \
  tests/metrics/test_rouge.py \
  tests/metrics/test_nlp_metrics.py -q

Result:

35 passed

All configured pre-commit checks also passed:

flake8, ruff check, yapf, trailing whitespace, YAML, end-of-file,
requirements, quote normalization, merge-conflict, and line-ending checks

Scope

This change only affects multi-select answer extraction, per-sample Rouge aggregation, and ANLS length normalization. It does not change benchmark prompts, dataset loading, model generation, target labels, or unrelated metrics.

[Yunnglin]

LGTM!