Add vision evaluation metrics (exact_match, relaxed_accuracy, word_sort_ratio)

[jiafatom]

Summary

Add vision evaluation metrics to the Olive evaluator framework, enabling VQA, ChartQA, and OCR model evaluation following the same pattern as #2444 (speech metrics).

• exact_match: Case-insensitive string equality for VQA tasks (ScienceQA, TextVQA, MMMU, ai2d, MathVista)
• relaxed_accuracy: ±5% numeric tolerance for ChartQA (configurable tolerance)
• word_sort_ratio: Word-level overlap ratio for OCR evaluation

Changes

• olive/evaluator/metric.py — Add EXACT_MATCH, RELAXED_ACCURACY, WORD_SORT_RATIO to AccuracySubType enum
• olive/evaluator/accuracy.py — Add ExactMatch, RelaxedAccuracy, WordSortRatio classes
• olive/evaluator/olive_evaluator.py — Add _inference_vision() path in OnnxEvaluator and PyTorchEvaluator; add task-metric validation that throws if metric is incompatible with task type
• olive/data/component/pre_process_data.py — Add vision_vqa_pre_process data component with Olive-style sampling (--limit/--seed)
• olive/data/container/huggingface_container.py — Add vision-vqa, vision-chart-qa, vision-ocr task types
• olive/olive_config.json — Add vision extra dependencies (Pillow)

Task-Metric Validation

Each vision task restricts which metrics are valid:

Task Type	Allowed Metric
vision-vqa	exact_match
vision-chart-qa	relaxed_accuracy
vision-ocr	word_sort_ratio

If a user selects an incompatible metric, a clear exception is raised.

Testing

20 unit tests added covering all three metrics with edge cases (all passing).

Usage

{
  "metrics": [{
    "name": "vision_eval",
    "type": "accuracy",
    "sub_types": [
      {"name": "exact_match", "higher_is_better": true}
    ],
    "data_config": {
      "type": "HuggingfaceContainer",
      "task_type": "vision-vqa",
      "params": {
        "data_name": "HuggingFaceH4/ScienceQA",
        "split": "test"
      }
    }
  }]
}

[Copilot]

Pull request overview

This PR extends Olive’s evaluator framework with three vision-oriented “accuracy” sub-metrics intended for VQA/ChartQA/OCR evaluation, following the existing pattern used for speech metrics.

Changes:

• Adds new AccuracySubType enum values for exact_match, relaxed_accuracy, and word_sort_ratio.
• Implements ExactMatch, RelaxedAccuracy, and WordSortRatio metric classes and adds unit tests for their core behavior.
• Introduces a vision inference path in the evaluator and adds HuggingFace container task mappings plus a new vision_vqa_pre_process component and a vision extra dependency.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
olive/evaluator/metric.py	Adds new accuracy sub-type enum values for vision metrics.
olive/evaluator/accuracy.py	Implements the three new vision metric computations.
olive/evaluator/olive_evaluator.py	Adds vision inference path and task↔metric validation logic for vision tasks.
olive/data/component/pre_process_data.py	Adds a new vision_vqa_pre_process pre-processing component with sampling support.
olive/data/container/huggingface_container.py	Registers new HuggingFace task types mapping to the vision pre-process component.
olive/olive_config.json	Adds a vision extra dependency entry.
test/evaluator/test_accuracy.py	Adds unit tests covering the new vision metrics.

[jiafatom]

Supported Vision Benchmarks

These metrics map to standard public vision benchmarks:

Metric	Benchmarks	Description
exact_match	AI2D, ScienceQA, TextVQA, MathVista, MMMU, InterGPS	Case-insensitive string equality
relaxed_accuracy	ChartQA	±5% numeric tolerance for chart/math answers
word_sort_ratio	OCR	Word-level overlap ratio

Recipe PR

Evaluation recipe using these metrics with Qwen3-VL-2B-Instruct on AI2D: microsoft/olive-recipes#434

[jambayk]

could create a branch directly on the repo and open the PR from there? CI cannot run on forked PRs because of credentials.

[jiafatom]

Closing in favor of a new PR from the upstream repo branch (CI requires non-fork PRs).

[jiafatom]

See #2476