Fix MTEBEvaluator: device mapping, padding-free inference, last-token pooling, L2 normalization

[natke]

Fixes several issues in the MTEBEvaluator for embedding model evaluation:

Device mapping

Maps Olive's Device.GPU ("gpu") to PyTorch's "cuda" when initializing SentenceTransformer in the HF evaluation path. Also handles indexed devices (e.g. gpu:0 → cuda:0).

Padding-free inference for GenAI

GenAI's Generator does not accept an attention_mask, so padded batches produce contaminated hidden states via self-attention to padding tokens. Fix: process each sentence individually with only its real tokens, eliminating padding entirely.

Last-token pooling

Replaced mean pooling with last-token pooling in the GenAI and ORT wrappers to match models like Qwen3-Embedding that use pooling_mode_lasttoken=True.

L2 normalization

Added L2 normalization after pooling in the base encode() method, matching the 2_Normalize module in the SentenceTransformer pipeline.

Results

These fixes close the score gap between HF and GenAI evaluation:

• Before: HF 0.785 vs GenAI 0.651 (STS17 main_score)
• After: HF 0.785 vs GenAI 0.785

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Improves correctness and consistency of MTEB embedding evaluation across HF / ORT / GenAI backends by aligning device strings, pooling strategy, padding behavior, and embedding normalization.

Changes:

• Map Olive gpu / gpu:<idx> device strings to PyTorch cuda / cuda:<idx> for SentenceTransformer initialization.
• Switch ORT + GenAI wrappers from mean pooling to last-token pooling; avoid padding in GenAI by encoding each sequence using only real tokens.
• Add L2 normalization to ORT embeddings to match SentenceTransformer’s Normalize module.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
olive/evaluator/olive_evaluator.py	Normalizes Olive device strings to PyTorch-compatible cuda strings in HF evaluation path.
olive/evaluator/mteb_ort.py	Adds L2 normalization, switches pooling to last-token, and removes padding from GenAI inference by per-sample processing.