Fix evaluator token metrics not persisted in red teaming results

[slister1001]

Problem

Since the transition from the old evaluation-processor path to the Foundry/RAIServiceScorer path, evaluator token metrics (\promptTokens, \completionTokens) from the sync eval API response are silently dropped from red teaming output items.

In version 1.14.0, the old _evaluation_processor\ stored the full sync eval response as _eval_run_output_item, which carried \properties.metrics\ through to the output. The new Foundry path evaluates inline via \RAIServiceScorer\ and sets \�valuation_result: None, bypassing that extraction path entirely. A fallback mechanism via \scorer_token_usage\ was added, but it has a camelCase/snake_case key mismatch bug.

Root Cause

_extract_token_usage()\ in _rai_scorer.py\ searches for snake_case keys (\prompt_tokens, \completion_tokens) but the sync eval API returns camelCase keys (\promptTokens, \completionTokens). This causes the extraction to silently return an empty dict, so \scorer_token_usage\ is never set and the fallback in _result_processor.py\ never fires.

Both extraction paths are affected:

• Path 1 (\sample.usage): Raw JSON responses use camelCase keys
• Path 2 (
  esults[].properties.metrics): Also uses camelCase keys

Fix

Updated _extract_token_usage()\ to handle both camelCase and snake_case keys, normalizing to snake_case for downstream consumers. Uses a shared _extract_from_dict()\ helper that checks both key styles.

Tests

Added 2 new tests:

• \ est_score_metadata_includes_token_usage_from_sample_camelcase\ — verifies camelCase keys from raw JSON \sample.usage\
• \ est_score_metadata_includes_token_usage_from_result_properties_camelcase\ — verifies camelCase keys from \properties.metrics\

All 4 token usage tests pass. Black formatting verified.

[Copilot]

Pull request overview

Fixes red teaming output persistence of evaluator token metrics by updating the Foundry RAIServiceScorer token-usage extraction to support both camelCase (raw JSON) and snake_case (SDK model) response keys, ensuring downstream consumers consistently receive snake_case.

Changes:

• Updated _extract_token_usage() to accept both camelCase and snake_case token-usage keys and normalize output to snake_case.
• Added unit tests covering camelCase token usage extraction from both sample.usage and results[].properties.metrics.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_foundry/_rai_scorer.py	Normalizes token usage extraction across camelCase/snake_case sync-eval response shapes.
sdk/evaluation/azure-ai-evaluation/tests/unittests/test_redteam/test_foundry.py	Adds regression tests for camelCase token usage extraction paths.