[Evaluation] Additional red team e2e tests

[slister1001]

Description

Please add an informative description that covers that changes made by the pull request and link all relevant issues.

If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[Copilot]

Pull request overview

Adds additional end-to-end coverage for RedTeam “Foundry” execution and aligns a few red team internals/outputs with expected contracts and error semantics.

Changes:

• Expanded test_red_team_foundry.py with new Foundry e2e scenarios (model-config targets, agent targets, new risk categories, multi-turn strategies, and contract error paths).
• Fixed Foundry baseline objective cache lookup keying to use the same risk-category→objective mapping as the generator path.
• Treated leftover pending/running statuses as terminal failures when producing final run status, and surfaced RAI evaluation service “error outcome” as undetermined instead of attack success.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
sdk/evaluation/azure-ai-evaluation/tests/e2etests/test_red_team_foundry.py	Adds substantial Foundry red team e2e coverage across targets, strategies, and risk categories.
sdk/evaluation/azure-ai-evaluation/tests/conftest.py	Updates OpenAI/test-proxy routing configuration used by recordings/playback.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_result_processor.py	Adjusts final run-level status determination semantics after scan completion.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_red_team.py	Fixes baseline objective cache key mismatch in Foundry execution path.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_foundry/_rai_scorer.py	Detects evaluation-service error outcomes and raises so PyRIT marks results as undetermined.
sdk/evaluation/azure-ai-evaluation/CHANGELOG.md	Documents the bug fixes included in this PR.

[nagkumar91]

Code Review

🟡 XPIA agent fallback catches all exceptions silently (Medium)

In _red_team.py, the XPIA prompt fetch for agent targets catches Exception and logs only at debug level:

except Exception as agent_error:
    if target_type_str == "agent":
        self.logger.debug(f"Agent-type XPIA prompt fetch failed ({agent_error}), falling back to model-type")

This swallows ALL exceptions — network failures, bugs, timeouts, JSON parse errors — not just expected 404/auth errors. For agent targets, these errors vanish into debug logs making troubleshooting very difficult.

Fix: Catch specific expected exception types (e.g., HttpResponseError) and re-raise unexpected ones, or at minimum log at warning level.

---

🟡 Agent credential fallback — bare except at debug level (Medium)

In _generated_rai_client.py, the agent token fallback silently swallows all errors:

try:
    token = self.token_manager.credential.get_token(TokenScope.DEFAULT_AZURE_MANAGEMENT.value).token
    headers["aml-aca-token"] = token
except Exception:
    self.logger.debug("Could not set aml-aca-token from existing credential", exc_info=True)

If the credential is misconfigured or expired, execution continues without the token header. The subsequent service call will fail with a cryptic auth error that's very hard to trace back to this swallowed exception.

Fix: Log at warning level instead of debug.

---

🟡 _fallback_response returns [] on missing request (Medium)

In _rai_service_target.py, when neither message nor prompt_request is in retry kwargs:

request = retry_state.kwargs.get("message") or retry_state.kwargs.get("prompt_request")
if request is None:
    logger.warning("_fallback_response: no 'message' or 'prompt_request' in retry kwargs")
    return []

This is the retry error callback after 5 attempts are exhausted. Returning [] breaks the List[Message] contract — callers accessing response[0] will get IndexError. Consider raising an exception instead since this represents an unrecoverable error state.

---

🟠 Import inside method in _rai_scorer.py (Low)

async def _score_piece_async(self, ...):
    from azure.ai.evaluation._common.rai_service import (
        _SYNC_TO_LEGACY_METRIC_NAMES, _LEGACY_TO_SYNC_METRIC_NAMES,
    )

This runs on every scoring call. Move to module level for clarity and minor perf improvement.

[nagkumar91]

All four issues from previous review are addressed:

1. XPIA agent fallback — log level raised from debug to warning ✅
2. Agent credential fallback — log level raised from debug to warning ✅
3. _fallback_response empty list — now raises RuntimeError instead of returning [] ✅
4. Import inside method — moved to module-level in _rai_scorer.py ✅

LGTM.