Azure AI Evaluation SDK v1.13.7: start_red_team_run() produces runs with target=null, causing completed external scans to be invisible in Foundry portal #44314
Description
Environment
SDK & package: Azure AI Evaluation SDK azure-ai-evaluation v1.13.7
Python: 3.10.x (also observed on 3.9.x)
Auth: DefaultAzureCredential()
Project architecture:
Legacy Hub: MachineLearningServices/workspaces (primary)
Also tested with Foundry: CognitiveServices/accounts via project endpoint
Region: (e.g., eastus)
OS: Windows 11 / Ubuntu 22.04 (observed on both)
Portal: Azure AI Foundry (Evaluations → AI red teaming)
Description
When initiating AI Red Teaming scans against non‑Azure‑hosted targets (external callbacks, Kong, LM Studio, etc.) using RedTeam.start_red_team_run() / scan(), the SDK uploads a RedTeamUpload that contains a displayName but leaves the target field as null.
The run completes successfully and is retrievable via az rest.
The run does not appear in the Foundry portal’s AI red teaming list, which appears to filter on target.
Portal‑initiated runs (GUI) against Foundry‑deployed models populate target and display correctly.
All 10 SDK‑initiated external runs show target=null and are invisible despite status=Completed.
This behavior is reproducible primarily in legacy Hub projects; the Foundry architecture using AIProjectClient with model configs sets target and shows up, but the external callback path remains affected.
Impact
Portal visibility gap: Completed external scans are invisible in Foundry UI, breaking centralized reporting, review workflows, and compliance evidence gathering.
Operational friction: Teams must use az rest/raw APIs to find runs; portal drill‑downs (risk breakdown, row‑level attack/response) are unavailable.
CI/CD & governance: Automation that expects portal surfacing of evaluation artifacts cannot rely on SDK for external targets.
Repro steps
Create a RedTeam using the Evaluation SDK pointing to a Hub project (legacy ML workspace) or Foundry project endpoint:
Pythonfrom azure.ai.evaluation.red_team import RedTeam, RiskCategoryfrom azurefrom azure.identity import DefaultAzureCredentialazure_ai_project = { "subscription_id": "...", "resource_group_name": "...", "project_name": "..."} # or Foundry project URLred_team = RedTeam( azure_ai_project=azure_ai_project, credential=DefaultAzureCredential(), risk_categories=[RiskCategory.Violence, RiskCategory.HateUnfairness])# External (non‑Azure‑hosted) callback targetasync def external_callback(prompt: str) -> str: return "external system response"Show more lines
Confirm completion via CLI:
Shellaz rest --method get --url "https://{account}.services.ai.azure.com/api/projects/{project}/redteams/runs/{runId}"Show more lines
Observe payload:
JSON{ "id": "4b4ce898-045f-48b3-9335-9b18c338a97f", "displayName": "scan_20251205_195954", "target": null, "status": "Completed"Show more lines
Open Foundry portal → Project → Evaluations → AI red teaming.
Actual: Run does not appear.
Expected: Completed run should be listed with drill‑down.
Expected behavior
For any completed red team run—including external callback targets—the SDK should persist a non‑null target descriptor (or the portal should not require target for visibility). The run should appear in the Foundry portal’s AI red teaming tab with full drill‑down.
Actual behavior
RedTeamUpload.target == null for SDK‑initiated external scans.
Portal hides those runs; only GUI‑initiated or SDK runs with explicit target configs display.
Additional observations & references
Foundry docs show portal visibility when a target is set via AIProjectClient and model configuration; external callback examples do not demonstrate a persisted target value, aligning with our observations.
Cloud run with AIProjectClient and target config: Learn: “Run AI Red Teaming Agent in the cloud” (payload includes target) [AIProjectC...soft Learn | Learn.Microsoft.com]
Local agent & external callback pattern (no persisted target shown): Learn: “Run AI Red Teaming Agent locally” [Run AI Red...soft Learn | Learn.Microsoft.com]
Portal viewing guidance (assumes run is logged with usable metadata): Learn: “View AI red teaming results” / “Run scans…”
Hypothesis
The SDK’s serialization path for external callback targets does not construct a target object (e.g., ExternalCallback or GenericEndpoint), leaving it null.
The Foundry portal UI filters/queries rely on target to display runs and do not include runs with target=null.
Requested fix
SDK enhancement:
Provide a target schema for external/non‑Azure targets (e.g., type: ExternalCallback, with identifier fields), and populate it when scan(target=callable) is used.
Alternatively, allow setting a target descriptor explicitly in RedTeam.scan()/start_red_team_run() for callbacks.
Portal tolerance (optional):
Do not filter out runs where target == null; display them with a generic “External target” label, so results remain discoverable.
Backfill/patch API (nice to have):
Support updating run metadata to attach a target after creation, enabling visibility for already completed runs.
Workarounds we tested
AIProjectClient with model configuration → visible (sets target).
External callback with SDK → invisible (no target).
Manual review via az rest → possible but not viable for teams relying on portal analytics.
Logs / payloads
Example run (IDs sanitized above). We can provide full request/response traces privately if needed.
Severity
High for customers using external endpoints: prevents usage of Foundry portal for red teaming result review, reporting, and governance.