-
Notifications
You must be signed in to change notification settings - Fork 0
RBAC 401 Investigation
title: WI-11 Resolution and RBAC 401 Investigation description: Verified recovery of the hosted runtime and MCP path, with the historical diagnosis retained for reference.
WI-11 is resolved for this implementation. Run 34178081808 passed the complete staging-to-production pipeline on 2026-09-08 UTC. Staging version 6 completed eight captures and all 21 model-judge checks; production version 34 became active and passed its exact-version smoke and exception checks. See Release Evidence for source artifacts, hashes, and screenshots.

| Layer | Resolution or verified control |
|---|---|
| Historical model 401 | A later v32 redeployment first demonstrated model-call recovery with the same runtime principal; the full v34 production release now passes. No server-side root-cause report establishes why the earlier denial stopped |
| Runtime authorization | Account-scoped Foundry User and Cognitive Services OpenAI User are checked for the actual instance principal; Blueprint identity is not a valid runtime RBAC substitute |
| Legacy tool-resolution 404 | Replaced unsupported tools/resolve calls with the versioned security-tools/versions/1/mcp?api-version=v1 endpoint |
| Toolbox connection error | Corrected connection category from GenericHttp to RemoteTool, retaining None upstream auth for synthetic public tools |
| MCP protocol/auth | Updated MCP transport and negotiation; fresh Entra tokens for the AI audience; strict per-specialist tool allowlists |
| Incomplete specialist evidence | Propagated original incident context; required independent anomaly lookups; aligned synthetic identifiers and fixture contracts |
| Quality gate defects | Composer-specific judging context, separator-free reports, bounded eventual-output retrieval, strict unchanged thresholds |
| Production pre-deploy failure | Discovered the remotely routed version before azd show on fresh CI runners; promoted evaluated MCP digests |
The old cache/replica explanation below was a hypothesis, not a confirmed Azure root cause. A working manual agent narrowed the investigation but did not prove that all hosted authentication failures were platform defects. Model authentication and MCP resolution were distinct failures; the transport change should not be described as the proven fix for the 401.
Support request 2609040400007027 remains a historical reference. This documentation update
does not claim that Microsoft supplied an RCA, closed the support case, or changed the state
of any external work item. Attach the successful run to support follow-up as appropriate.
The remaining sections preserve what was observed and believed during the outage. Their present-tense failure descriptions and escalation instructions are historical, superseded by the verified resolution above; do not follow them as current operating guidance.
azd ai agent invoke threat-assessment-agent intermittently, and now
consistently, fails with a 401 PermissionDenied error when the hosted agent
calls Azure OpenAI chat completions:
ERROR: agent error (server_error): Error code: 401 - {'error': {'code': 'PermissionDenied',
'message': 'The principal `59a21b26-...` lacks the required data action
`Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action`
to perform `POST /openai/deployments/{deployment-id}/chat/completions` operation.'}}
Important
The RBAC configuration on the customer side is correct and stable. Every angle of investigation below confirms the role assignments, scopes, and inheritance are exactly as they should be. The remaining explanation is a platform-side issue, not a configuration gap.
The Foundry Playground's live Log stream surfaces the underlying Python process's stdout, including the raw exception. This is the actual traceback captured at agent version 19:

The same raise self._make_status_error_from_response(...) failure was
captured again at agent version 31 (a different day, after a full
re-provision in between), confirming the failure is not a one-off:

The same failure was also observed at version 17, the same day as the version-19 capture above, confirming it is not tied to any single agent version:

Both deploy-and-evaluate.yml and hosted-agent-cd.yml had push triggers,
so a single push could kick off two overlapping provisioning runs against the
same environment. Removed the push triggers from both workflows
(workflow_dispatch only) to eliminate the race and the duplicate-run
complaint that came with it. Re-ran the smoke test in isolation: still failed
every attempt, and the error message changed shape (see below). Ruled out as
the sole cause.
Checked the hosted agent's per-instance managed identity
(59a21b26-5c3a-42aa-ad7f-05fe701fb25f) against the Cognitive Services
account scope. It has both:
-
Foundry User(assigned 2026-09-03T20:41 UTC) -
Cognitive Services OpenAI User(assigned 2026-09-04T01:56 UTC)
Both assignments correctly inherit down to the project scope and the
gpt-4o-mini model deployment scope, confirmed with
az role assignment list --include-inherited. Ruled out.
Each hosted-agent version could in theory provision a new per-version identity, leaving an older RBAC assignment orphaned. Compared the identity across agent versions 9 and 31 of the same agent: both report the same Instance Identity Principal ID and the same Blueprint reference. The instance identity is stable across versions for a given agent. Ruled out.
azd ai agent show lists a second identity per agent version, the Blueprint
Principal ID (e227baaa-48b3-4505-a19f-d840bfa2400d, appId
809d5ac4-8b64-4d0c-895a-c33f3e404b93). It had zero role assignments, so an
attempt was made to grant it Cognitive Services OpenAI User directly. Azure
RBAC rejects this outright:
ERROR: (PrincipalTypeNotSupported) Principals of type
#microsoft.graph.agentIdentityBlueprintPrincipal cannot validly be used in
role assignments.
The blueprint is a per-agent identity template used by the platform to mint per-version and per-instance identities. It is not a runtime credential holder and is not a valid target for role assignments. Ruled out.
Reproduced the 401 locally with azd ai agent invoke against the same
environment, outside of any GitHub Actions run. Same error, same principal.
Ruled out as a CI-only artifact.
At the time of a failing local test call, the Cognitive Services OpenAI User assignment was already about 10 hours old. Normal Azure RBAC
propagation completes within 30 minutes, so 10 hours categorically rules out
propagation lag as an explanation.
Forced a complete re-provision that ignores stored Bicep deployment state:
azd provision --no-prompt --no-stateThe ARM deployment succeeded with zero failed operations
(az deployment group show and az deployment operation group list
both confirm this). A follow-up plain azd provision --no-prompt then
reported no changes, confirming the state was consistent afterward.
azd ai agent show confirmed the agent itself was unaffected: same version,
same instance identity, same blueprint. Retesting azd ai agent invoke
immediately after still failed with 401, but the error message text changed:
| When | Message |
|---|---|
| Before re-provision |
The principal 59a21b26-... lacks the required data action Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action to perform POST ... (specific, names the exact data action) |
| After re-provision |
Principal does not have access to API/Operation. (generic, no principal ID or data action named) |
Same identity, same role assignments, same scope, yet the platform returned a different error shape for the same operation. A static, deterministic authorization check would not do this. This is much more consistent with multiple backend replicas or edges holding inconsistent or cached views of the RBAC state, some serving an older detailed-error code path and others a newer generic-denial code path. Re-provisioning is not the lever here.
A same-day azd provision run also failed to provision
mcp-anomaly-server and mcp-defender-server with
ContainerAppOperationError: Operation expired, suggesting a stuck or
conflicting Container App revision. This is unrelated to the RBAC 401 and
turned out to be transient: both apps now show Succeeded/Running with a
single clean active revision each, and the most recent full re-provision
completed with zero failed operations. No cleanup action was needed.
Every angle of "is the RBAC correctly configured" has been exhausted and confirmed correct. The remaining explanation is a platform-side issue, most likely the Azure OpenAI or Cognitive Services data plane's own internal authorization cache for this account and hosted-agent combination not honoring or refreshing against the real ARM role assignments.
This is consistent with a separate, already-documented platform-preview gap
on the same account: the /agents/{agent}/tools/resolve data-plane endpoint
(api-version 2025-11-15-preview) returns an empty-body 404 unconditionally
for every connection-id and agent-name combination tried, which blocks the
specialist nodes from actually invoking their bound MCP tools through the
Foundry Toolbox. A graceful-degradation fix was implemented so the agent
still produces a coherent report with an explicit limitations note when tool
resolution is unavailable, rather than crashing the request.
Note
Do not keep re-testing RBAC configuration or re-provisioning as a fix for the 401. Both have been proven not to be the lever.
If this needs to go to Microsoft support or the Foundry product team, the following evidence establishes the customer-side configuration is correct and the failure is inconsistent across otherwise identical requests:
Trace IDs:
-
43e33d30813e5e493477c905dca2081c: local invoke before the fresh re-provision, specific data-action denial message. -
a5d5b5871a9a248da0c10d73fd00327b: local invoke after the fresh re-provision, same identity, generic denial message.
Role assignment IDs (both at the Cognitive Services account scope, both correctly inherited to project and deployment scopes):
-
e1ba7ccb-f6cd-4999-a14a-a5cd37f2b537:Foundry User -
db566441-1f4a-48b4-a2f8-5e645ec97415:Cognitive Services OpenAI User
Capture the actual failing request's token claims (aud, appid) from a
live 401 to confirm which identity the hosted-agent runtime actually
presented, rather than continuing to infer it from the deploy-time identity
listing. This remains the only untried diagnostic lever before escalating to
Microsoft support.
A later session re-confirmed the 401 with azd already at the latest
version (1.33.0), then went further than the RBAC audit above by actually
widening the hosted agent's instance identity permissions, to rule out any
remaining doubt that this is a scope or role-definition problem.
Caution
The role assignments below were created for diagnostic purposes only and
were removed again in the same session once the test concluded. Do not
assume they are still present — always check current state with
az role assignment list before relying on this section.
Temporarily granted the hosted agent's instance identity
(59a21b26-5c3a-42aa-ad7f-05fe701fb25f) each of the following, one at a
time, at both the Cognitive Services account scope and the Foundry project
scope:
Cognitive Services ContributorCognitive Services OpenAI ContributorAzure AI Administrator
azd ai agent invoke was re-run after each grant, with a wait for RBAC
propagation before each retry. Every combination still returned the same
401. All three temporary role assignments were then removed, restoring the
identity to its original Foundry User + Cognitive Services OpenAI User
state. This confirms the failure is not resolvable by adding more
permissions from the customer side, no matter how broad — reinforcing the
platform-bug conclusion above.
To rule out the two MCP servers (mcp-defender-server,
mcp-anomaly-server) as a contributing factor, scripts/test_mcp_servers.py
was run directly against both deployed Container Apps endpoints, bypassing
the hosted agent entirely. Both get_device_risk/list_vulnerabilities
(defender) and the anomaly-scoring tool responded correctly with their mock
data. This isolates the 401 to the hosted-agent-to-Azure-OpenAI call path
only; the MCP tool infrastructure itself is not implicated.
As the most conclusive test, a brand-new agent was created directly in the
Foundry portal UI ("Build an agent" flow, NOT the azd ai agent/LangGraph
hosted-agent path) in the same project, same tenant, same subscription:
- Name:
threat-assessment-manual-poc - Auto-provisioned its own
gpt-5Global Standard model deployment (portal-native agents get a dedicated deployment; they cannot reuse a project's existing connected-resource deployments) - Instructions set to a threat-assessment system prompt
- Both
defender-connandanomaly-connMCP tool connectors (already registered in this project from the original hosted-agent deployment) added as Tools
This agent chatted successfully on the first try, with zero 401 or permission errors:


Going further, the manual agent was also asked to actually invoke the MCP
tools ("Using your available tools, check the device risk for device-001
and list its vulnerabilities."). Foundry's Playground enforces a
human-in-the-loop approval gate for non-Foundry-native (MCP) tool calls; after
approving each of the two tool calls (get_device_risk, then
list_vulnerabilities), the agent returned a fully tool-grounded, correct
answer:
- Hostname:
yyz-gate-kiosk-01 - Platform: Windows10
- Risk score: High
-
CVE-2024-21306— Critical, CVSS 9.8 -
CVE-2024-30040— High, CVSS 8.1

This is the strongest evidence to date that the 401 is isolated specifically to the hosted-agent (LangGraph/Blueprint-identity) runtime path, not to the Foundry project's RBAC, model deployment access, or MCP tool infrastructure — all of which work correctly for a portal-native agent in the exact same project. See Manual Agent Workaround for the full walkthrough and how to reproduce it.
The deploy-and-evaluate.yml GitHub Actions pipeline's staging smoke-test
step had a retry loop (MAX_ATTEMPTS=8, 30s sleep between attempts) added
on the working theory that the 401 might be a transient RBAC-propagation
delay on a freshly deployed agent instance. The next real CI run after that
change (run 33830260451, commit cb8ff04) exercised all 8 attempts over
approximately 4 minutes and received the identical 401 PermissionDenied
error on every single attempt, each with a distinct trace ID (ruling out a
cached/idempotent failure artifact):
Attempt 1/8 ... 401 PermissionDenied
Attempt 2/8 ... 401 PermissionDenied (retrying, possible RBAC propagation delay)
...
Attempt 8/8 ... 401 PermissionDenied
##[error]azd ai agent invoke failed after 8 attempts (exit 1)
This independently corroborates the broadened-RBAC test above: waiting longer, or retrying more, does not resolve the failure. The workflow's retry budget and messaging were corrected accordingly (reduced to 3 attempts / 15s, now only guarding against a genuine one-off blip, with the comments pointing at this page and WI-11 instead of the disproven "propagation delay" theory) to stop burning CI minutes on a strategy proven ineffective against this specific platform bug.
A dedicated workflow_dispatch run was used to validate the corrected
retry logic. It first uncovered and required fixing an unrelated
deploy-blocking bug in azure.yaml (a duplicate connections: entry for
anomaly-conn/defender-conn under the security-tools toolbox, tracked
as WI-19 — see the Planning Log), which had been silently preventing
azd deploy from completing for staging. Once fixed, the pipeline reached
the smoke-test step for the first time since the retry-logic fix, with this
result:
--- Agent response (attempt 1/3) ---
ERROR: agent error (server_error): Error code: 401 - {'error': {'code':
'PermissionDenied', 'message': 'The principal `59a21b26-...` lacks the
required data action `Microsoft.CognitiveServices/accounts/OpenAI/
deployments/chat/completions/action` to perform `POST /openai/deployments/
{deployment-id}/chat/completions` operation.'}}
##[warning]Attempt 1 failed (exit 1); retrying in 15s.
--- Agent response (attempt 2/3) ---
ERROR: agent error (server_error): Error code: 401 - {'error': {'code':
'PermissionDenied', 'message': 'Principal does not have access to
API/Operation.'}}
##[warning]Attempt 2 failed (exit 1); retrying in 15s.
--- Agent response (attempt 3/3) ---
ERROR: agent error (server_error): Error code: 401 - {'error': {'code':
'PermissionDenied', 'message': 'Principal does not have access to
API/Operation.'}}
##[error]azd ai agent invoke failed after 3 attempts (exit 1) -- see
WI-11, this is a known persistent platform 401, not a transient RBAC
delay
Total elapsed time for the smoke-test step: ~46 seconds (down from ~4 minutes), with 3/3 attempts still failing identically as expected. The corrected diagnostic messaging rendered exactly as intended, correctly attributing the failure to WI-11 rather than a transient RBAC delay. This closes out validation of the retry-logic fix; the only remaining open action for WI-11 itself is filing an Azure support ticket (requires Azure Support/portal access this agent does not have).
A Microsoft Azure support request was opened via Help + support → New support request in the Azure Portal, using all of the evidence gathered above:
| Field | Value |
|---|---|
| Support request ID | 2609040400007027 |
| Title | "Support Request: issue with foundry" |
| Created | Fri, Sep 4, 2026 2:56:20 PM |
| Severity | C |
| Support plan | Azure Internal |
| Contact method | Email (emknafo@microsoft.com) |
| Resource |
aif-air-canada-threat-assessment-poc (Cognitive Services account) |
The description submitted was the full evidence summary from this page:
affected principal (59a21b26-5c3a-42aa-ad7f-05fe701fb25f), both error
message variants, the confirmed-correct role assignment IDs, the
10+-hour-old-assignment timing argument against propagation lag, the
zero-401 portal-native-agent isolation test, the 8/8-retry CI evidence, and
both reproduction trace IDs (43e33d30813e5e493477c905dca2081c,
a5d5b5871a9a248da0c10d73fd00327b).
Note
An IcM incident (https://portal.microsofticm.com/imp/v3/incidents/details/757637166)
was also referenced during triage and may already be tracking this same
underlying issue — worth checking for updates there in addition to the
Azure support request above.
This is now the authoritative tracking record for WI-11 follow-up. Check
support request 2609040400007027 for status/resolution before doing any
further self-service diagnosis on this issue.
Microsoft Support's first (AI-generated) reply hypothesized a root cause of
missing data-plane permission: the service principal lacking the
Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action
data action, and recommended (re-)assigning Cognitive Services OpenAI
User at the resource scope, then verifying with:
az role assignment list --assignee 59a21b26-5c3a-42aa-ad7f-05fe701fb25f `
--scope /subscriptions/64c3d212-40ed-4c6d-a825-6adfbdf25dad/resourceGroups/rg-air-canada-threat-assessment-poc/providers/Microsoft.CognitiveServices/accounts/aif-air-canada-threat-assessment-pocThis exact check was re-run live on 2026-09-04 and already passes — the role was assigned before the ticket was even filed:
| roleDefinitionName | scope | principalType |
|---|---|---|
| Foundry User | .../accounts/aif-air-canada-threat-assessment-poc |
ServicePrincipal |
| Cognitive Services OpenAI User | .../accounts/aif-air-canada-threat-assessment-poc |
ServicePrincipal |
A read-only check of the resource group's Activity Log (az monitor activity-log list, --status Failed, last 2 days) was also attempted to
look for the specific denied data action support referenced — it surfaced
only unrelated List Keys failures (likely a different caller blocked by
DisableLocalAuth), confirming the Activity Log does not capture
data-plane /chat/completions authorization decisions and can't be used
to self-diagnose the missing data action support described.
Conclusion: this first response is a generic/templated answer that re-covers ground already proven correct in this investigation (see "Final validation" above) rather than addressing the actual reported symptom. It should be treated as first-line triage, not a resolution — the reply sent back to support (below) supplies the already-passing verification output and asks for escalation to an engineer who can inspect server-side authorization logs for the specific denied data action.
Note
The tracking ID printed in the email body (2609040040007027) differs
by a transposed digit pair from the number shown at ticket creation
(2609040400007027). Confirm in the Azure Portal support-request list
which one matches before replying, in case this is a typo rather than a
different ticket.
Thanks for the quick response. The recommended check is already passing —
Cognitive Services OpenAI Userwas assigned to service principal59a21b26-5c3a-42aa-ad7f-05fe701fb25fat the exact resource scope you listed (aif-air-canada-threat-assessment-poc) well before this ticket was filed, and I re-verified it again just now with the sameaz role assignment listcommand you provided — same result, role present.Foundry Useris also assigned at the same scope. This was already checked and re-checked multiple times before opening this ticket (RBAC propagation is not a factor; the assignment is 10+ hours old). Could this be escalated to an engineer who can check server-side authorization/audit logs for the specific denied data action on trace IDs43e33d30813e5e493477c905dca2081canda5d5b5871a9a248da0c10d73fd00327b? A side-by-side portal-native agent using the same identity and model deployment succeeds with zero 401s, which points to something specific to the hosted-agent instance-identity code path rather than a missing RBAC assignment.
Support followed up with a written case summary (troubleshooting steps they logged on their side) and 4 outstanding questions, plus a close-or-escalate choice. Since the issue is not resolved, the correct choice is "I'm still having an issue and need further assistance" (the escalate link), not close. Answers gathered live on 2026-09-04 to attach to that escalation:
| # | Support's question | Answer (evidence) |
|---|---|---|
| 1 | Does the assigned role explicitly include the exact data action Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action? |
Yes, confirmed. az role definition list --name "Cognitive Services OpenAI User" against this subscription shows Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action explicitly listed in dataActions (role def 5e0bd9bd-7b93-4f28-af87-19fc36ad61bd, last updated 2025-04-29). |
| 2 | Is the token acquired at runtime correctly scoped to the OpenAI endpoint audience (https://cognitiveservices.azure.com/)? |
Cannot be verified from the customer side. The hosted agent's Instance Identity token acquisition happens inside Microsoft's managed Foundry hosting layer, not in this repo's code (graph.py's _get_llm_client() only builds the client from env vars; it does not control the platform's own credential/token plumbing for the hosted runtime). This is the key item for Microsoft engineering to check server-side. |
| 3 | Are there any conditional access or resource-level policies affecting data actions on the OpenAI resource? |
No blocking policy found — checked from multiple angles, including the "policy silently overrides the displayed value" scenario. disableLocalAuth: true (confirmed via az cognitiveservices account show) blocks only API-key auth, not AAD/managed-identity token auth (in use here). networkAcls is null (no default-deny rule) and privateEndpointConnections is empty — no network-layer block. publicNetworkAccess: Enabled was independently re-verified against the tenant's Azure Policy state (az policy state list --resource <this account>): only two custom policies from the MCAPSGovDeployPolicies management-group initiative have ever evaluated this resource — CognitiveServices_LocalAuth_Modify and CognitiveServices_Diagnostics_Enable (both Compliant) — neither touches network access. The initiative does contain an AIFoundryHub_PublicNetwork_Modify policy that force-disables public network access, but its rule only targets Microsoft.MachineLearningServices/workspaces resources with kind == 'Hub'; this deployment uses the newer Cognitive Services "AIServices" account + nested .../accounts/projects model (confirmed via az resource list on the resource group — no Microsoft.MachineLearningServices/workspaces resource exists at all), so that policy cannot apply here. Tenant Conditional Access policies (az rest against Microsoft Graph identity/conditionalAccess/policies) were also enumerated: none of the 5 enabled/reporting policies target service principals or workload identities (conditions.clientApplications is null on all of them) — they're all human-user-focused (MFA, sign-in risk, security info registration). |
| 4 | Could there be a mismatch between what the portal/CLI report and the principal's effective permissions at request time? |
No customer-side evidence of drift, and this is effectively the platform-bug hypothesis this investigation has been building toward. Every check (portal IAM, CLI az role assignment list, the role definition's own dataActions, and now Azure Policy state/definitions for both RBAC and network settings) agrees the assignment and access configuration are correct; the discrepancy — if any — must be inside Microsoft's own effective-permission evaluation for the hosted-agent Instance Identity code path, which only Microsoft engineering can inspect. |
Note
Answer to question 1 directly refutes the first (templated) response's implied doubt about whether the role "includes" the data action — it does, verifiably, in this exact subscription's role definition.
Thanks for the detailed follow-up. Here are direct answers to your 4 outstanding questions, gathered live against this subscription today:
- Does the role include the exact data action? Yes — confirmed from the role definition itself, not just the assignment.
az role definition list --name "Cognitive Services OpenAI User"showsMicrosoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/actionexplicitly listed indataActionsfor role5e0bd9bd-7b93-4f28-af87-19fc36ad61bd(last updated 2025-04-29), and that role is assigned to the service principal at the exact resource scope.- Is the token scoped to the right audience? This is not something I can verify from the customer side — the hosted agent's Instance Identity token acquisition happens inside Foundry's managed hosting layer, not in code I control. This is the item I most need an engineer to check server-side, ideally against trace IDs
43e33d30813e5e493477c905dca2081canda5d5b5871a9a248da0c10d73fd00327b.- Any conditional access / resource policy affecting this? No.
disableLocalAuth: trueonly blocks API-key auth, not the managed-identity/AAD auth in use here.networkAclsisnulland there are no private endpoint connections, so there's no network-layer block either. I also checked Azure Policy directly (not just the resource's displayed properties) to rule out a policy silently overridingpublicNetworkAccess— only two policies have ever evaluated this account (CognitiveServices_LocalAuth_Modify,CognitiveServices_Diagnostics_Enable, both compliant), and the one policy in our environment that can force-disable public network access only targets the olderMicrosoft.MachineLearningServices/workspacesHub resource type, which doesn't exist in this deployment. Tenant Conditional Access policies were also enumerated — none target service principals/workload identities.- Could there be a portal/CLI vs. effective-permissions mismatch? Nothing on my side shows drift — portal IAM,
az role assignment list, and the role definition's owndataActionsall agree. If there is a mismatch, it has to be inside the platform's own effective-permission evaluation for the hosted-agent Instance Identity, which I can't inspect from here.Since the underlying 401 is still happening and none of the above points to a customer-side configuration issue, please escalate this to an engineer who can check the server-side authorization path rather than closing the ticket.