feat(skills) + docs(tutorials): close data-plane RBAC gap that blocked first eval run#224
Merged
Merged
Conversation
…ortal Creating a Foundry project through the portal only assigns the user 'Foundry User' at the project scope. That role does not cover OpenAI data-plane actions on the parent AI Services account, where chat completions actually live - so every AI-assisted evaluator and every cloud-eval grader fails with PermissionDenied the first time a fresh workspace tries to run eval. Subscription Owner is also insufficient because the built-in Owner role has actions: ['*'] but dataActions: []. All three tutorials (prompt-agent quickstart, hosted-agent quickstart, end-to-end) now document the one-time 'az role assignment create' that grants 'Cognitive Services OpenAI User' at the resource-group scope of the Foundry account, with the exact error signature so future readers can self-diagnose if they skipped it. A future AgentOps Doctor check will detect the missing assignment pre-run; until then, this step is a documented manual prerequisite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The agentops-eval coding-agent skill now resolves the Foundry project endpoint from .azure/<env>/.env or .agentops/.env, looks up the backing AI Services account + resource group with az cognitiveservices account list, fetches the signed-in object ID, and runs an idempotent az role assignment create for 'Cognitive Services OpenAI User' at the resource-group scope BEFORE 'agentops eval analyze' / 'agentops eval run'. This mirrors the new manual step added in the same PR to all three tutorials and keeps the skill experience aligned: users running the skill against a fresh Foundry project no longer hit the 401 PermissionDenied that the portal's default 'Foundry User'-at-project assignment leaves behind. CHANGELOG entry added under [Unreleased]. Plugin skills mirror under plugins/agentops/skills/ regenerated via scripts/sync-skills.ps1 to keep the VS Code extension copy identical. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
placerda
changed the title
placerda
added a commit
that referenced
this pull request
Jun 1, 2026
* feat(skills) + docs(tutorials): close data-plane RBAC gap that blocked first eval run (#224) * docs(tutorials): document data-plane RBAC step missing from Foundry portal Creating a Foundry project through the portal only assigns the user 'Foundry User' at the project scope. That role does not cover OpenAI data-plane actions on the parent AI Services account, where chat completions actually live - so every AI-assisted evaluator and every cloud-eval grader fails with PermissionDenied the first time a fresh workspace tries to run eval. Subscription Owner is also insufficient because the built-in Owner role has actions: ['*'] but dataActions: []. All three tutorials (prompt-agent quickstart, hosted-agent quickstart, end-to-end) now document the one-time 'az role assignment create' that grants 'Cognitive Services OpenAI User' at the resource-group scope of the Foundry account, with the exact error signature so future readers can self-diagnose if they skipped it. A future AgentOps Doctor check will detect the missing assignment pre-run; until then, this step is a documented manual prerequisite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(skills): preflight data-plane RBAC in agentops-eval skill The agentops-eval coding-agent skill now resolves the Foundry project endpoint from .azure/<env>/.env or .agentops/.env, looks up the backing AI Services account + resource group with az cognitiveservices account list, fetches the signed-in object ID, and runs an idempotent az role assignment create for 'Cognitive Services OpenAI User' at the resource-group scope BEFORE 'agentops eval analyze' / 'agentops eval run'. This mirrors the new manual step added in the same PR to all three tutorials and keeps the skill experience aligned: users running the skill against a fresh Foundry project no longer hit the 401 PermissionDenied that the portal's default 'Foundry User'-at-project assignment leaves behind. CHANGELOG entry added under [Unreleased]. Plugin skills mirror under plugins/agentops/skills/ regenerated via scripts/sync-skills.ps1 to keep the VS Code extension copy identical. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: prepare release 0.3.5 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the 401
PermissionDeniedthat every fresh Foundry workspacehit on the first
agentops eval runbecause the Foundry portal onlyassigns
Foundry Userat the project scope, which does not coverMicrosoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/actionon the parent AI Services account where chat completions live.
Subscription
Ownerdoes not save you either: its built-in roledefinition has
actions: ['*']butdataActions: [].What changed
Skill (packaged, ships in v0.3.5). The
agentops-evalcoding-agentskill gains a new Step 0.5 - Ensure data-plane RBAC on the AI Services
account that runs before
agentops eval analyze. The skill agent now:AZURE_AI_FOUNDRY_PROJECT_ENDPOINTfrom.azure/<env>/.envor.agentops/.env.az cognitiveservices account list.az ad signed-in-user show.az role assignment createforCognitive Services OpenAI Userat the RG scope.Tutorials (docs, ship immediately on merge). All three quickstarts
gain the same self-contained 'Grant your identity data-plane access to
the AI Services account' section, with the exact error signature and
the one-liner so users running the tutorial manually have parity with
the skill flow:
docs/tutorial-prompt-agent-quickstart.mddocs/tutorial-hosted-agent-quickstart.mddocs/tutorial-end-to-end.mdPlugin sync.
plugins/agentops/skills/agentops-eval/SKILL.mdregenerated via
scripts/sync-skills.ps1so the VS Code extensioncopy stays identical.
CHANGELOG. Detailed entry added under
[Unreleased]; releasenotes promotion happens in the next
release/v0.3.5cut.Validation
python -m pytest tests/ -x -q-> 833 passed, 1 skipped.Cognitive Services OpenAI Userat the resource-group scope makesagentops eval runsucceed on the same workspace that previouslyfailed with FAILED_EXECUTION at the grader.
Follow-up
Doctor will eventually run this check pre-flight rather than relying on
skill / tutorial prose - tracked separately, not in this PR.