docs(skill): require Cognitive Services OpenAI User as prereq RBAC role#203
Merged
Conversation
Foundry `azure_ai_evaluator` graders impersonate the OIDC principal to call OpenAI; without `Cognitive Services OpenAI User` on the underlying AI Services account the graders fail with a 401 PermissionDenied and every cloud eval metric returns null. Verified end-to-end on placerda/agentops-prompt-quickstart: after granting the role, the first PR run goes green from scratch. - agentops-workflow SKILL.md: pre-dispatch checks now list both Foundry User (Foundry project) AND Cognitive Services OpenAI User (AI Services account), with role ids and az role assignment create commands for each. - tutorial-prompt-agent-quickstart.md: step 12's Copilot prompt and the workflow-skill walkthrough list both roles. - tutorial-end-to-end.md: both workflow-skill prompts list both roles. - docs/ci-github-actions.md: prerequisite section lists both roles with the OpenAI graders' failure mode spelled out. - plugins/agentops/skills/agentops-workflow/SKILL.md: synced from src/. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
placerda
added a commit
that referenced
this pull request
May 29, 2026
…le (#203) (#204) Foundry `azure_ai_evaluator` graders impersonate the OIDC principal to call OpenAI; without `Cognitive Services OpenAI User` on the underlying AI Services account the graders fail with a 401 PermissionDenied and every cloud eval metric returns null. Verified end-to-end on placerda/agentops-prompt-quickstart: after granting the role, the first PR run goes green from scratch. - agentops-workflow SKILL.md: pre-dispatch checks now list both Foundry User (Foundry project) AND Cognitive Services OpenAI User (AI Services account), with role ids and az role assignment create commands for each. - tutorial-prompt-agent-quickstart.md: step 12's Copilot prompt and the workflow-skill walkthrough list both roles. - tutorial-end-to-end.md: both workflow-skill prompts list both roles. - docs/ci-github-actions.md: prerequisite section lists both roles with the OpenAI graders' failure mode spelled out. - plugins/agentops/skills/agentops-workflow/SKILL.md: synced from src/. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Foundry
azure_ai_evaluatorgraders impersonate the OIDC principal to call OpenAI; withoutCognitive Services OpenAI Useron the underlying AI Services account the graders fail with a 401PermissionDeniedonMicrosoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/actionand every cloud eval metric returnsnull. The first PR run in the tutorial used to fail every time — even on a happy-path setup — because the tutorial only asked users to grant Foundry User. Pairs with #201 (which surfaces the buried error message into the report).Verified end-to-end on
placerda/agentops-prompt-quickstart: after granting the role to the GHA OIDC SP, the first PR run goes green from scratch (run26625808174).Files
src/agentops/templates/skills/agentops-workflow/SKILL.md— pre-dispatch checks now list both Foundry User (Foundry project scope) and Cognitive Services OpenAI User (AI Services account scope), with role ids53ca6127-db72-4b80-b1b0-d745d6d5456dand5e0bd9bd-7b93-4f28-af87-19fc36ad61bdplus the matchingaz role assignment createcommands.plugins/agentops/skills/agentops-workflow/SKILL.md— synced fromsrc/viascripts/sync-skills.ps1.docs/tutorial-prompt-agent-quickstart.md— step 12's Copilot prompt and the workflow-skill walkthrough list both roles.docs/tutorial-end-to-end.md— both workflow-skill prompts list both roles.docs/ci-github-actions.md— prerequisite section spells out the OpenAI graders' failure mode and lists both roles.CHANGELOG.md— newChangedentry under[Unreleased].Verification
pytest tests/→ 792 passed, 3 skipped (46.58s).gh workflow run agentops-pr.yml→ success, both jobs green.Notes
This is documentation-only — no behavior change in the toolkit itself. The toolkit-side fix that surfaces the error message when this role IS missing already shipped in #201/#202. Together: skill now prevents the failure mode, toolkit now diagnoses it clearly if it happens.