Release v0.3.6#227
Merged
Merged
Conversation
…d first eval run (#224) * docs(tutorials): document data-plane RBAC step missing from Foundry portal Creating a Foundry project through the portal only assigns the user 'Foundry User' at the project scope. That role does not cover OpenAI data-plane actions on the parent AI Services account, where chat completions actually live - so every AI-assisted evaluator and every cloud-eval grader fails with PermissionDenied the first time a fresh workspace tries to run eval. Subscription Owner is also insufficient because the built-in Owner role has actions: ['*'] but dataActions: []. All three tutorials (prompt-agent quickstart, hosted-agent quickstart, end-to-end) now document the one-time 'az role assignment create' that grants 'Cognitive Services OpenAI User' at the resource-group scope of the Foundry account, with the exact error signature so future readers can self-diagnose if they skipped it. A future AgentOps Doctor check will detect the missing assignment pre-run; until then, this step is a documented manual prerequisite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(skills): preflight data-plane RBAC in agentops-eval skill The agentops-eval coding-agent skill now resolves the Foundry project endpoint from .azure/<env>/.env or .agentops/.env, looks up the backing AI Services account + resource group with az cognitiveservices account list, fetches the signed-in object ID, and runs an idempotent az role assignment create for 'Cognitive Services OpenAI User' at the resource-group scope BEFORE 'agentops eval analyze' / 'agentops eval run'. This mirrors the new manual step added in the same PR to all three tutorials and keeps the skill experience aligned: users running the skill against a fresh Foundry project no longer hit the 401 PermissionDenied that the portal's default 'Foundry User'-at-project assignment leaves behind. CHANGELOG entry added under [Unreleased]. Plugin skills mirror under plugins/agentops/skills/ regenerated via scripts/sync-skills.ps1 to keep the VS Code extension copy identical. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts: # CHANGELOG.md
…226) When cloud/local evaluator workers error out on a subset of rows (most commonly data-plane RBAC that is still propagating), no dataset row has every grader return a score, so items_passed_all is 0 and �gentops eval run reports Threshold status: FAILED even though every computable threshold passed. This produced confusing phantom quality failures on the first run after granting the Cognitive Services OpenAI User role. - CLI now detects errored graders combined with all-thresholds-passed and prints a Warning clarifying this is an execution failure, names the RBAC-propagation cause, surfaces the first grader error, and advises waiting + re-running. Exit-code contract unchanged. - Added _grader_error_summary helper + focused unit tests. - Corrected RBAC propagation guidance (several minutes, intermittent FAILED-with-green-thresholds symptom) in the prompt-agent, hosted-agent and end-to-end tutorials and the agentops-eval skill; re-synced plugin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v0.3.6
Automated release branch created from
develop.What happened
release/v0.3.6created fromdevelopCHANGELOG.mdupdated: versioned section[0.3.6]added0.3.6(package.json, plugin.json, marketplace.json)Next steps
maingit tag v0.3.6 && git push origin v0.3.6git checkout develop && git merge main && git push origin developChecklist
v0.3.6pushed