fix: pin azd ai agents extension to an existing version - #373
Merged
Conversation
The generated CI pinned `azure.ai.agents` to `1.0.0`, which has never been published to the azd extension registry. Every generated pipeline targeting a Foundry hosted agent failed at `Install pinned azd AI agents extension` with `ERROR: no extensions found`, on both GitHub Actions and Azure DevOps. The highest published version is `1.0.0-beta.9`, confirmed against https://aka.ms/azd/extensions/registry and `azd extension list`. Verified end to end: with this version the PR gate installs the extension, `agentops eval run` completes, results publish to New Foundry Evaluations, and the gate reports `Threshold status: PASSED`. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
Paulo Lacerda (placerda)
added a commit
that referenced
this pull request
Aug 7, 2026
…pair ADO PR pipeline YAML (#391) * fix(cicd): give azd credentials in the ADO eval stage and repair PR pipeline YAML The 0.8.5 fixes for issue #379 were verified live on GitHub Actions only. This audit generated the Azure DevOps pipelines for every kind (pr, dev, qa, prod) and every eval-runner variant (azd, cloud, local), parsed the output, and compared it against the GitHub Actions equivalents. Two of the twelve generated files were broken. The Azure DevOps eval stage installed `azd` and the pinned `azure.ai.agents` extension but never ran `azd config set auth.useAzCliAuth "true"`. When a project uses the azd evaluation backend, `agentops eval run` shells out to `azd ai agent eval run`, so that stage needs azd credentials just as much as provision and deploy do. `AzureCLI@2` completes the service-connection handshake and leaves the Azure CLI authenticated, but `azd` keeps a separate credential store and ignores that session unless `auth.useAzCliAuth` is set. This is exactly the trap issue #379 fixed, surviving in a stage nobody had run. The provision and deploy stages already set the flag, so the eval stage was inconsistent with the rest of the same pipeline. The generated Azure DevOps PR pipeline was not valid YAML at all. The report-comment step embeds a multi-line `python -c` snippet whose continuation lines sat at column 0. A column-0 line terminates the enclosing `bash: |` block scalar, so Azure DevOps rejected the file with a parse error before running a single step. The lines are now indented into the block. The trailing backslashes still join them into one logical line, so the Python is byte-for- byte equivalent once bash strips the continuations; this was confirmed by running the joined command against sample thread JSON. Everything else checked out. `azd` is installed explicitly via `aka.ms/install-azd.sh` rather than assumed present, which is correct because Microsoft-hosted agents do not ship it. The extension pin from issue #373 is applied everywhere. Variable references use ADO `$(VAR)` macro syntax with a variable group, and no `${{ }}` expression syntax leaks in. `useAzCliAuth` is the right mechanism for ADO rather than the GitHub Actions `azd auth login --federated-credential-provider github`, since every azd call runs inside `AzureCLI@2` and the delegated `AzureCliCredential` works for both client-secret and workload-identity-federation service connections. Two regression tests cover the gaps. One asserts the install, pin, and auth triple in the eval stage of all four pipelines. The other parses every generated ADO pipeline across all eval-runner variants and rejects any stray column-0 line, which is the failure mode that made the PR pipeline unusable. Both were confirmed to fail with the respective fix reverted. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee * fix(cicd): authenticate azd in the GitHub Actions eval job The Azure DevOps audit that landed in the previous commit surfaced the same defect on the GitHub Actions side. When a project uses the azd evaluation backend, `_github_eval_substitutions` emitted `Azure/setup-azd@v2` and the pinned `azure.ai.agents` extension install, then went straight to running `agentops eval run`. That command shells out to `azd ai agent eval run`, and the azd binary had no credentials at that point. `azure/login@v3` authenticates the Azure CLI, not azd. azd keeps its own credential store, so `az` calls succeed in the job while every azd call fails. This is exactly the issue #379 trap that PR #386 fixed for the provision and deploy jobs, reproduced in the eval job because that job builds its steps in a different function. The fix mirrors the working provision and deploy jobs rather than copying the Azure DevOps approach, because the mechanisms are not interchangeable. Azure DevOps needs `azd config set auth.useAzCliAuth "true"` since every azd call runs inside `AzureCLI@2`. GitHub Actions needs an explicit `azd auth login --federated-credential-provider github`. The new step reads `vars.AZURE_CLIENT_ID` and `vars.AZURE_TENANT_ID`, which every eval job already passes to its `azure/login@v3` step, and every workflow template already declares `permissions: id-token: write`, so no new repository wiring is required. Two regression tests cover the GitHub Actions side. The first walks the eval job in the pr, dev, qa, and prod workflows and asserts the setup, pinned install, and federated login all appear. The second is a structural invariant across every generated workflow: the count of azd extension installs must equal the count of federated logins and must be greater than zero, so a future azd call site that copies the install step but forgets the credentials fails the suite. Both tests were verified to fail against the unfixed generator. Regenerating the full matrix (azd, cloud, and local eval runners) produces 15 workflow files that all parse, and no job that invokes azd lacks an auth step. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every workflow generated by
agentops workflow generatefor a Foundry hosted agent target fails in CI at theInstall pinned azd AI agents extensionstep:azure.ai.agents1.0.0 has never been published. The highest version in the official registry is1.0.0-beta.9:This blocks 100% of generated pipelines targeting a hosted agent, on GitHub Actions and Azure DevOps alike (
services/cicd.py:440and:639share the same constant). The only workaround today is hand-editing the generated workflow, which defeats the purpose ofworkflow generate.Change
One-line pin correction in
src/agentops/pipeline/official_eval.py, plus the matching assertion intests/unit/test_cicd.py.The env-var override (
AGENTOPS_AZD_AI_AGENTS_EXTENSION_VERSION) is untouched, so anyone pinning a different version keeps working.Verification
Reproduced the failure and confirmed the fix on a real repo generated from the hosted-agent template, against
agentops-accelerator==0.8.3andazd 1.29.0.After the fix the PR gate goes green end to end:
Job conclusion:
success.Unit tests:
Follow-up (not in this PR)
When
agentops.yamlsetsexecution: cloud, the generated workflow still emitsAzure/setup-azd@v2+azd extension install, butagentops eval runnever shells out to azd — Foundry runs the agent and evaluators server-side. Forexecution: cloudboth steps are dead weight and an avoidable failure point. Filed separately.