docs: correct published guidance that blocks readers mid-tutorial - #394
Merged
Conversation
The published site builds from this orphan `docs` branch, so content that drifted from `develop` has been shipping uncorrected. Several pages told readers to run commands that do not exist or that fail on the documented install, and one section omitted the OIDC trap that breaks federated login outright. OIDC federated credentials (ci-github-actions.md) Symptom: `azure/login` fails with AADSTS700213 "No matching federated identity record found", even though the credential looks correct. Cause: accounts with a customized subject claim prefix send numeric account and repository IDs in the OIDC subject (`repo:<owner>@<id>/<repo>@<id>:environment:<env>`). Entra compares the subject literally, so a credential built from the plain-name form never matches. The page had no `sub_claim_prefix` guidance at all. Fix: port the verified section from develop. Covers reading the prefix with `gh api repos/<owner>/<repo>/actions/oidc/customization/sub`, the two-credential-per-environment pattern, Bash and PowerShell helpers (including the `$repo:` scoped-variable trap), credential readback, and a troubleshooting table for AADSTS700213, AADSTS53003 and the `AuthorizationFailed` subscription-scope case on `azd provision`. `execution: cloud` with hosted agents (concepts, evaluation, how-it-works, tutorial-http-agent, tutorial-hosted-agent) Symptom: four pages stated cloud eval supports Foundry prompt agents only, contradicting the product's headline scenario. Cause: stale text. `agentops_config._parse_hosted_agent_reference()` exists specifically to resolve `/agents/<name>/versions/<version>` out of a hosted URL, and `cloud_runner` accepts both `foundry_prompt` and `foundry_hosted` targets. Fix: state that cloud supports hosted agent URLs carrying the versioned path, and add a worked `execution: cloud` example to the hosted-agent tutorial. Install commands omitted the `[agent]` extra (index, foundry-ops-workbook, all three tutorials) Symptom: two of three tutorials failed at their final step. `agentops cockpit` raises ImportError when the extra is absent. Cause: every documented install line was plain `pip install agentops-accelerator`. Fix: use `pip install "agentops-accelerator[agent]"` throughout, with a short note on what the extra provides. Also correct `pip install -e ".[dev]"` in how-it-works.md; `dev` is a PEP 735 dependency group, not an extra, so that command fails. Use `uv sync --group dev`. Non-existent `agentops eval compare` (tutorial-hosted-agent.md) Symptom: the regression-detection step could not be completed. Cause: no such command. The real mechanism is `agentops eval run --baseline <path/to/results.json>`. Fix: rewrite the step around the real flag. Link to a private repository (tutorial-http-agent.md) Symptom: anonymous visitors got a 404 on the reference-workflows link. Cause: `placerda/gpt-rag-orchestrator-agentops` is private. Fix: point readers at `agentops workflow generate` instead. Other mentions of that name are the reader naming their own repo and are left as-is. Empty sitemap (mkdocs.yml) Symptom: `sitemap.xml` shipped with zero URLs, hurting indexing. Cause: `site_url` was unset. Fix: set `site_url: https://azure.github.io/agentops/`. Local build now emits 21 URLs. azd in CI (ci-github-actions.md) Adds a short section covering three traps the site did not document: azd refuses to auto-install extensions on CI runners and `azure.ai.agents` is pinned to 1.0.0-beta.9; azd keeps a separate credential store and does not inherit the `azure/login` session; Azure DevOps reuses the CLI session via `azd config set auth.useAzCliAuth "true"` while GitHub Actions needs `azd auth login --federated-credential-provider github`. Every claim was verified against origin/develop source before writing. `mkdocs build` succeeds; the 9 strict-mode warnings are pre-existing on the base commit and unchanged by this PR. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
5 tasks
1. Baseline comparison does not gate CI. `exit_code_from` in `pipeline/orchestrator.py` returns 0/2 from `summary.overall_passed` alone, which is the configured thresholds. `pipeline/comparison.py` only computes deltas. A run can regress against a baseline and still exit 0. The hosted-agent tutorial claimed the opposite. It now says `--baseline` is a review signal, thresholds enforce, and explains that the generated workflows read a baseline only when `.agentops/baseline/results.json` is committed. 2. AADSTS53003 was misdiagnosed as a wrong-tenant problem. It is BlockedByConditionalAccess. Workload identities are in scope of Conditional Access, so a policy requiring MFA, a compliant device, or a named location will block a GitHub-hosted runner. Wrong tenant or wrong client id surfaces as AADSTS700016, which now has its own row. 3. A versioned hosted agent URL is necessary but not sufficient for `execution: cloud`. `core/agentops_config.py` extracts only name and version; `pipeline/cloud_runner.py` builds the client from the separately configured project endpoint and submits name plus version. The URL host and project segment are discarded. Both evaluation.md and the hosted-agent tutorial now state that AZURE_AI_FOUNDRY_PROJECT_ENDPOINT must point at the project holding that agent version. 4. Contributor commands used the wrong interpreter. `uv sync` populates `.venv` but does not activate it, so a bare `python` can be the system interpreter without the dev dependencies. CI uses `uv run pytest`. The commands now use `uv run`. Also corrected the claim that `pip install -e ".[dev]"` fails: `dev` is absent from optional-dependencies, so pip warns about an unknown extra and installs the project without the group. Added a link to the uv install guide. Strict build produces the same nine pre-existing warnings as origin/docs. No new warnings introduced. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
The published site had nine broken links that mkdocs --strict flags and that render as dead links for readers. All nine pointed at repository files with relative paths like `../src/agentops/...`, which resolve inside the docs tree, not the repo. Those files are not part of the MkDocs site, so the links were always broken in the published output. Rewrote them as absolute GitHub URLs against main: docs/how-it-works.md 6 links to src/agentops modules docs/doctor-explained.md 1 link to the WAF checklist CSV docs/e2e-live-setup.md 1 link to .github/workflows/e2e.yml docs/release-process.md 1 link to CONTRIBUTING.md Each target was verified to exist on origin/main before rewriting. Also fixed a stale in-page anchor in release-process.md. The link said `#8-production-release-pipeline-pypi` but the section had been renumbered to 9. Five pages were built into the site but reachable only by direct URL, because nothing in the nav pointed at them: concepts.md, how-it-works.md, release-process.md, e2e-live-setup.md, and e2e-live-architecture.md. Concepts is reader-facing and now sits under Overview. The other four are contributor-facing and now live under a new Contributing section. Checked release-process.md and the e2e pages for sensitive content first: they name GitHub secrets such as PYPI_TOKEN and explain how to mint them, but contain no secret values. mkdocs build --strict now completes with zero warnings. It previously aborted with nine. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this targets
docsThe published site (https://aka.ms/agentops-accelerator → https://azure.github.io/agentops/) is built from the orphan
docsbranch..github/workflows/deploy-docs.ymlchecks outref: docsand runsmkdocs gh-deploy --remote-branch gh-pages.docshas no merge base withdevelop, so nothing landed ondevelopcan change the live site. These are surgical content corrections on the branch that actually publishes.This is not the structural consolidation of
docsintodevelop. That remains tracked separately.What was broken, and what this fixes
1. OIDC federated credentials had no
sub_claim_prefixguidancedevelop:docs/ci-github-actions.mdcarries a ~210-line OIDC section. The published version was ~52 lines and omittedsub_claim_prefixentirely: the term appears 6 times ondevelopand 0 times on the live site.This is the trap that breaks federated login. When an account carries a customized subject claim prefix, GitHub sends numeric account and repository IDs in the OIDC subject:
Entra compares
subjectliterally. There are no wildcards and no normalization, so a credential built fromrepo:<owner>/<repo>:environment:<env>never matches and login fails withAADSTS700213: No matching federated identity record found.Ported from
develop, merged by hand so thedocsbranch keeps its own structure and the four stepsdeveloplacks:gh api repos/<owner>/<repo>/actions/oidc/customization/sub, and why you readsub_claim_prefixitself rather than theuse_default/use_immutable_subjectbooleans next to it."repo:$repo:environment:$env"silently breaks because$repo:parses as a scoped variable.az ad app federated-credential list.AADSTS700213,AADSTS53003(usually a wrongAZURE_TENANT_ID, not a real Conditional Access policy) andAuthorizationFailedonazd provisionwheninfra/main.bicepdeclarestargetScope = 'subscription'.2.
execution: cloudwas documented as prompt-agents-onlyFour published passages said cloud eval does not support hosted agents. The code says otherwise:
core/agentops_config.py_parse_hosted_agent_reference()exists specifically to pull/agents/<name>/versions/<version>out of a hosted URL. Its docstring: "Server-side evaluation (execution: cloud) needs thatname/versionpair to build theazure_ai_agenttarget."pipeline/cloud_runner.pyaccepts{"foundry_prompt", "foundry_hosted"}and rejects everything else.Corrected in
concepts.md,evaluation.md,how-it-works.mdandtutorial-http-agent.md. Added a workedexecution: cloudexample totutorial-hosted-agent.md.HTTP endpoints and raw model deployments genuinely are not supported by cloud; those rows are unchanged.
3. Every install command omitted the
[agent]extraagentops cockpitraisesImportErrorwhen[agent]is absent (agent/cockpit.py), and so doesagentops agent serve. Cockpit is instructed in 9 places across the site, including the final step of two of the three tutorials. A reader following the documented install could not finish them.Confirmed against
develop:pyproject.tomlthat the real extras are exactlymcpandagent. All install lines now usepip install "agentops-accelerator[agent]", with a one-line note on what the extra buys.Also corrected
how-it-works.md, which told contributors to runpip install -e ".[dev]".devis a PEP 735 dependency group, not an extra, so that command fails. Replaced withuv sync --group dev, which is what CI runs.4.
agentops eval comparedoes not existtutorial-hosted-agent.mdbuilt its regression-detection step around it. There is nocomparesubcommand incli/app.py. The real mechanism is the--baselineoption oneval run, help text "Path to a previous results.json to compare this run against." The step is rewritten around it.5. Link to a private repository
tutorial-http-agent.mdlinked toplacerda/gpt-rag-orchestrator-agentops/tree/develop/.github/workflows.gh repo view --json visibilityconfirms it is private, so anonymous visitors to the public Azure site get a 404. Replaced with a pointer toagentops workflow generate, which produces the same files in the reader's own repo. Other mentions of that name are the reader naming their own repo and are left alone.6. Empty
sitemap.xmlmkdocs.ymlhad nosite_url, so the generated sitemap contained zero URLs. Set tohttps://azure.github.io/agentops/. Local build now emits 21 URLs.7. azd in CI
Nothing on the site covered three real traps. Added a concise section:
azure.ai.agentsis pinned to1.0.0-beta.9(pipeline/official_eval.py), overridable viaAGENTOPS_AZD_AI_AGENTS_EXTENSION_VERSION.azure/loginCLI session.azd config set auth.useAzCliAuth "true"; GitHub Actions needsazd auth login --client-id ... --federated-credential-provider github.All three are verbatim behaviours of
services/cicd.py:_azd_cli_setup_steps().Verification
Every claim was checked against
origin/developsource before writing, viagit show origin/develop:<path>. No command, flag or version in this PR is unverified.mkdocs buildsucceeds.mkdocs build --strictaborts with 9 warnings, but the identical 9 warnings are present on the base commit (verified by stashing and rebuilding). They are pre-existing links fromdocs/*.mdintosrc/and.github/, which are absent from this orphan branch. This PR neither adds nor fixes them.Deliberately out of scope
The
tutorial-prompt-agent.mdre-port, the fullevaluation.mdreplacement, generated doctor-checks content, nav changes, and a dedicated troubleshooting page. All are tracked with the structural consolidation work.Correction to the prior audit
The audit report concluded
placerda/gpt-rag-orchestrator-agentopshad been deleted. It has not; it is private. The user-facing outcome is the same 404, but the cause matters if anyone tries to restore the link.