diff --git a/AGENTS.md b/AGENTS.md index e6b8a87..8b4e212 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -221,9 +221,9 @@ Coverage highlights: docs/ ├── concepts.md # Core concepts and evaluation scenarios ├── how-it-works.md # Architecture and request flow -├── tutorial-prompt-agent-quickstart.md # Quickstart for Foundry Prompt Agents -├── tutorial-hosted-agent-quickstart.md # Quickstart for Foundry hosted / HTTP agents -├── tutorial-end-to-end.md # Full Foundry + AgentOps release-readiness workshop +├── tutorial-prompt-agent-quickstart.md # Tutorial for Foundry Prompt Agents +├── tutorial-hosted-agent-quickstart.md # Tutorial for Foundry hosted / HTTP agents +├── tutorial-end-to-end.md # End-to-end release-readiness tutorial (Foundry + AgentOps) ├── doctor-explained.md # Doctor checks, history, and operations model ├── ci-github-actions.md # CI/CD setup ├── release-process.md # Release and versioning diff --git a/CHANGELOG.md b/CHANGELOG.md index 9d4cff5..dae6c75 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,22 @@ This format follows [Keep a Changelog](https://keepachangelog.com/) and adheres ## [Unreleased] ### Changed +- **Tutorial wording: "quickstart" → "tutorial", "workshop" → "tutorial".** + The three documentation entries that were labeled "Prompt Agent quickstart", + "Hosted Agent quickstart", and "End-to-end workshop" now read as "Foundry + Prompt Agent tutorial", "Hosted or HTTP Agent tutorial", and "End-to-end + tutorial" across `README.md`, `plugins/agentops/README.md`, `AGENTS.md`, + `docs/concepts.md`, `docs/doctor-explained.md`, the `agentops-workflow` + skill (both synced copies), and the H1s + cross-references inside each + tutorial doc. The README description for the end-to-end tutorial now also + states explicitly that it **extends** either of the type-specific tutorials + (sandbox → dev → qa → prod plus Foundry red-team scans plus + trace-to-regression promotion) so the difference between the three is + obvious at a glance. The "quickstart" framing no longer fits doc bodies + that grew past 1000 lines covering multi-environment promotion, regression + injection, Doctor evidence, and Cockpit. The tutorial **filenames are + intentionally preserved** (`tutorial-*-quickstart.md`) to keep inbound + links and bookmarks stable. - **Skill + tutorial guidance now require `Cognitive Services OpenAI User` as a prerequisite RBAC role.** The `agentops-workflow` skill, `tutorial-prompt-agent-quickstart.md`, `tutorial-end-to-end.md`, and `docs/ci-github-actions.md` now instruct users diff --git a/README.md b/README.md index 775ce57..6ae8fb7 100644 --- a/README.md +++ b/README.md @@ -210,9 +210,9 @@ Cockpit sections, in display order: ## Documentation -- [Prompt Agent quickstart](docs/tutorial-prompt-agent-quickstart.md) - use this when the Foundry target is `agent: name:version`. -- [Hosted Agent quickstart](docs/tutorial-hosted-agent-quickstart.md) - use this when the target is a Foundry hosted or HTTP endpoint URL. -- [End-to-end workshop](docs/tutorial-end-to-end.md) - complete Foundry + AgentOps journey: create, debug, evaluate, release, observe, red-team follow-through, and trace regression. +- [Foundry Prompt Agent tutorial](docs/tutorial-prompt-agent-quickstart.md) - use this when the Foundry target is `agent: name:version`. Walks the sandbox → dev journey with a PR gate. +- [Hosted or HTTP Agent tutorial](docs/tutorial-hosted-agent-quickstart.md) - use this when the target is a Foundry hosted or HTTP endpoint URL. Same sandbox → dev journey for endpoint-based agents. +- [End-to-end tutorial](docs/tutorial-end-to-end.md) - extends either of the above with the full sandbox → dev → qa → prod promotion, Foundry red-team scans, and trace-to-regression promotion. - [Core concepts](docs/concepts.md) - [How it works](docs/how-it-works.md) - [Doctor explained](docs/doctor-explained.md) diff --git a/docs/concepts.md b/docs/concepts.md index a2a77c4..68fffce 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -180,11 +180,11 @@ AgentOps auto-selects common evaluation patterns from the dataset: Use one of the three hands-on tutorials for scenario coverage: -- [Prompt Agent quickstart](tutorial-prompt-agent-quickstart.md) for Foundry +- [Foundry Prompt Agent tutorial](tutorial-prompt-agent-quickstart.md) for Foundry prompt agents referenced as `name:version`. -- [Hosted Agent quickstart](tutorial-hosted-agent-quickstart.md) for Foundry +- [Hosted or HTTP Agent tutorial](tutorial-hosted-agent-quickstart.md) for Foundry hosted endpoints, generic HTTP agents, RAG services, and code-based workflows. -- [End-to-end workshop](tutorial-end-to-end.md) for the complete Foundry + +- [End-to-end tutorial](tutorial-end-to-end.md) for the complete Foundry + AgentOps loop, including CI/CD, observability, red-team follow-through, Doctor, release evidence, and trace regression. diff --git a/docs/doctor-explained.md b/docs/doctor-explained.md index b24a05d..d8d7ca6 100644 --- a/docs/doctor-explained.md +++ b/docs/doctor-explained.md @@ -3,7 +3,7 @@ A 10-minute read for a platform, observability, or AI engineer (and the engineering managers who own those teams) who runs `agentops doctor` for the first time. For step-by-step setup, see -the [end-to-end workshop](tutorial-end-to-end.md). +the [end-to-end tutorial](tutorial-end-to-end.md). ## 1. What the Doctor is - and isn't @@ -411,7 +411,7 @@ members and CI. ## 11. Next steps - Walk through a full setup with Azure resources: - [end-to-end workshop](tutorial-end-to-end.md). + [end-to-end tutorial](tutorial-end-to-end.md). - Open the workspace command center: `agentops cockpit` shows eval history, Doctor findings, CI/CD status, telemetry readiness, and Foundry/Azure navigation. diff --git a/docs/tutorial-end-to-end.md b/docs/tutorial-end-to-end.md index 64fc9e6..5a79331 100644 --- a/docs/tutorial-end-to-end.md +++ b/docs/tutorial-end-to-end.md @@ -1,9 +1,9 @@ # End-to-end tutorial: release readiness for Foundry agents -This tutorial is the full path. Use it after one of the quickstarts when you +This tutorial is the full path. Use it after one of the type-specific tutorials when you want to validate the complete develop -> evaluate -> release -> observe loop across **sandbox**, **dev**, **qa**, and **prod** environments. The two -quickstarts cover the same loop for a single agent type in a sandbox + dev +type-specific tutorials cover the same loop for a single agent type in a sandbox + dev arrangement; this tutorial expands the journey through every release stage. It is inspired by the Azure Samples repo @@ -409,7 +409,7 @@ agentops workflow generate ` > not yet contain the agent, it reads that block plus `prompt_file` and > creates the first version automatically. No per-environment manual > seeding. See the -> [prompt-agent quickstart](tutorial-prompt-agent-quickstart.md) for the +> [prompt-agent tutorial](tutorial-prompt-agent-quickstart.md) for the > full multi-environment journey. Before running that workflow, make the PR gate runnable in GitHub. Install the diff --git a/docs/tutorial-hosted-agent-quickstart.md b/docs/tutorial-hosted-agent-quickstart.md index af0bc5d..5af4d7d 100644 --- a/docs/tutorial-hosted-agent-quickstart.md +++ b/docs/tutorial-hosted-agent-quickstart.md @@ -1,6 +1,6 @@ -# Quickstart: Foundry Hosted Agent or HTTP Agent (sandbox → dev with PR gate) +# Tutorial: Foundry Hosted Agent or HTTP Agent (sandbox → dev with PR gate) -Use this quickstart when the agent is reachable as an endpoint URL. The +Use this tutorial when the agent is reachable as an endpoint URL. The example creates a small **Travel Agent** HTTP endpoint locally (your **sandbox**), then shows how to swap in a deployed Foundry Hosted Agent or cloud-hosted URL (your **dev** environment) for CI. @@ -14,7 +14,7 @@ arrangement: normalized `results.json`, runs Doctor with `--severity-fail critical` so regressions block the PR, and produces release evidence. -The toolkit benefit is the same as the prompt-agent quickstart, adapted +The toolkit benefit is the same as the prompt-agent tutorial, adapted for endpoint-based agents: you author and iterate against a local sandbox, then let CI verify the deployed dev environment is still healthy on every PR. Production-readiness gates (eval thresholds plus @@ -54,7 +54,7 @@ unexpected permission prompts. | You can create an Entra app registration with federated credentials, or an admin is ready to provide the client ID, tenant ID, and subscription ID. | The workflow skill can wire OIDC cleanly; without this, CI cannot authenticate to Azure. | | Copilot or your coding-agent CLI is signed in before you ask it to run AgentOps skills. | The skill handoff assumes an authenticated coding-agent session that can read the repo and propose GitHub/Azure setup steps. | -Unlike the Prompt Agent quickstart, this endpoint tutorial does not point the +Unlike the Prompt Agent tutorial, this endpoint tutorial does not point the generated PR workflow at `ai-agent-evals`. Hosted and HTTP agents are evaluated through the AgentOps local runner because CI must invoke your endpoint, extract the response, apply repo thresholds, and write the normalized `results.json`. @@ -132,7 +132,7 @@ green dev → ready for promotion to qa / prod > evaluate the PR's *unmerged* code unless your CI does a per-PR > ephemeral deploy. If you need PR-time regression catching for hosted > agents, the workflow skill can guide you through adding a per-PR -> ephemeral deploy step (out of scope for this quickstart). The +> ephemeral deploy step (out of scope for this tutorial). The > sandbox loop (local FastAPI + `agentops eval run`) is the > equivalent author-side gate. @@ -674,7 +674,7 @@ explicitly) so it uses dev. The Foundry project endpoint plus the agent URL environment) together let CI evaluate the deployed dev endpoint and land results in the dev observability target. -Use the same workflow-skill handoff pattern as the Prompt Agent quickstart, but +Use the same workflow-skill handoff pattern as the Prompt Agent tutorial, but keep the scope to the hosted endpoint: ```powershell @@ -699,7 +699,7 @@ permission. Open both Doctor outputs. The report explains the findings; the evidence pack summarizes what a reviewer needs to decide whether the endpoint is releasable. -In a fresh quickstart, warnings about production telemetry, CI history, or trace +In a fresh tutorial workspace, warnings about production telemetry, CI history, or trace regression history are expected and useful: they show what remains before this local endpoint becomes an operated service. @@ -761,7 +761,7 @@ You are done when: `.azure/qa/.env` and `.azure/prod/.env`, set GitHub Environments with the right `AGENTOPS_AGENT_ENDPOINT`, and use `agentops workflow generate --kinds qa,prod --force`. -- **Walk through the prompt-agent quickstart** at +- **Walk through the prompt-agent tutorial** at [tutorial-prompt-agent-quickstart.md](tutorial-prompt-agent-quickstart.md) to see the full prompt-as-code regression journey (stage-then-eval at PR time, no per-PR deploys required) and contrast the two diff --git a/docs/tutorial-prompt-agent-quickstart.md b/docs/tutorial-prompt-agent-quickstart.md index 6aea54a..b3180ea 100644 --- a/docs/tutorial-prompt-agent-quickstart.md +++ b/docs/tutorial-prompt-agent-quickstart.md @@ -1,6 +1,6 @@ -# Quickstart: Foundry Prompt Agent (sandbox → dev with PR gate) +# Tutorial: Foundry Prompt Agent (sandbox → dev with PR gate) -Use this quickstart when you want a Foundry-managed prompt agent referenced as +Use this tutorial when you want a Foundry-managed prompt agent referenced as `name:version`. The example creates a small **Travel Agent** in Foundry and then uses AgentOps to add repo-side readiness, a PR gate that catches regressions before merge, a `dev` deploy workflow, Doctor evidence, and @@ -1077,7 +1077,7 @@ evidence. Read the output in this order: | `Findings: N (M critical ...)` | The severity rollup; critical items are what you discuss first. | | `Finding summary` | The terminal triage list. | -In a fresh quickstart it is normal to see warnings for scheduled CI +In a fresh tutorial workspace it is normal to see warnings for scheduled CI (you only generated `pr` and `dev`), continuous evaluation, qa/prod deploys, explicit thresholds, or red-team scans. Treat those as the hardening backlog. The eval gates and the dev deploy loop are diff --git a/plugins/agentops/README.md b/plugins/agentops/README.md index be2de80..ea73735 100644 --- a/plugins/agentops/README.md +++ b/plugins/agentops/README.md @@ -81,7 +81,7 @@ Run the AgentOps watchdog and summarize production latency findings. ## Links - [AgentOps Accelerator](https://github.com/Azure/agentops) -- [Prompt Agent quickstart](https://github.com/Azure/agentops/blob/main/docs/tutorial-prompt-agent-quickstart.md) -- [Hosted Agent quickstart](https://github.com/Azure/agentops/blob/main/docs/tutorial-hosted-agent-quickstart.md) -- [End-to-end workshop](https://github.com/Azure/agentops/blob/main/docs/tutorial-end-to-end.md) +- [Foundry Prompt Agent tutorial](https://github.com/Azure/agentops/blob/main/docs/tutorial-prompt-agent-quickstart.md) +- [Hosted or HTTP Agent tutorial](https://github.com/Azure/agentops/blob/main/docs/tutorial-hosted-agent-quickstart.md) +- [End-to-end tutorial](https://github.com/Azure/agentops/blob/main/docs/tutorial-end-to-end.md) - [How it works](https://github.com/Azure/agentops/blob/main/docs/how-it-works.md) diff --git a/plugins/agentops/skills/agentops-workflow/SKILL.md b/plugins/agentops/skills/agentops-workflow/SKILL.md index e53fe76..db15e44 100644 --- a/plugins/agentops/skills/agentops-workflow/SKILL.md +++ b/plugins/agentops/skills/agentops-workflow/SKILL.md @@ -62,7 +62,7 @@ by discovering the whole Azure subscription. - `azd env get-values` when `azure.yaml` exists and azd is available. - `.github/workflows/agentops-*.yml`. 2. Read the generated workflows to determine exactly which GitHub environments - and variables are needed. For the prompt-agent quickstart, `pr` normally + and variables are needed. For the prompt-agent tutorial, `pr` normally means only `environment: dev`. 3. Treat `dev` here as a GitHub Actions environment for OIDC and variables. It normally points at the Foundry project already configured by `agentops init`; @@ -302,7 +302,7 @@ so the PR-comment step can post. ### GitHub Actions (OIDC) At the GitHub Environment level when the workflow declares an environment -(preferred for the quickstart), or at repository level when intentionally shared +(preferred for the tutorials), or at repository level when intentionally shared across environments, set: - `AZURE_CLIENT_ID` - App registration / managed identity used for OIDC. diff --git a/src/agentops/templates/skills/agentops-workflow/SKILL.md b/src/agentops/templates/skills/agentops-workflow/SKILL.md index e53fe76..db15e44 100644 --- a/src/agentops/templates/skills/agentops-workflow/SKILL.md +++ b/src/agentops/templates/skills/agentops-workflow/SKILL.md @@ -62,7 +62,7 @@ by discovering the whole Azure subscription. - `azd env get-values` when `azure.yaml` exists and azd is available. - `.github/workflows/agentops-*.yml`. 2. Read the generated workflows to determine exactly which GitHub environments - and variables are needed. For the prompt-agent quickstart, `pr` normally + and variables are needed. For the prompt-agent tutorial, `pr` normally means only `environment: dev`. 3. Treat `dev` here as a GitHub Actions environment for OIDC and variables. It normally points at the Foundry project already configured by `agentops init`; @@ -302,7 +302,7 @@ so the PR-comment step can post. ### GitHub Actions (OIDC) At the GitHub Environment level when the workflow declares an environment -(preferred for the quickstart), or at repository level when intentionally shared +(preferred for the tutorials), or at repository level when intentionally shared across environments, set: - `AZURE_CLIENT_ID` - App registration / managed identity used for OIDC.