diff --git a/CHANGELOG.md b/CHANGELOG.md index 8da3248..5952281 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,20 @@ This format follows [Keep a Changelog](https://keepachangelog.com/) and adheres ## [Unreleased] ### Added +- **Auto-bootstrap empty Foundry projects on first deploy.** New optional + `prompt_agent_bootstrap` block in `agentops.yaml` lets the prompt-agent + deploy workflow create the first version of an agent in a dev / qa / prod + Foundry project that does not yet have one. When the stage step looks up + the seed agent and gets a 404, it reads the model deployment (required) + plus optional `description`, `model_parameters`, and `tools` from + `prompt_agent_bootstrap`, combines them with `prompt_file`, and creates + the first version automatically. The deployment artifact records the new + `action: "bootstrapped"` for that first run; subsequent deploys follow + the normal reuse / next-version flow. Eliminates the previous + per-environment manual seeding step. `agentops workflow analyze` now + warns when a prompt-agent workspace is missing this block. Authentication + (401 / 403) and other non-404 errors continue to propagate — the + bootstrap path only triggers on a genuine "agent does not exist" 404. - **`--doctor-gate` flag on `agentops workflow generate`.** New option `--doctor-gate critical|warning|none` controls the Doctor severity floor in the PR workflow template. Default is `critical`, which makes the PR diff --git a/docs/tutorial-end-to-end.md b/docs/tutorial-end-to-end.md index dc0a242..f9166cb 100644 --- a/docs/tutorial-end-to-end.md +++ b/docs/tutorial-end-to-end.md @@ -402,6 +402,16 @@ agentops workflow generate ` > warnings during hardening sprints. Use `--doctor-gate none` to make Doctor > advisory-only (the pre-`--doctor-gate` behavior). +> **Promoting prompt agents across multiple Foundry projects?** Add a +> `prompt_agent_bootstrap` block (model deployment plus optional +> description, model_parameters, and tools) to `agentops.yaml`. When the +> deploy workflow runs against a dev / qa / prod Foundry project that does +> not yet contain the agent, it reads that block plus `prompt_file` and +> creates the first version automatically. No per-environment manual +> seeding. See the +> [prompt-agent quickstart](tutorial-prompt-agent-quickstart.md) for the +> full multi-environment journey. + Before running that workflow, make the PR gate runnable in GitHub. Install the AgentOps workflow skill if needed: diff --git a/docs/tutorial-prompt-agent-quickstart.md b/docs/tutorial-prompt-agent-quickstart.md index 4971b4f..8923022 100644 --- a/docs/tutorial-prompt-agent-quickstart.md +++ b/docs/tutorial-prompt-agent-quickstart.md @@ -52,8 +52,9 @@ permission prompts. | Check | Why it matters | |---|---| | Azure CLI is installed and `az login` succeeds with the tenant that owns the Foundry projects. | AgentOps, Foundry SDK calls, and CI setup all need the same Azure identity context. | -| You can create **two** Foundry projects in the same Azure subscription (or have two existing projects you can use). | The tutorial uses a sandbox project for authoring and experimentation plus a shared dev project for the PR gate; the PR workflow stages candidates in dev. | -| You can publish a prompt agent in each Foundry project. | The tutorial seeds the same `travel-agent:1` baseline in both projects so the deploy workflow has a known template to look up. | +| You can create **two** Foundry projects in the same Azure subscription (or have two existing projects you can use). | The tutorial uses a sandbox project for authoring and experimentation plus a shared dev project for the PR gate. You only need to publish the agent in sandbox — CI auto-bootstraps it in dev (and later qa / prod). | +| You can publish a prompt agent in the **sandbox** Foundry project. | The tutorial seeds `travel-agent:1` only in sandbox. Dev / qa / prod start empty; the prompt-agent deploy workflow creates the first version in those projects automatically using `prompt_agent_bootstrap` defaults plus `prompt_file`. | +| The **same model deployment name** (for example `gpt-4o-mini`) exists in every Foundry project you plan to deploy to. | `prompt_agent_bootstrap.model` is a single value reused for every environment. If dev does not have that deployment, the first auto-bootstrap fails. | | You can create or attach Application Insights for at least the dev Foundry project. | Foundry Traces, the Operate dashboard, Doctor, and Cockpit need telemetry to tell the observability story. Sandbox observability is optional. | | You can push to the tutorial GitHub repository and run GitHub Actions. | The PR gate only runs after the repo is pushed. | | GitHub CLI is authenticated with `gh auth login` if you use the PR commands in this tutorial. | The regression step opens PRs and sends the reader directly to the workflow run. | @@ -70,8 +71,10 @@ sandbox Foundry project dev Foundry project (authoring + experimentation; (shared environment, PR gate target, used by you or the team) where merge deploys land) │ │ - │ travel-agent:1 (seed) │ travel-agent:1 (seed, same instructions) - │ travel-agent:2,3,4,... (free saves) │ travel-agent:2,3,... (created by CI per PR / deploy) + │ travel-agent:1 (you create this seed) │ (empty — no agent here yet; + │ travel-agent:2,3,4,... (free saves) │ CI auto-creates travel-agent:1 + │ │ on the first deploy via + │ │ prompt_agent_bootstrap) │ │ └──── git is the source of truth ─────────►│ .agentops/prompts/travel-agent.md @@ -84,7 +87,12 @@ Two ideas to internalize: `.agentops/prompts/travel-agent.md` is what CI reads and what reviewers diff. Each Foundry project's version numbers count its own saves and are environment-local. -2. **Cross-environment identity is the SHA, not the number.** AgentOps +2. **You only author the agent in sandbox.** Dev, qa, and prod start + empty. When the prompt-agent deploy workflow runs against an empty + environment, it reads `prompt_agent_bootstrap` from `agentops.yaml` + plus `prompt_file`, then creates `travel-agent:1` automatically in + that environment. You never seed dev / qa / prod by hand. +3. **Cross-environment identity is the SHA, not the number.** AgentOps embeds `agentops.prompt_sha256` and `agentops.git_sha` into every Foundry version it creates, and writes the same identifiers into the per-environment deploy artifact `foundry-agent.json`. When you ask @@ -98,10 +106,10 @@ have a real `foundry-agent.json` artifact to open. | Step | Main tool | What you do | AgentOps role | |---|---|---|---| -| Create two Foundry projects | Foundry portal (or `microsoft-foundry` skill) | Create `travel-agent-sandbox` and `travel-agent-dev`; seed `travel-agent:1` in both. | No ownership; AgentOps consumes the published baselines. | +| Create two Foundry projects | Foundry portal (or `microsoft-foundry` skill) | Create `travel-agent-sandbox` (where you author) and `travel-agent-dev` (left empty — CI seeds it). | No ownership; AgentOps consumes the published baseline from sandbox and bootstraps dev. | | Author in sandbox | Foundry playground | Iterate on the prompt safely in sandbox Foundry. | Optional spot-check via local `agentops eval run`. | | Promote the prompt to git | Editor | Copy validated instructions into `.agentops/prompts/travel-agent.md`. | The CI gate reads this file. | -| First green PR + dev deploy | GitHub Actions + Foundry dev project | Push prompt, open PR, watch CI stage a candidate in dev, evaluate it, run Doctor; merge; deploy lands in dev. | Owns the gate, the threshold decision, the Doctor blocking step, the deploy artifact, and the release evidence. | +| First green PR + dev deploy | GitHub Actions + Foundry dev project | Push prompt, open PR, watch CI auto-bootstrap `travel-agent:1` in dev from `prompt_agent_bootstrap` (the dev project is still empty at this point), evaluate it, run Doctor; merge; deploy lands in dev. | Owns the gate, the bootstrap-on-first-deploy, the threshold decision, the Doctor blocking step, the deploy artifact, and the release evidence. | | Force a regression | Editor + GitHub Actions | Edit the prompt to a worse version, push, observe BOTH eval threshold failure AND Doctor regression CRITICAL. | Catches the regression at PR time, not after merge. | | Fix and redeploy | Editor + GitHub Actions | Restore prompt, push, PR green, merge, deploy. | Records the recovery. | | Review readiness | AgentOps Doctor + Cockpit | Check CI, eval, telemetry, evidence, and links. | Turns scattered signals into release blockers, warnings, evidence files, and next actions. | @@ -208,14 +216,19 @@ I want to set up two Azure AI Foundry projects in the same subscription for an AgentOps tutorial: 1. travel-agent-sandbox - the authoring and experimentation space - (used by me, or shared with my team for iteration). + (used by me, or shared with my team for iteration). I will publish + the seed prompt agent (travel-agent:1) here manually in the next + step. 2. travel-agent-dev - shared dev environment used by CI as the PR gate - target and the dev deploy target. + target and the dev deploy target. Leave this project EMPTY. CI will + auto-create the first agent version here on the first deploy using + AgentOps' prompt_agent_bootstrap defaults. For each project, please: - Create the project (any region with a chat-capable deployment is fine). -- Make sure a chat-capable model deployment (gpt-4o-mini works) is - available in the project. +- Make sure the SAME chat-capable model deployment name is available in + both projects (gpt-4o-mini works). Same name is important: AgentOps + uses a single bootstrap model value for every environment. - Attach or create an Application Insights resource for telemetry, starting with the dev project. @@ -224,19 +237,19 @@ Show me the planned changes and the resulting endpoints before applying. If the skill is not available, use Path A. -## 4. Seed `travel-agent:1` in both Foundry projects +## 4. Seed `travel-agent:1` in the sandbox project -The deploy workflow looks up the agent reference from `agentops.yaml` -inside each environment's Foundry project as a **template**. It copies -that template's model deployment, kind, name, and other settings, then -replaces the instructions with whatever is in `prompt_file`. That means -`travel-agent:1` must already exist in **both** projects when CI runs, -with identical settings. +You only author the agent in **one place**: your sandbox Foundry +project. Dev (and later qa / prod) start empty. The first time the +prompt-agent deploy workflow runs against an empty environment, it reads +`prompt_agent_bootstrap` from `agentops.yaml` plus `prompt_file` and +creates the first version automatically. You do **not** repeat this +manual step for every environment. -In **each** project (sandbox first, then dev), do the same thing: +In the **sandbox** project only: 1. Open the [Azure AI Foundry portal](https://ai.azure.com) and select - the project. + the `travel-agent-sandbox` project. 2. Go to the agents area and create a new prompt-based agent. 3. Use these values: @@ -262,19 +275,28 @@ In **each** project (sandbox first, then dev), do the same thing: prices, or availability. ``` -5. Save and publish the agent. Foundry assigns version `1` (`travel-agent:1`). -6. Confirm both projects now show `travel-agent:1` with the same - instructions and the same model deployment. +5. Save and publish the agent. Foundry assigns version `1` + (`travel-agent:1`) in the sandbox project. The dev project still has + no agent at this point — that is expected. + +> **Why not seed dev too?** Forcing the operator to recreate the same +> prompt agent in every environment is exactly the manual drift problem +> AgentOps is here to eliminate. Step 9 adds a `prompt_agent_bootstrap` +> block to `agentops.yaml`; the first PR / deploy run against dev reads +> those defaults plus `prompt_file` and creates `travel-agent:1` in dev +> with the metadata trail (`agentops.prompt_sha256`, `agentops.git_sha`). +> Subsequent runs follow the normal reuse / next-version flow. > **Prompt-as-code captures only the instructions.** Later in the -> tutorial you will commit `.agentops/prompts/travel-agent.md` to git and -> let CI use it as the prompt source. That file does not capture the -> model deployment, parameters (temperature, top-p), tools, or other -> agent settings — those stay on the Foundry agent definition. If you -> ever need to change one of those, change it on the seed agent in -> **every** environment manually, or treat that change as a new release -> with its own review process. AgentOps will not detect drift in -> non-prompt fields. +> tutorial you will commit `.agentops/prompts/travel-agent.md` to git +> and let CI use it as the prompt source. That file does not capture +> the model deployment, parameters (temperature, top-p), tools, or +> other agent settings — those come from `prompt_agent_bootstrap` on +> the first deploy and stay on the Foundry agent definition afterwards. +> Use the same model deployment name in every Foundry project so the +> single `prompt_agent_bootstrap.model` value works everywhere without +> per-environment tweaks. AgentOps will not detect drift in non-prompt +> fields between environments. ## 5. Try the agent in the sandbox playground @@ -438,22 +460,37 @@ prices, or availability. '@ | Set-Content -Encoding utf8 .agentops\prompts\travel-agent.md ``` -Then tell `agentops.yaml` where to find the file: +Then tell `agentops.yaml` where to find the file and add +`prompt_agent_bootstrap` so CI can auto-create the agent in dev (and +later qa / prod) on the first deploy: ```yaml version: 1 agent: travel-agent:1 dataset: .agentops/data/travel-smoke.jsonl prompt_file: .agentops/prompts/travel-agent.md -``` - -The `agent: travel-agent:1` value is now a **seed pointer**. CI uses it to -look up the existing agent in the current environment's Foundry project, -copies its definition (model deployment, name, kind), and replaces the -instructions with the contents of `prompt_file`. If the prompt is -byte-identical to the looked-up seed's instructions, CI re-uses the same -Foundry version. If it differs, Foundry auto-creates the next version -number in that project. +prompt_agent_bootstrap: + model: gpt-4o-mini + description: "Helps plan short trips and explains tradeoffs." +``` + +The `agent: travel-agent:1` value is now a **seed pointer**. CI uses it +to look up the existing agent in the current environment's Foundry +project: + +- If the agent exists (the sandbox case, and every environment after + the first successful deploy), CI copies the looked-up definition + (model deployment, name, kind), replaces the instructions with the + contents of `prompt_file`, and either re-uses the same Foundry version + (when the prompt is byte-identical) or lets Foundry auto-create the + next number in that project (when it differs). +- If the agent does **not** exist (the empty dev / qa / prod case on + the first deploy), CI reads `prompt_agent_bootstrap` for the model + deployment (and optional `description`, `model_parameters`, `tools`) + and creates the first `travel-agent:1` from those defaults plus + `prompt_file`. The deploy artifact for that run records + `action: "bootstrapped"`. Subsequent deploys follow the + reuse-or-create flow above and ignore the bootstrap block. > **Versioning, in one paragraph.** You are not pinning Foundry's > version number — you are pinning the prompt. The number that gets @@ -466,6 +503,16 @@ number in that project. > when you want to repoint at a different stable seed version in > Foundry — not on every prompt change. +> **Keep `project_endpoint` out of `agentops.yaml` for multi-env work.** +> When `project_endpoint` is set in `agentops.yaml`, it wins over the +> `AZURE_AI_FOUNDRY_PROJECT_ENDPOINT` environment variable that azd +> environments rely on. That makes every command target the same +> Foundry project regardless of which env is active, which defeats the +> sandbox / dev / qa / prod split. The wizard does the right thing by +> default (it writes the endpoint to `.azure//.env`, not to +> `agentops.yaml`). If you ever copied the endpoint into `agentops.yaml` +> manually, delete it now. + ## 10. Check the selected eval runner ```powershell @@ -505,10 +552,20 @@ This creates two workflow files: The PR workflow now has two jobs: 1. **`stage-candidate`** — stages an ephemeral Foundry prompt-agent - candidate in the **dev** Foundry project (not sandbox) by copying - `travel-agent:1`'s definition and replacing instructions with - `prompt_file`. Writes `.agentops/deployments/agentops.candidate.yaml` - pointing at the staged candidate. + candidate in the **dev** Foundry project (not sandbox). + - On the **very first PR**, dev is still empty. The stage step reads + `prompt_agent_bootstrap` from `agentops.yaml` plus `prompt_file` + and creates `travel-agent:1` in dev. The candidate it evaluates is + that newly-bootstrapped version. The stage step reports + `action: bootstrapped`. + - On every subsequent PR, dev has a seed. The stage step looks up + `travel-agent:1`'s definition, replaces the instructions with + `prompt_file`, and either re-uses the same version (when the + prompt is byte-identical to the seed) or lets Foundry auto-create + the next number. The stage step reports `reused` or `created`. + In all cases, the workflow writes + `.agentops/deployments/agentops.candidate.yaml` pointing at the + staged candidate. 2. **`eval`** — runs `agentops eval run` against the candidate, then runs Doctor with `--severity-fail critical`. @@ -617,21 +674,33 @@ gh run view $runId --web gh run watch $runId --exit-status ``` -What you should see in the PR workflow run: +What you should see in the **first** PR workflow run (dev is still +empty at this point): 1. **Stage Foundry prompt candidate (PR)** job runs first. The - `prompt_deploy stage` step looks up `travel-agent:1` in the dev - project and compares the instructions in `prompt_file` against that - seed. - - If they are byte-identical: the stage step reports `reused` and - uses `travel-agent:1` as the candidate (no new version created). - - If they differ: Foundry auto-creates the next number (likely - `travel-agent:2`) and the stage step reports `created`. + `prompt_deploy stage` step tries to look up `travel-agent:1` in the + dev project and gets a 404. Because `agentops.yaml` includes a + `prompt_agent_bootstrap` block, the step: + - reads the `model` (`gpt-4o-mini`) and optional `description` from + `prompt_agent_bootstrap`, + - reads the instructions from `prompt_file`, + - creates `travel-agent:1` in the dev project from those defaults, + - reports `action: bootstrapped` and uses the freshly-bootstrapped + `travel-agent:1` as the candidate. 2. **AgentOps eval (PR gate)** job runs second. It evaluates the - candidate (re-used or created) using cloud eval. Thresholds pass. + bootstrapped candidate using cloud eval. Thresholds pass. Doctor runs with `--severity-fail critical`; advisory findings are listed but do not fail the job. +On every PR after this one, dev already has `travel-agent:1` as a seed, +so the stage step takes the normal lookup path: + +- If `prompt_file` is byte-identical to the seed's instructions: the + stage step reports `reused` and uses `travel-agent:1` as the + candidate (no new version created). +- If `prompt_file` differs: Foundry auto-creates the next number + (likely `travel-agent:2`) and the stage step reports `created`. + Now open a feature branch, modify a non-functional file (or just rerun the workflow), open a PR, and merge it once green: @@ -651,14 +720,15 @@ same version as the PR run), evaluates it, runs `prompt_deploy record` to mark it as the dev deployment, and uploads the deployment artifact. Open the deploy run and download the `foundry-agent-dev-deployment` -artifact. Inside, open `foundry-agent.json`: +artifact. Inside, open `foundry-agent.json`. On the very first deploy +into an empty dev project (the bootstrap case), the file looks like: ```json { "environment": "dev", "agent_source": "travel-agent:1", - "agent_candidate": "travel-agent:2", - "action": "created", + "agent_candidate": "travel-agent:1", + "action": "bootstrapped", "agentops": { "prompt_sha256": "9c3a...e0b1", "git_sha": "5f1a2c...", @@ -667,6 +737,12 @@ artifact. Inside, open `foundry-agent.json`: } ``` +On every subsequent deploy, `action` switches to either `reused` (when +the prompt is byte-identical to the previous seed) or `created` (when +Foundry auto-created a new version because the prompt changed), and +`agent_candidate` reflects the actual version that was evaluated and +recorded. + That `prompt_sha256` + `git_sha` pair is what the mental-model diagram at the start of the tutorial referred to as **cross-environment identity**. When you later add qa and prod deploys, each environment diff --git a/plugins/agentops/skills/agentops-config/SKILL.md b/plugins/agentops/skills/agentops-config/SKILL.md index a027436..e153097 100644 --- a/plugins/agentops/skills/agentops-config/SKILL.md +++ b/plugins/agentops/skills/agentops-config/SKILL.md @@ -76,6 +76,20 @@ thresholds: groundedness: ">=3" avg_latency_seconds: "<=30" +# Prompt-agent only: auto-bootstrap empty Foundry projects on first deploy. +# When the deploy workflow runs against a Foundry project that does not yet +# contain the agent named in `agent:`, AgentOps reads this block plus +# `prompt_file` and creates the first version automatically. Recommended +# for multi-environment prompt-agent workflows (sandbox → dev → qa → prod) +# so operators do not have to manually recreate the seed agent in every +# Foundry project. +prompt_agent_bootstrap: + model: gpt-4o-mini # required - same deployment name in every env + description: "Helps plan short trips." + # model_parameters: # optional - temperature, top_p, etc. + # temperature: 0.2 + # tools: [] # optional - tool definitions + # Publish results to the Foundry Evaluations panel. # - execution: local + publish: true → Classic Foundry (uploads metrics) # - execution: cloud → New Foundry (server-side run; diff --git a/plugins/agentops/skills/agentops-workflow/SKILL.md b/plugins/agentops/skills/agentops-workflow/SKILL.md index 084bfd2..1766fe6 100644 --- a/plugins/agentops/skills/agentops-workflow/SKILL.md +++ b/plugins/agentops/skills/agentops-workflow/SKILL.md @@ -405,16 +405,26 @@ Prompt-agent workflows: 1. read `prompt_file` from `agentops.yaml` or `AGENTOPS_AGENT_PROMPT_FILE`; -2. create or reuse a candidate Foundry prompt-agent version from that file; -3. generate `.agentops/deployments/agentops.candidate.yaml`; -4. run `agentops eval run` against the candidate version; -5. record `.agentops/deployments/foundry-agent.json` as a deployment +2. look up the seed agent in the active environment's Foundry project; + if it does not exist (typical first deploy into dev / qa / prod), + read the optional `prompt_agent_bootstrap` block from + `agentops.yaml` (required `model`, optional `description`, + `model_parameters`, `tools`) plus `prompt_file` and create the + first version automatically (recorded as `action: "bootstrapped"`); +3. otherwise create or reuse a candidate Foundry prompt-agent version + from `prompt_file`; +4. generate `.agentops/deployments/agentops.candidate.yaml`; +5. run `agentops eval run` against the candidate version; +6. record `.agentops/deployments/foundry-agent.json` as a deployment artifact only when the gate passes. This avoids the bad pattern of evaluating one agent version and deploying a different prompt. The invariant is: **evaluated version == deployed version**. Foundry manages agent versions; AgentOps owns the repo-side gate and -deployment record. +deployment record. For multi-environment prompt-agent workflows +(sandbox → dev → qa → prod), strongly recommend adding the +`prompt_agent_bootstrap` block so operators do not have to manually +recreate the seed agent in every Foundry project. If this is not a Foundry prompt agent and azd is not ready, generate `--kinds pr` only or use `--deploy-mode placeholder`. Do not ship diff --git a/src/agentops/core/agentops_config.py b/src/agentops/core/agentops_config.py index 2213cf3..6d39e27 100644 --- a/src/agentops/core/agentops_config.py +++ b/src/agentops/core/agentops_config.py @@ -204,6 +204,86 @@ def _version_non_empty(cls, value: str) -> str: return value +class PromptAgentBootstrap(BaseModel): + """Bootstrap defaults for prompt-agent CI/CD when the target Foundry + project does not yet contain the seed agent referenced by ``agent``. + + AgentOps' Foundry prompt-agent deployment path normally looks up an + existing seed (``name:version``) in the target project, clones its + definition, and replaces the instructions with ``prompt_file``. That + forces every environment (sandbox, dev, qa, prod) to have the agent + pre-created manually. + + When ``prompt_agent_bootstrap`` is set, the deployment step instead + bootstraps the agent in any environment whose target Foundry project + is still empty (the seed lookup returns 404) using these values plus + the contents of ``prompt_file``. The action recorded in the + deployment artifact will be ``bootstrapped`` for that first run. + + This block is **only** consulted on the not-found code path. Once + the agent exists in the target project, the reuse / next-version + flow takes over and ``prompt_agent_bootstrap`` is ignored — changing + ``model`` here will not migrate an existing dev agent to a new + deployment. Treat schema changes beyond ``instructions`` as a + deliberate operations event. + + Fields: + + ``model`` + Required. Azure OpenAI / Foundry model deployment name to use + when creating the agent. Must exist with the same name in every + environment that may bootstrap (sandbox, dev, qa, prod). + + ``description`` + Optional human-readable description recorded on the agent. + + ``model_parameters`` + Optional dict of model parameters (e.g. ``{"temperature": 0.2}``) + passed through to the Foundry ``PromptAgentDefinition``. + + ``tools`` + Optional list of tool definitions (JSON-serializable dicts that + match the Foundry tools schema) registered with the agent at + bootstrap time. + """ + + model: str = Field( + ..., + description=( + "Model deployment name. Must exist with the same name in " + "every Foundry project that may bootstrap from this config." + ), + ) + description: Optional[str] = Field( + None, + description="Optional human-readable description for the agent.", + ) + model_parameters: Optional[Dict[str, Any]] = Field( + None, + description=( + "Optional model parameters dict (e.g. {'temperature': 0.2}) " + "passed through to Foundry PromptAgentDefinition." + ), + ) + tools: Optional[List[Dict[str, Any]]] = Field( + None, + description=( + "Optional tool definitions (JSON dicts matching Foundry " + "tools schema) registered with the agent at bootstrap." + ), + ) + + model_config = ConfigDict(extra="forbid", protected_namespaces=()) + + @field_validator("model") + @classmethod + def _model_non_empty(cls, value: str) -> str: + value = value.strip() + if not value: + raise ValueError("prompt_agent_bootstrap.model must be non-empty") + return value + + # --------------------------------------------------------------------------- # Top-level config # --------------------------------------------------------------------------- @@ -339,6 +419,15 @@ class AgentOpsConfig(BaseModel): default_factory=DatasetSyncConfig, description="Cloud evaluation dataset submission policy.", ) + prompt_agent_bootstrap: Optional[PromptAgentBootstrap] = Field( + None, + description=( + "Optional bootstrap defaults used when the prompt-agent " + "deployment target is empty (seed lookup returns 404). " + "Lets CI/CD auto-create the agent in dev/qa/prod from " + "sandbox-only authoring. See PromptAgentBootstrap docs." + ), + ) model_config = ConfigDict(extra="forbid") diff --git a/src/agentops/pipeline/prompt_deploy.py b/src/agentops/pipeline/prompt_deploy.py index f0031b7..5d26241 100644 --- a/src/agentops/pipeline/prompt_deploy.py +++ b/src/agentops/pipeline/prompt_deploy.py @@ -16,7 +16,11 @@ from pathlib import Path from typing import Any, Dict, Optional -from agentops.core.agentops_config import AgentOpsConfig, classify_agent +from agentops.core.agentops_config import ( + AgentOpsConfig, + PromptAgentBootstrap, + classify_agent, +) from agentops.core.config_loader import load_agentops_config from agentops.utils.yaml import load_yaml, save_yaml @@ -37,6 +41,12 @@ def stage_prompt_agent_candidate( The generated eval config points at the candidate version, so the CI gate evaluates the same Foundry agent definition that the deploy stage records. + + If the target Foundry project does not yet contain the seed agent + referenced by ``agent`` (the SDK raises a 404), the function falls back to + bootstrapping the agent using ``prompt_agent_bootstrap`` from + ``agentops.yaml``. This lets CI/CD provision dev/qa/prod from a + sandbox-only authoring flow without requiring a manual portal step. """ config_path = config_path.resolve() @@ -64,41 +74,46 @@ def stage_prompt_agent_candidate( "or AZURE_AI_FOUNDRY_PROJECT_ENDPOINT" ) - current = _get_agent_version(endpoint, target.name, target.version) - definition = getattr(current, "definition", None) or _get_mapping_value(current, "definition") - if definition is None: - raise ValueError( - f"Foundry agent {target.name}:{target.version} did not include a definition" - ) + prompt_hash = hashlib.sha256(instructions.encode("utf-8")).hexdigest() - kind = str(_get_definition_value(definition, "kind") or "").lower() - if kind != "prompt": - raise ValueError( - f"Foundry agent {target.name}:{target.version} is kind {kind!r}; " - "prompt-agent deployment only supports kind 'prompt'" - ) + try: + current = _get_agent_version(endpoint, target.name, target.version) + except Exception as exc: # noqa: BLE001 — narrowed by _is_not_found_error below + if _is_not_found_error(exc): + current = None + else: + raise + + metadata = _deployment_metadata( + environment=environment, + prompt_hash=prompt_hash, + ) + description = ( + f"AgentOps {environment} candidate from " + f"{resolved_prompt.as_posix()} ({prompt_hash[:12]})" + ) - prompt_hash = hashlib.sha256(instructions.encode("utf-8")).hexdigest() - current_instructions = _get_definition_value(definition, "instructions") or "" - if str(current_instructions) == instructions: - candidate_version = target.version - action = "reused" - created = current - else: - candidate_definition = _copy_definition(definition) - _set_definition_value(candidate_definition, "instructions", instructions) - created = _create_agent_version( - endpoint, - target.name, - candidate_definition, - metadata=_deployment_metadata( - environment=environment, - prompt_hash=prompt_hash, - ), - description=( - f"AgentOps {environment} candidate from " - f"{resolved_prompt.as_posix()} ({prompt_hash[:12]})" - ), + if current is None: + # Bootstrap path: target project does not yet contain the seed agent. + if config.prompt_agent_bootstrap is None: + raise ValueError( + f"Foundry agent {target.name}:{target.version} does not exist " + f"in project {endpoint}, and 'prompt_agent_bootstrap' is not " + "configured in agentops.yaml.\n\n" + "Either create the agent manually in the target project, or " + "add prompt_agent_bootstrap defaults so CI/CD can create the " + "first version automatically. Minimal example:\n\n" + " prompt_agent_bootstrap:\n" + " model: \n\n" + "Then re-run the deploy workflow." + ) + created = _bootstrap_prompt_agent( + endpoint=endpoint, + agent_name=target.name, + bootstrap=config.prompt_agent_bootstrap, + instructions=instructions, + metadata=metadata, + description=description, ) candidate_version = str( getattr(created, "version", None) @@ -107,7 +122,44 @@ def stage_prompt_agent_candidate( ) if not candidate_version: raise ValueError("Foundry create_version did not return a version") - action = "created" + action = "bootstrapped" + else: + definition = getattr(current, "definition", None) or _get_mapping_value(current, "definition") + if definition is None: + raise ValueError( + f"Foundry agent {target.name}:{target.version} did not include a definition" + ) + + kind = str(_get_definition_value(definition, "kind") or "").lower() + if kind != "prompt": + raise ValueError( + f"Foundry agent {target.name}:{target.version} is kind {kind!r}; " + "prompt-agent deployment only supports kind 'prompt'" + ) + + current_instructions = _get_definition_value(definition, "instructions") or "" + if str(current_instructions) == instructions: + candidate_version = target.version + action = "reused" + created = current + else: + candidate_definition = _copy_definition(definition) + _set_definition_value(candidate_definition, "instructions", instructions) + created = _create_agent_version( + endpoint, + target.name, + candidate_definition, + metadata=metadata, + description=description, + ) + candidate_version = str( + getattr(created, "version", None) + or _get_mapping_value(created, "version") + or "" + ) + if not candidate_version: + raise ValueError("Foundry create_version did not return a version") + action = "created" eval_config_path = eval_config_path.resolve() output_path = output_path.resolve() @@ -160,7 +212,13 @@ def summarize_deployment(record_path: Path, *, environment: str) -> Dict[str, An handle.write("## Foundry prompt-agent deployment\n\n") handle.write(f"- **Environment:** {environment}\n") handle.write(f"- **Agent version:** `{candidate}`\n") + handle.write(f"- **Action:** `{action}`\n") handle.write(f"- **Prompt hash:** `{record.get('prompt_sha256', '')}`\n") + if action == "bootstrapped": + handle.write( + "- Agent did not exist in this environment; created from " + "`prompt_agent_bootstrap` defaults.\n" + ) if record.get("workflow_url"): handle.write(f"- **Workflow:** {record['workflow_url']}\n") return record @@ -203,6 +261,54 @@ def _get_agent_version(endpoint: str, agent_name: str, agent_version: str) -> An return client.agents.get_version(agent_name, agent_version) +def _is_not_found_error(exc: BaseException) -> bool: + """True when ``exc`` is an Azure 404 / ResourceNotFound, false otherwise. + + Deliberately narrow: 401/403/5xx / generic errors must propagate so + callers see auth, RBAC, or transport failures clearly instead of being + masked by a bootstrap path. + """ + + status = getattr(exc, "status_code", None) + if status == 404: + return True + try: + from azure.core.exceptions import ResourceNotFoundError # noqa: WPS433 + except ImportError: + return False + return isinstance(exc, ResourceNotFoundError) + + +def _bootstrap_prompt_agent( + *, + endpoint: str, + agent_name: str, + bootstrap: PromptAgentBootstrap, + instructions: str, + metadata: Dict[str, str], + description: str, +) -> Any: + """Create the first version of a prompt agent from bootstrap defaults.""" + + definition: Dict[str, Any] = { + "kind": "prompt", + "model": bootstrap.model, + "instructions": instructions, + } + if bootstrap.model_parameters: + definition["model_parameters"] = dict(bootstrap.model_parameters) + if bootstrap.tools: + definition["tools"] = [dict(tool) for tool in bootstrap.tools] + + return _create_agent_version( + endpoint, + agent_name, + definition, + metadata=metadata, + description=bootstrap.description or description, + ) + + def _create_agent_version( endpoint: str, agent_name: str, diff --git a/src/agentops/services/workflow_analysis.py b/src/agentops/services/workflow_analysis.py index 868775e..aa86778 100644 --- a/src/agentops/services/workflow_analysis.py +++ b/src/agentops/services/workflow_analysis.py @@ -163,6 +163,22 @@ def analyze_workflow_project(directory: Path) -> WorkflowAnalysis: str(agentops["prompt_file"]), ) ) + if prompt_agent and not agentops.get("prompt_agent_bootstrap"): + signals.append( + WorkflowSignal( + "prompt_agent_bootstrap_missing", + "Prompt-agent bootstrap defaults", + ( + "agentops.yaml targets a Foundry prompt agent but does not declare " + "`prompt_agent_bootstrap`. Without it, CI in an empty target Foundry " + "project (dev/qa/prod) fails with a 404 instead of auto-creating the " + "first version. Add `prompt_agent_bootstrap: { model: }` " + "to enable auto-bootstrap." + ), + "agentops.yaml", + confidence="medium", + ) + ) bicep_files = _find_files(root, "*.bicep") if bicep_files: @@ -723,6 +739,7 @@ def _signal_type(key: str) -> str: "agentops_cloud_evaluation": "Eval runner", "azd_project": "Deploy mode", "prompt_file": "Prompt source", + "prompt_agent_bootstrap_missing": "Prompt-agent bootstrap", "bicep_infra": "Infrastructure", "ailz_manifest": "Landing zone", "ailz_preflight": "Preflight", @@ -752,9 +769,11 @@ def _agentops_signal(root: Path) -> Dict[str, Any]: ) } prompt_file = data.get("prompt_file") if isinstance(data, dict) else None + bootstrap = data.get("prompt_agent_bootstrap") if isinstance(data, dict) else None return { "prompt_agent": target.kind == "foundry_prompt", "prompt_file": prompt_file, + "prompt_agent_bootstrap": bool(bootstrap), "signal": WorkflowSignal( "agentops_config", "AgentOps config", diff --git a/src/agentops/templates/agentops.yaml b/src/agentops/templates/agentops.yaml index b8252a2..910fb31 100644 --- a/src/agentops/templates/agentops.yaml +++ b/src/agentops/templates/agentops.yaml @@ -27,6 +27,24 @@ dataset: .agentops/data/smoke.jsonl # # prompt_file: .agentops/prompts/agent-instructions.md +# Optional. Bootstrap defaults for the *first* deploy into an empty Foundry +# project (e.g. brand-new dev / qa / prod). When the prompt-agent deploy +# workflow finds that 'agent' does not exist yet in the target environment, +# it creates the first version from these defaults plus the contents of +# 'prompt_file'. Subsequent deploys ignore this block and follow the normal +# reuse / next-version flow. Requires 'prompt_file' to be set. +# +# Tip: use the same model deployment name across sandbox / dev / qa / prod so +# the bootstrap payload works in every Foundry project without per-env tweaks. +# +# prompt_agent_bootstrap: +# model: gpt-4o-mini +# description: "Helps plan short trips and explains tradeoffs." +# # Optional model parameters and tool definitions are passed through as-is. +# # model_parameters: +# # temperature: 0.2 +# # tools: [] + # Optional. Override the auto-selected pass/fail thresholds. AgentOps fills in # sensible defaults for the metrics that are auto-selected. # diff --git a/src/agentops/templates/skills/agentops-config/SKILL.md b/src/agentops/templates/skills/agentops-config/SKILL.md index a027436..e153097 100644 --- a/src/agentops/templates/skills/agentops-config/SKILL.md +++ b/src/agentops/templates/skills/agentops-config/SKILL.md @@ -76,6 +76,20 @@ thresholds: groundedness: ">=3" avg_latency_seconds: "<=30" +# Prompt-agent only: auto-bootstrap empty Foundry projects on first deploy. +# When the deploy workflow runs against a Foundry project that does not yet +# contain the agent named in `agent:`, AgentOps reads this block plus +# `prompt_file` and creates the first version automatically. Recommended +# for multi-environment prompt-agent workflows (sandbox → dev → qa → prod) +# so operators do not have to manually recreate the seed agent in every +# Foundry project. +prompt_agent_bootstrap: + model: gpt-4o-mini # required - same deployment name in every env + description: "Helps plan short trips." + # model_parameters: # optional - temperature, top_p, etc. + # temperature: 0.2 + # tools: [] # optional - tool definitions + # Publish results to the Foundry Evaluations panel. # - execution: local + publish: true → Classic Foundry (uploads metrics) # - execution: cloud → New Foundry (server-side run; diff --git a/src/agentops/templates/skills/agentops-workflow/SKILL.md b/src/agentops/templates/skills/agentops-workflow/SKILL.md index 084bfd2..1766fe6 100644 --- a/src/agentops/templates/skills/agentops-workflow/SKILL.md +++ b/src/agentops/templates/skills/agentops-workflow/SKILL.md @@ -405,16 +405,26 @@ Prompt-agent workflows: 1. read `prompt_file` from `agentops.yaml` or `AGENTOPS_AGENT_PROMPT_FILE`; -2. create or reuse a candidate Foundry prompt-agent version from that file; -3. generate `.agentops/deployments/agentops.candidate.yaml`; -4. run `agentops eval run` against the candidate version; -5. record `.agentops/deployments/foundry-agent.json` as a deployment +2. look up the seed agent in the active environment's Foundry project; + if it does not exist (typical first deploy into dev / qa / prod), + read the optional `prompt_agent_bootstrap` block from + `agentops.yaml` (required `model`, optional `description`, + `model_parameters`, `tools`) plus `prompt_file` and create the + first version automatically (recorded as `action: "bootstrapped"`); +3. otherwise create or reuse a candidate Foundry prompt-agent version + from `prompt_file`; +4. generate `.agentops/deployments/agentops.candidate.yaml`; +5. run `agentops eval run` against the candidate version; +6. record `.agentops/deployments/foundry-agent.json` as a deployment artifact only when the gate passes. This avoids the bad pattern of evaluating one agent version and deploying a different prompt. The invariant is: **evaluated version == deployed version**. Foundry manages agent versions; AgentOps owns the repo-side gate and -deployment record. +deployment record. For multi-environment prompt-agent workflows +(sandbox → dev → qa → prod), strongly recommend adding the +`prompt_agent_bootstrap` block so operators do not have to manually +recreate the seed agent in every Foundry project. If this is not a Foundry prompt agent and azd is not ready, generate `--kinds pr` only or use `--deploy-mode placeholder`. Do not ship diff --git a/tests/unit/test_agentops_config.py b/tests/unit/test_agentops_config.py index a6832e0..7aad7bb 100644 --- a/tests/unit/test_agentops_config.py +++ b/tests/unit/test_agentops_config.py @@ -10,6 +10,7 @@ from agentops.core.agentops_config import ( AgentOpsConfig, DatasetSyncConfig, + PromptAgentBootstrap, Threshold, classify_agent, ) @@ -254,6 +255,69 @@ def test_dataset_sync_rejects_empty_name(self) -> None: } ) + def test_prompt_agent_bootstrap_defaults_to_none(self) -> None: + cfg = AgentOpsConfig(version=1, agent="my-rag:3", dataset="./qa.jsonl") + assert cfg.prompt_agent_bootstrap is None + + def test_prompt_agent_bootstrap_accepts_model_only(self) -> None: + cfg = AgentOpsConfig.model_validate( + { + "version": 1, + "agent": "travel-agent:1", + "dataset": "./qa.jsonl", + "prompt_agent_bootstrap": {"model": "gpt-4o-mini"}, + } + ) + assert isinstance(cfg.prompt_agent_bootstrap, PromptAgentBootstrap) + assert cfg.prompt_agent_bootstrap.model == "gpt-4o-mini" + assert cfg.prompt_agent_bootstrap.description is None + assert cfg.prompt_agent_bootstrap.model_parameters is None + assert cfg.prompt_agent_bootstrap.tools is None + + def test_prompt_agent_bootstrap_accepts_full_payload(self) -> None: + cfg = AgentOpsConfig.model_validate( + { + "version": 1, + "agent": "travel-agent:1", + "dataset": "./qa.jsonl", + "prompt_agent_bootstrap": { + "model": "gpt-4o-mini", + "description": "Plans short trips and explains tradeoffs.", + "model_parameters": {"temperature": 0.2, "top_p": 0.9}, + "tools": [{"type": "function", "name": "lookup"}], + }, + } + ) + bootstrap = cfg.prompt_agent_bootstrap + assert bootstrap is not None + assert bootstrap.model_parameters == {"temperature": 0.2, "top_p": 0.9} + assert bootstrap.tools == [{"type": "function", "name": "lookup"}] + + def test_prompt_agent_bootstrap_rejects_empty_model(self) -> None: + with pytest.raises(ValidationError): + AgentOpsConfig.model_validate( + { + "version": 1, + "agent": "travel-agent:1", + "dataset": "./qa.jsonl", + "prompt_agent_bootstrap": {"model": " "}, + } + ) + + def test_prompt_agent_bootstrap_rejects_unknown_fields(self) -> None: + with pytest.raises(ValidationError): + AgentOpsConfig.model_validate( + { + "version": 1, + "agent": "travel-agent:1", + "dataset": "./qa.jsonl", + "prompt_agent_bootstrap": { + "model": "gpt-4o-mini", + "totally_unknown": True, + }, + } + ) + def test_cloud_execution_rejects_publish_false(self) -> None: """execution: cloud + publish: false is a contradiction.""" with pytest.raises(ValidationError, match="always publishes"): diff --git a/tests/unit/test_prompt_deploy.py b/tests/unit/test_prompt_deploy.py index f225111..2816cb7 100644 --- a/tests/unit/test_prompt_deploy.py +++ b/tests/unit/test_prompt_deploy.py @@ -124,3 +124,253 @@ def test_stage_prompt_agent_candidate_reuses_unchanged_prompt( assert record["action"] == "reused" assert record["candidate_agent"] == "support-agent:3" + + +def _make_not_found(status: int = 404) -> Exception: + """Build an exception that ``_is_not_found_error`` will treat as 404.""" + + exc = Exception("simulated foundry 404") + setattr(exc, "status_code", status) + return exc + + +def test_stage_prompt_agent_candidate_bootstraps_empty_environment( + tmp_path: Path, + monkeypatch, +) -> None: + """When the target Foundry project is empty (404) and bootstrap defaults + are configured, the helper should create the first agent version from + those defaults plus the prompt file, and record ``action: bootstrapped``. + """ + + config = tmp_path / "agentops.yaml" + dataset = tmp_path / "data.jsonl" + prompt = tmp_path / "prompt.md" + dataset.write_text('{"input":"hi","expected":"hello"}\n', encoding="utf-8") + prompt.write_text("freshly authored instructions\n", encoding="utf-8") + config.write_text( + "\n".join( + [ + "version: 1", + "agent: travel-agent:1", + "dataset: data.jsonl", + "prompt_file: prompt.md", + "project_endpoint: https://example.services.ai.azure.com/api/projects/p", + "prompt_agent_bootstrap:", + " model: gpt-4o-mini", + " description: Helps plan short trips and explains tradeoffs.", + " model_parameters:", + " temperature: 0.2", + ] + ), + encoding="utf-8", + ) + + def fake_get(endpoint, name, version): + raise _make_not_found() + + created = SimpleNamespace(id="agent-version-1", version="1") + captured: dict = {} + + def fake_create(endpoint, name, definition, *, metadata, description): + captured["definition"] = definition + captured["metadata"] = metadata + captured["description"] = description + return created + + monkeypatch.setattr(prompt_deploy, "_get_agent_version", fake_get) + monkeypatch.setattr(prompt_deploy, "_create_agent_version", fake_create) + + record = prompt_deploy.stage_prompt_agent_candidate( + config_path=config, + environment="dev", + output_path=tmp_path / ".agentops/deployments/foundry-agent.json", + eval_config_path=tmp_path / ".agentops/deployments/agentops.candidate.yaml", + ) + + assert record["action"] == "bootstrapped" + assert record["candidate_agent"] == "travel-agent:1" + assert captured["definition"]["kind"] == "prompt" + assert captured["definition"]["model"] == "gpt-4o-mini" + assert captured["definition"]["instructions"] == "freshly authored instructions\n" + assert captured["definition"]["model_parameters"] == {"temperature": 0.2} + assert captured["description"] == "Helps plan short trips and explains tradeoffs." + assert captured["metadata"]["agentops.env"] == "dev" + + +def test_stage_prompt_agent_candidate_bootstrap_missing_config_raises_actionable_error( + tmp_path: Path, + monkeypatch, +) -> None: + """When the target project is empty (404) but bootstrap defaults are not + configured, the helper should raise a clear error guiding the user to + add ``prompt_agent_bootstrap`` to ``agentops.yaml``. + """ + + config = tmp_path / "agentops.yaml" + dataset = tmp_path / "data.jsonl" + prompt = tmp_path / "prompt.md" + dataset.write_text('{"input":"hi","expected":"hello"}\n', encoding="utf-8") + prompt.write_text("instructions\n", encoding="utf-8") + config.write_text( + "\n".join( + [ + "version: 1", + "agent: travel-agent:1", + "dataset: data.jsonl", + "prompt_file: prompt.md", + "project_endpoint: https://example.services.ai.azure.com/api/projects/p", + ] + ), + encoding="utf-8", + ) + + def fake_get(endpoint, name, version): + raise _make_not_found() + + monkeypatch.setattr(prompt_deploy, "_get_agent_version", fake_get) + monkeypatch.setattr( + prompt_deploy, + "_create_agent_version", + lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("unexpected create")), + ) + + import pytest + + with pytest.raises(ValueError) as excinfo: + prompt_deploy.stage_prompt_agent_candidate( + config_path=config, + environment="dev", + output_path=tmp_path / ".agentops/deployments/foundry-agent.json", + eval_config_path=tmp_path / ".agentops/deployments/agentops.candidate.yaml", + ) + + message = str(excinfo.value) + assert "prompt_agent_bootstrap" in message + assert "travel-agent:1" in message + assert "model:" in message + + +def test_stage_prompt_agent_candidate_auth_errors_propagate( + tmp_path: Path, + monkeypatch, +) -> None: + """403 / non-404 errors from ``get_version`` must NOT trigger the + bootstrap path; they must surface so the operator sees the real cause. + """ + + config = tmp_path / "agentops.yaml" + dataset = tmp_path / "data.jsonl" + prompt = tmp_path / "prompt.md" + dataset.write_text('{"input":"hi","expected":"hello"}\n', encoding="utf-8") + prompt.write_text("instructions\n", encoding="utf-8") + config.write_text( + "\n".join( + [ + "version: 1", + "agent: travel-agent:1", + "dataset: data.jsonl", + "prompt_file: prompt.md", + "project_endpoint: https://example.services.ai.azure.com/api/projects/p", + "prompt_agent_bootstrap:", + " model: gpt-4o-mini", + ] + ), + encoding="utf-8", + ) + + def fake_get(endpoint, name, version): + raise _make_not_found(status=403) + + monkeypatch.setattr(prompt_deploy, "_get_agent_version", fake_get) + monkeypatch.setattr( + prompt_deploy, + "_create_agent_version", + lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("unexpected create")), + ) + + import pytest + + with pytest.raises(Exception) as excinfo: + prompt_deploy.stage_prompt_agent_candidate( + config_path=config, + environment="dev", + output_path=tmp_path / ".agentops/deployments/foundry-agent.json", + eval_config_path=tmp_path / ".agentops/deployments/agentops.candidate.yaml", + ) + + # The error should be the original 403, NOT a "missing bootstrap" hint. + message = str(excinfo.value) + assert "prompt_agent_bootstrap" not in message + assert getattr(excinfo.value, "status_code", None) == 403 + + +def test_stage_prompt_agent_candidate_ignores_bootstrap_when_seed_exists( + tmp_path: Path, + monkeypatch, +) -> None: + """When the seed already exists, ``prompt_agent_bootstrap`` must be + ignored — the reuse / next-version flow takes over. Bootstrap only + affects the not-found code path. + """ + + config = tmp_path / "agentops.yaml" + dataset = tmp_path / "data.jsonl" + prompt = tmp_path / "prompt.md" + dataset.write_text('{"input":"hi","expected":"hello"}\n', encoding="utf-8") + prompt.write_text("same instructions\n", encoding="utf-8") + config.write_text( + "\n".join( + [ + "version: 1", + "agent: travel-agent:7", + "dataset: data.jsonl", + "prompt_file: prompt.md", + "project_endpoint: https://example.services.ai.azure.com/api/projects/p", + "prompt_agent_bootstrap:", + " model: gpt-4o", # different from current seed's model + ] + ), + encoding="utf-8", + ) + current = SimpleNamespace( + id="agent-version-7", + version="7", + definition={ + "kind": "prompt", + "model": "gpt-4o-mini", # bootstrap.model would override this if used + "instructions": "same instructions\n", + }, + metadata={}, + ) + monkeypatch.setattr( + prompt_deploy, + "_get_agent_version", + lambda endpoint, name, version: current, + ) + monkeypatch.setattr( + prompt_deploy, + "_create_agent_version", + lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("unexpected create")), + ) + + record = prompt_deploy.stage_prompt_agent_candidate( + config_path=config, + environment="prod", + output_path=tmp_path / ".agentops/deployments/foundry-agent.json", + eval_config_path=tmp_path / ".agentops/deployments/agentops.candidate.yaml", + ) + + assert record["action"] == "reused" + assert record["candidate_agent"] == "travel-agent:7" + + +def test_is_not_found_error_handles_404_and_rejects_others() -> None: + """Sanity-check the not-found classifier so the bootstrap fallback stays + narrow. + """ + + assert prompt_deploy._is_not_found_error(_make_not_found(404)) + assert not prompt_deploy._is_not_found_error(_make_not_found(403)) + assert not prompt_deploy._is_not_found_error(_make_not_found(500)) + assert not prompt_deploy._is_not_found_error(Exception("no status")) diff --git a/tests/unit/test_workflow_analysis.py b/tests/unit/test_workflow_analysis.py index 75fa5e4..a9fd1b5 100644 --- a/tests/unit/test_workflow_analysis.py +++ b/tests/unit/test_workflow_analysis.py @@ -51,6 +51,33 @@ def test_analysis_recommends_prompt_agent_without_azd(tmp_path: Path) -> None: assert analysis.classification == "Foundry prompt-agent project" assert any(signal.key == "prompt_file" for signal in analysis.signals) assert any("prompt_deploy stage" in " ".join(stage.commands) for stage in analysis.stages) + # Without prompt_agent_bootstrap, analyze must warn the user so first + # deploy into an empty Foundry project does not 404. + assert any( + signal.key == "prompt_agent_bootstrap_missing" for signal in analysis.signals + ) + + +def test_analysis_silent_when_prompt_agent_bootstrap_present(tmp_path: Path) -> None: + (tmp_path / "agentops.yaml").write_text( + "\n".join( + [ + "version: 1", + "agent: quickstart-agent:2", + "prompt_file: .agentops/prompts/agent-instructions.md", + "dataset: data.jsonl", + "prompt_agent_bootstrap:", + " model: gpt-4o-mini", + ] + ), + encoding="utf-8", + ) + + analysis = analyze_workflow_project(tmp_path) + + assert not any( + signal.key == "prompt_agent_bootstrap_missing" for signal in analysis.signals + ) def test_analysis_recommends_cloud_eval_for_supported_prompt_agent(tmp_path: Path) -> None: