feat(eval): add --agent and AGENTOPS_AGENT override for the eval target - #392
Merged
Conversation
…oduced
`agentops.yaml` pins a fully-qualified Foundry agent whose URL ends in a
version segment (`.../agents/helpdeskbot/versions/11`). The generated dev, QA,
and prod pipelines run `azd provision`, which publishes a new agent version,
and then run the eval gate. Nothing between those two steps retargeted the
pin, so the gate scored version 11 while the pipeline had just shipped version
12. The run went green on an artifact it had already replaced. A regression
introduced by the deploy could not fail its own gate, and in `prod` that same
shape gates a release.
The seam for the fix already existed and was dead. `RunOptions.agent_override`
sits at `orchestrator.py:56` and is consumed by all three execution backends
(`_run_evaluation_local`, `_run_evaluation_cloud`, `_run_evaluation_azd`) via
`classify_agent(options.agent_override or config.agent, config.protocol)`.
Repo-wide, `agent_override` appeared in exactly those four lines: no CLI flag,
no env var, no tests. Wiring it up is strictly cheaper and less risky than
inventing a parallel mechanism.
The reporter's first choice was for the deploy job to emit the resolved
version as a job output. That is not implementable against the shipped
templates. The only job that could observe a new version is `provision`, which
runs `azd provision` and nothing else, and `azd` does not surface a Foundry
agent version in any form AgentOps parses. Resolving "latest" at eval time was
the second option and is equally unavailable: the Foundry SDK surface used in
this repo is `client.agents.get_version(name, version)` and
`client.agents.create_version(name, body=...)`, with no list-versions helper to
build "latest" on. So this implements the consumption half of option 1, which
is the half that has to exist regardless of who supplies the value.
`agentops eval run` now takes `--agent`, falling back to `$AGENTOPS_AGENT`.
A bare number replaces just the version segment of the configured target; a
full agent reference replaces the target outright; empty or unset leaves the
config untouched, so existing pipelines are unchanged. Every generated eval
step forwards the variable: GitHub Actions emits
`AGENTOPS_AGENT: ${{ env.AGENTOPS_AGENT || vars.AGENTOPS_AGENT }}` so a
job-level env wins and a repo variable is the fallback, and Azure DevOps emits
`AGENTOPS_AGENT: $(AGENTOPS_AGENT)`. Both cover placeholder, azd, and
prompt-agent deploy modes across dev, QA, and prod. The official Foundry
eval-runner branch is left alone because it does not shell out to
`agentops eval run`.
Azure DevOps leaves `$(NAME)` in the environment verbatim when the variable is
undefined, unlike GitHub which substitutes an empty string. Without a guard
that literal would be parsed as an agent expression, so an unexpanded CI token
is treated as no override. That keeps ADO templates working without forcing
every pipeline to declare a `variables:` default.
`workflow analyze` now reports the pinned version as an `agent_version_pin`
signal plus a warning naming `AGENTOPS_AGENT`, so the drift is visible before
it reaches CI. The signal is restricted to `foundry_hosted` targets because
prompt-agent deploys were never affected: `stage_prompt_agent_candidate`
already writes `agentops.candidate.yaml` with a fresh `agent` value that the
eval step consumes.
Two corrections to the issue as filed. The templates order jobs
`provision -> eval -> deploy` (azd) and `eval -> build -> deploy`
(placeholder), not `provision -> deploy -> eval`; the substance holds because
`azd provision` is what bumps the agent version. And the bug is narrower than
reported, since prompt-agent mode was already immune.
Fixes #388
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
The previous commit on this branch claimed to fix stale-version evaluation. Review found that claim is wrong, and the reason is worth recording rather than quietly correcting. Nothing populates AGENTOPS_AGENT. A grep across src/ and templates/ finds the constant, the CLI consumer, the six injection sites, and nothing else. On GitHub Actions `${{ env.AGENTOPS_AGENT || vars.AGENTOPS_AGENT }}` resolves to an empty string; on Azure DevOps `$(AGENTOPS_AGENT)` arrives verbatim and the unexpanded-token guard strips it. Both paths fall back to the pin, so a user reproducing #388 sees behavior identical to before this branch. The injected expressions also cannot become correct by adding a producer later. They read same-job env scope, which cannot receive a value from a prior job or stage. That needs declared `jobs.<id>.outputs` plus `needs.<job>.outputs.*` on GitHub Actions, or explicit `stageDependencies` mapping on Azure DevOps. And the eval gate is structurally pre-deploy in every generated template, so if the deploy step is what publishes the new Foundry version then no value of AGENTOPS_AGENT can be correct at the time eval runs. Fixing #388 requires restructuring the templates, not assigning a variable. The override seam is still useful on its own, so it stays. It is now described as a feature: `agentops eval run --agent` and the AGENTOPS_AGENT fallback let a caller retarget a run without editing tracked config. The CHANGELOG entry moved from Fixed to Added and no longer claims to resolve stale-version evaluation. The `workflow analyze` warning was actively harmful and is gone. It fired on every correctly-pinned hosted project, including eval-only repos with no deploy pipeline at all, and it told users to export a value that no supported mechanism produces. The signal now fires only when the repo also has a generated deploy pipeline, which is the only place a deploy step could move the target underneath the gate, and its text describes the limitation instead of instructing an impossible action. Three cicd tests asserted only that a literal expression string appeared in the rendered YAML, which passes against a pipeline that does nothing. They now carry docstrings saying so, joined by a test that asserts no generated workflow declares job outputs or assigns AGENTOPS_AGENT, and by a parametrized CLI test that pins the actual runtime behavior: empty, whitespace, and unexpanded CI tokens all fall back to the configured agent. An empty value is deliberately not a hard failure, because every pipeline generated today passes exactly that and failing closed would break all of them. Also noted in the helper docstring: a non-numeric override is returned verbatim with no validation, so malformed input surfaces as a confusing classification failure rather than a clear message. Worth tightening when a producer lands. Unit suite: 1145 passed, 5 skipped. Refs #388 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
# Conflicts: # CHANGELOG.md
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
An override seam for the eval target, not a fix for #388. The first push on this
branch claimed to fix stale-version evaluation. It does not, and the reasons are
worth writing down so nobody rediscovers them.
agentops eval runnow accepts--agent, falling back to theAGENTOPS_AGENTenvironment variable. A bare number (
--agent 12) replaces just the versionsegment of the configured target; a full agent reference replaces it outright.
Unset means unchanged.
RunOptions.agent_overridealready existed atorchestrator.py:56and wasalready consumed by all three execution backends via
classify_agent(options.agent_override or config.agent, ...). A repo-wide grepfor
agent_overridereturned exactly four lines: the field and the threeconsumers. Nothing ever set it. This connects the CLI to that seam.
#388 stays open. No
Fixes/Closeshere.Why this does not fix #388
Nothing produces
AGENTOPS_AGENT. A grep acrosssrc/andtemplates/finds the constant, the CLI consumer, the six injection sites, and nothing else.
On GitHub Actions
${{ env.AGENTOPS_AGENT || vars.AGENTOPS_AGENT }}resolves toan empty string. On Azure DevOps
$(AGENTOPS_AGENT)arrives verbatim and theunexpanded-token guard strips it. Both paths fall back to the pin, so a user on
this build reproducing #388 sees behavior identical to 0.8.4.
A producer cannot be bolted on without restructuring the templates. Two
reasons, either of which is sufficient:
job order is
eval -> build -> deployfor placeholder mode andprovision -> eval -> deployfor azd mode, not theprovision -> deploy -> evalthe issue describes. If the deploy step is what publishes the newFoundry version, then no value of
AGENTOPS_AGENTcan be correct at the timeeval runs, because that version does not exist yet.
envscope, which cannot receive avalue from a prior job or stage. Producing one requires declared
jobs.<id>.outputsplusneeds.<job>.outputs.*on GitHub Actions, orexplicit
stageDependenciesmapping on Azure DevOps. Those six expressionswould have to be rewritten, not just fed.
Which azd verb creates the new agent version
Not determinable from this repository, and that is a real answer rather than a
dodge: this repo does not own the file that decides.
Evidence checked:
azure.yamlis shipped anywhere undersrc/. The generated azd workflowsinvoke a user-authored
azure.yamland fail with "No azure.yaml found"when it is absent.
provision, deployment packaging, and any hooks declared in azure.yaml"
(
agentops-deploy-dev-azd.yml).azd ai agentappears only as eval subcommands (eval init,eval generate,eval run,eval show) acrossazd_runner.py,azd_eval_init.py,orchestrator.py, and docs. There is noazd ai agent create/publish/deployanywhere in the repo.docs/ci-github-actions.md:260describesazd provisionas creating the ARMdeployment from
infra/main.bicep. Nothing describes either verb publishing aFoundry agent version.
client.agents.create_versioncall is inprompt_deploy.py:496, which isprompt-agent mode, not azd mode.
So the answer depends on the user's own
azure.yaml. An agent declared as anARM/Bicep resource under
infra/lands duringazd provision. An agentpublished by a
postdeployhook or a service definition lands duringazd deploy. Whoever fixes #388 needs to determine this per-project, or pick amechanism that does not depend on the answer (for example, having eval resolve
"latest" at run time, or moving the gate after deploy with an explicit rollback).
Prompt-agent mode is the one case the repo does answer, and it is already
immune:
prompt_deploy.stage_prompt_agent_candidatewrites.agentops/deployments/agentops.candidate.yamlwith a freshagentvalue, andcicd.py:906points the prompt-agent eval step at it.workflow analyzeThe previous push added a warning that fired on every correctly-pinned hosted
project, including eval-only repos with no deploy pipeline, and told users to
export
AGENTOPS_AGENT"with the version the deploy step resolved" when nothingproduces that. Unactionable advice at scale is worse than silence.
The warning is gone. The signal now fires only when the repo also has a generated
deploy pipeline, which is the only place a deploy step could move the target
underneath the gate, and its text describes the limitation and points at #388
rather than instructing an impossible action.
Known limitations of this seam
OFFICIAL_EVAL_RUNNERignores the override entirely. That branch(
cicd.py:523-590) routes throughofficial_eval.py, which callsclassify_agent(config.agent, ...)with no override parameter.--agenthaszero effect there.
--config {config_path}with no override. Once eval can follow a deployedversion, those gates would score a different artifact than eval did.
apply_agent_version_overridereturns anon-numeric override verbatim, so malformed input (
12abc, a truncatedendpoint URL) surfaces as a confusing
classify_agentfailure rather than aclear message. Noted in the helper docstring; worth tightening when a producer
lands and starts generating these programmatically.
An empty
AGENTOPS_AGENTis deliberately not a hard failure. Every pipelinethis tool generates today passes exactly that, so failing closed would break all
of them.
Tests
tests/unit/test_eval_agent_override.pycovers precedence (--agentbeats theenv var), the unexpanded-
$(NAME)guard, verbatim full references, and a loudfailure when the configured target has no version slot. These were verified
failing-first by stashing the source changes.
Two tests exist specifically to prevent this PR from reading as more than it is:
test_no_generated_workflow_produces_a_value_for_the_overrideasserts that nogenerated workflow declares job outputs or assigns
AGENTOPS_AGENT. It failsthe moment a producer lands, which is when the Generated workflows evaluate a stale agent version: agentops.yaml pins versions/N and deploy never rewrites it #388 notes need updating.
test_generated_ci_today_evaluates_the_pin_because_nothing_sets_the_overridepins the real runtime behavior: empty, whitespace, and unexpanded CI tokens all
fall back to the configured agent.
The three render assertions in
test_cicd.pynow carry docstrings stating thatthey check plumbing only and pass against a non-functional pipeline.
Unit suite: 1145 passed, 5 skipped (baseline 1116/5).
Relationship to #391
No semantic conflict. The two changes touch disjoint constructs inside the same
string literals, #391's stricter assertions hold under these changes, and neither
change is silently dropped. Only
CHANGELOG.mdconflicts, mechanically. Mergeorder does not matter.