Generated workflows evaluate a stale agent version: agentops.yaml pins versions/N and deploy never rewrites it

[placerda]

Summary

The generated dev/qa/prod workflows deploy a new agent version and then evaluate a different, older one. agentops.yaml pins a fully-qualified agent URL that includes the version segment, and nothing in the deploy job rewrites it, so every CI run evaluates whatever version was current when agentops init was executed.

Repro

Observed end-to-end while rehearsing the Foundry hosted-agent flow (execution: cloud, protocol: responses).

1. agentops init writes a pinned target into agentops.yaml:

   agent: https://<account>.services.ai.azure.com/api/projects/<project>/agents/helpdeskbot/versions/11

2. The generated dev workflow runs provision → deploy → eval.

3. deploy publishes the agent and Foundry creates version 12.

4. eval then runs against version 11, because agentops.yaml still names it.

The run is green. The evaluated artifact is not the one that was just deployed.

Why this matters

This is the quality gate for the deploy. Silently evaluating a stale version means:

• A regression introduced by the deploy cannot fail the gate.
• The eval results attached to the run do not describe the deployed agent.
• The failure mode is invisible: no warning, no diff, green checks.

It is worse in prod, where the same workflow shape is used to gate a release.

Expected behaviour

At minimum, the eval job should evaluate the version the deploy job just produced. Options, roughly in order of preference:

1. Deploy job emits the resolved version as a job output, and the eval job consumes it to override the target (e.g. an AGENTOPS_AGENT env var or a --agent flag). Keeps agentops.yaml declarative and avoids CI writing to the repo.
2. Support an unversioned agent URL in agentops.yaml that resolves to "latest" at eval time. Simple, but loses the ability to pin deliberately.
3. Deploy job rewrites agentops.yaml in the workspace before eval. Works, but mutating tracked config mid-run is the least pleasant of the three.

Whichever is chosen, agentops workflow analyze should probably surface the pin so the drift is visible before it reaches CI.

Notes

• Reproduced against agentops-accelerator 0.8.4.
• Not related to Deploy workflows: provision and deploy jobs never install the azd extension nor run 'azd auth login' #379 / execution: cloud still emits setup-azd and azd extension install steps #375; those are about azd CLI setup and eval-runner selection in the same workflows. This one is about the eval target.