Operations

---

title: Deployment and Operations description: Current release workflow, identity prerequisites, evidence, and manual recovery boundaries.

Release path

Use deploy-and-evaluate.yml for the evaluated release path. The manual compatibility entry point, hosted-agent-cd.yml, calls that same reusable workflow, including staging evaluation, production approvals, and monitoring. Neither release entry point fires automatically on every push. The reusable workflow owns the shared deployment queue; the wrapper does not acquire a second lock.

flowchart TD
  Dispatch[Manual dispatch: selected commit] --> Tests[Lint and regression tests]
  Tests --> Infra[Bicep build and what-if]
  Infra --> Images[Build MCP images and resolve SHA-256 digests]
  Images --> Stage[Provision isolated staging and deploy candidate]
  Stage --> Smoke[Responses smoke, contract, streaming]
  Smoke --> Eval[Capture 8 cases and enforce strict quality gate]
  Eval --> Approval[Production environment approval]
  Approval --> Previous[Resolve current traffic selector and active version]
  Previous --> Prod[Provision evaluated MCP digests and rebuild agent source]
  Prod --> MonitorApproval[Production monitoring approval]
  MonitorApproval --> Monitor[Exact-version smoke and exception query]
  Prod -->|failure| Recovery[Manual recovery required]
  Monitor -->|failure| Recovery

Loading

Production is a remote rebuild of the evaluated source, not promotion of the identical hosted-agent runtime image. The MCP images are promoted by the exact evaluated digest. The production toolbox remains pinned to version 1; its two connection targets must continue to point at production MCP apps. Bicep owns those RemoteTool connections.

Identity and environment prerequisites

GitHub jobs use secretless Azure OIDC login. The CI deployment identity and the hosted agent runtime instance identity are different principals. Runtime model access is checked by scripts/configure-agent-rbac.sh: account-scoped Foundry User and Cognitive Services OpenAI User, with assignment scope, principal, and conditions validated. The Blueprint identity is not the runtime role-assignment target.

Repository variables include Azure client/tenant/subscription IDs, region, resource group, production project ID/endpoint, model name, ACR name, and Log Analytics workspace name. Staging constructs its own account/project endpoint and rejects production MCP URLs. Keep the production environment's required-reviewer protection configured. Do not remove it to make a release succeed.

Fresh CI runners do not possess azd's deployed-agent bookkeeping. The production resolver reads the remote agent's enabled state and endpoint selector, accepts one 100% FixedRatio rule with an explicit numeric version or @latest, seeds local azd name/version values, then verifies active status. Disabled agents, split routing, malformed selectors, and inactive versions fail closed. This fixes the pre-deploy failure in run 34174694298 without skipping rollback evidence or deploying first to discover a prior version.

Inspect a release

gh run view 34178081808 --repo devopsabcs-engineering/foundry-hosted-agents
gh run download 34178081808 --repo devopsabcs-engineering/foundry-hosted-agents -n evaluation-evidence

Use a new output directory when downloading another run. Inspect run-identity.json, captured.json, candidate-policy.json, and results.json together. A green judge summary alone is insufficient. The production artifact contains the before/after agent show responses; azd environment artifacts are short-lived operational state and should not be copied into public documentation or treated as a durable backup.

Repeat execution

Provisioning targets existing named resources and role assignments. A release still creates new immutable agent versions and build records; it is not a no-op deployment. Model wording, timing, and telemetry can differ between equivalent requests.

Trend publication uses the source run ID and attempt as its history key. Replaying completed run 34298522363, attempt 1, through publisher runs 34300067748 and 34300104333 succeeded twice without creating another wiki commit. This verifies publication replay, not release or model-output equivalence.

The web pilot also supports completed-message replay within an owner-bound, in-memory conversation. See Web Chat Pilot for its retry and expiry boundaries.

Monitoring and recovery

The production smoke step validates a completed, nonempty Responses SSE result for the new version. It allows three fresh-session attempts with 15-second delays. An HTTP 200 or an active deployment alone is insufficient because SSE can carry application errors.

The exception check queries the Log Analytics workspace directly:

AppExceptions | where TimeGenerated > ago(10m) | count

Any positive count breaches this workflow's threshold. An invalid or missing count fails as unknown health. The query is workspace-wide, not scoped to the new agent version, and does not establish trace completeness or sustained health. Continuous production quality evaluation and alert routing remain separate validation work.

If production provisioning, deployment, or monitoring fails:

1. Inspect the exact failed step and both production evidence artifacts.
2. Determine whether infrastructure or agent traffic changed; a pre-deploy lookup failure does not itself mean production changed.
3. Compare the prior routed version, newly created version, MCP image digests, and toolbox configuration. An agent-only reversal cannot undo changed MCP infrastructure.
4. Have the authorized operator choose and execute a recovery using a currently verified platform operation. Record the actual route and repeat a version-specific smoke test.

There is no automatic rollback, canary rollout, or tested disaster-recovery procedure in this workflow. Do not redeploy current source and label it a rollback. The failure job raises a manual-recovery signal; it does not reverse deployment. Establish recovery-time objectives, ownership, and a rehearsed rollback before a live-data production pilot.