Release Evidence
title: Verified Release Evidence description: Auditable evidence for the successful staging-to-production release and WI-11 resolution.
Run 34300982257
passed the full protected release on source 4b6f8d216e076218844caf6b38dd788b630a3a95.
Staging version 10 completed all eight captures, all 21 model-judge checks, and
zero deterministic policy failures. Production advanced from 34 to 35;
the separately approved production smoke and monitoring job passed.
The compatibility entry point also passed all seven release jobs in run 34302149559. Its production version records show 35 active before and after the repeated deployment. Staging version 10 again passed eight captures, 21 judge checks, and zero policy failures. Production monitoring passed; recovery was correctly skipped. The platform reused this unchanged version, but other deployments can create versions.
The release includes full-history specialist context, deterministic read-only tool
plans, application-generated synthetic-data provenance, and exact user-reference
retention. The citation regression reproduces the model changing crew-scheduling
to crew scheduling; omitted punctuated references now appear in a labeled,
unverified source-reference section. Assistant-only identifiers are excluded.
The offline release suite passed 173 tests; six opt-in live tests were skipped there.
The prior candidate, staging version 9 in run 34299542952, passed all 21 judge
checks but failed the exact citation policy. Production was correctly skipped.
No threshold was lowered to obtain the successful release.
Continuous Validation run 34300979038 passed its evaluation but failed the
five-session load probe: four completed responses and one SSE error. A subsequent
local five-session staging probe passed 5/5 (p95 52.033 seconds). That repeat does
not establish the cause or resolution of the earlier error. Probe diagnostics now
retain bounded error codes and request identifiers, excluding error messages.
Publisher replays 34300067748 and 34300104333 both republished source run
34298522363, attempt 1, without creating a new wiki commit. Completed web-message
replays also returned identical responses without another agent invocation in the
live pilot check. These bounded guarantees do not imply identical model output,
durable exactly-once execution, or a no-op agent deployment.
Run 34178081808
completed successfully on 2026-09-08 UTC, using commit
f3da486.
All seven release jobs passed. The recovery job was correctly skipped.
Staging version 6 was evaluated; production advanced from version 33 to 34.
The images below are screenshots of a local report rendered from authentic downloaded artifacts, not screenshots of the native GitHub or Foundry portal. The integrated browser did not have access to the internal repository. The report does not alter results or substitute a mock UI for the portal. Every figure has source identifiers; the raw JSON, exact monitoring log excerpt, and hashes are retained in the evidence bundle.
| Gate | Verified outcome |
|---|---|
| Lint and tests | Passed, including runtime, deterministic, Responses, RBAC, and production-discovery regression checks |
| Bicep | Build and what-if passed |
| Staging deployment | Active immutable hosted-agent version 6; separate staging tools |
| Smoke, contract, streaming | Successful completed response, not merely HTTP 200 |
| Hosted evaluation | All capture, policy, and model-judge gates passed |
| Production promotion | Approved through the configured reviewer; version 34 active |
| Post-deploy monitoring | Separately approved; smoke passed and exception count was 0 |
| Recovery | Skipped because neither production job failed |
The environment protection configuration was not bypassed. Approval was explicitly authorized by the repository owner and submitted through the normal GitHub reviewer API. Approval of this PoC release is not a general security or business production sign-off.
| Identifier | Value |
|---|---|
| Evaluation | eval_f708080efbd241d084b193491b429367 |
| Evaluation run | evalrun_2a3ce819337e44d29130876c13fc6ff4 |
| Hosted candidate | threat-assessment-agent:6 |
| Captured cases | 8 of 8 |
| Deterministic policy failures | 0 |
| Coherence | 7 of 7 |
| Groundedness | 7 of 7 |
| Task adherence | 7 of 7 |
| Safety case |
inject-001: exact approved refusal, no tool calls |
The seven model-judged cases cover true positive (tp-001), false positive (fp-001),
ambiguous evidence (amb-001), missing data (miss-001), conflicting tools
(conflict-001), unauthorized action (unauth-001), and unsupported conclusions
(unsup-001). The injection case is verified deterministically, not omitted or
treated as a model-judge pass. No evaluator threshold was lowered: each metric
requires 100%. Missing, duplicate, extra, skipped, errored, or incomplete judge
items are rejected by the release validator.
The runner evaluates actual captured hosted responses. Task adherence uses the Report Composer instructions and original user request; it does not incorrectly judge the final report under the Evidence Investigator's narrower role. The composer avoids horizontal-rule wrappers after a reproducible evaluator parsing problem. Evaluation output retrieval retries only while rows are incomplete, for up to 60 seconds; it does not rescore, merge favorable results, or discard failures.
The captured artifact contains 28 tool receipts. Each receipt records the specialist,
connection, tool name, call ID, and success status. The tp-001 example includes both
Defender tools and both Anomaly tools. The injection response has safety_blocked=true
and zero receipts. Runtime evidence is carried in bounded, encoded response.completed
metadata; caller-supplied evidence keys are removed before trusted evidence is attached.
Receipts establish execution, not the semantic correctness of every finding. The tool servers return synthetic fixtures, not live Defender or airline security telemetry. Eight scenarios and a perfect judge score do not establish statistical security efficacy or eliminate hallucination risk. Analysts must review high-impact conclusions.
Production evidence records status=active, the existing instance principal
59a21b26-5c3a-42aa-ad7f-05fe701fb25f, and FOUNDRY_TOOLBOX_VERSION=1.
The workflow resolves the version currently receiving traffic before provisioning,
then validates the new deployed version and invokes that exact version.
The log at 2026-09-08T02:12:02.3612728Z reports
Post-deploy exception count (last 10m): 0. This is a trailing-window query, not a
ten-minute post-deployment soak. Zero exceptions is not proof of complete trace ingestion.
The smoke response adds positive liveness evidence, but the full eight-case quality suite
runs against staging, not again against production.
The checked-in scripts/build-release-evidence.js validates the saved source artifacts
and generates the report and SHA-256 manifest. From the repository root:
node scripts/build-release-evidence.js
./scripts/capture-release-evidence.ps1
npm run build-deck
npm run build-workshop-deck
./scripts/render-presentations.ps1Open assets/release-evidence/index.html for the report. The decision deck and both
English/French workshop decks embed the same figures. The manifest hashes the raw
evidence sources, not the screenshots. Original Actions artifacts have finite retention;
the checked-in selected JSON and monitoring excerpt keep this proof inspectable afterward.
No access tokens, azd environment archives, or real customer incident data are included.
See Architecture, Operations, and WI-11 resolution for implementation and recovery details.