Context
PR #1791 removed the smoke-failure-reporter smoke case. It resolved its
targets by exact ADO definition name, which the lane model abolishes for
cases — a case is now a ref queued against a shared lane, not a definition —
and two of the three names it watched (Daily safe-output smoke canary 2545,
Daily smoke az CLI access 2546) are deleted at cutover. It could not be
repaired by editing names.
Its intent is still worth keeping:
A failed scheduled smoke run should become a GitHub issue, because nobody
watches ADO.
Right now nothing does this. A nightly candidate or daily released orchestrator
failure is visible only in ADO.
Proposal
File the issue from the orchestrator, deterministically, with no agent in
the loop.
scripts/ado-script/src/executor-e2e/github-issue.ts (279 lines) already does
exactly this job for the executor-e2e harness: scoped PAT, dedupe by exact
title, structured body, and diagnoseGitHubAuthFailure for token problems. It
exports fileFailureIssue, buildIssueTitle, renderIssueBody,
findOpenIssueByTitle and createGitHubIssue, so the reusable surface is
already factored out.
Sketch:
- Lift the generic parts of
executor-e2e/github-issue.ts into a shared
module (or import it directly — it is test-harness code, not a shipped
bundle, so either is fine).
- Add a failure step to
tests/smoke/orchestrator-steps.yml, gated on
failed() and on the run being scheduled, so PR and manual runs stay
silent — their failures are already surfaced on the PR.
- Title on the orchestrator + mode, e.g.
[smoke-failure] ado-aw candidate compiler smoke (candidate), so dedupe
collapses a recurring nightly failure into one issue.
- Body: the results table
renderResultsTable already produces, plus the
build URL and the per-case ref names.
Why this is better than what was deleted
|
old reporter |
proposed |
| Mechanism |
claude-sonnet-4.6 agent, 20 min |
deterministic step, seconds |
| Decides "did it fail?" by |
reading build logs via MCP |
the orchestrator's own exit code |
| Credentials |
GitHub PAT in an agent's reach |
PAT in a non-agent step |
| Tooling |
azure-devops MCP toolset |
none |
| Target discovery |
ADO definition names (now wrong) |
the orchestrator itself |
| Coverage |
3 named definitions |
every case, via the run that owns them |
Credential note
No smoke case currently files GitHub issues, so ADO_AW_GITHUB_TOKEN is
provisioned nowhere and the agentic lane holds no GitHub PAT beyond
GITHUB_TOKEN (Copilot CLI auth). This work re-introduces an issues PAT — put
it on the orchestrator definitions, not the lane, so it stays outside every
agent's blast radius. scripts/rotate-agentplayground-secrets.ps1 has a
comment marking where to add it.
Acceptance
References
Context
PR #1791 removed the
smoke-failure-reportersmoke case. It resolved itstargets by exact ADO definition name, which the lane model abolishes for
cases — a case is now a ref queued against a shared lane, not a definition —
and two of the three names it watched (
Daily safe-output smoke canary2545,Daily smoke az CLI access2546) are deleted at cutover. It could not berepaired by editing names.
Its intent is still worth keeping:
Right now nothing does this. A nightly candidate or daily released orchestrator
failure is visible only in ADO.
Proposal
File the issue from the orchestrator, deterministically, with no agent in
the loop.
scripts/ado-script/src/executor-e2e/github-issue.ts(279 lines) already doesexactly this job for the executor-e2e harness: scoped PAT, dedupe by exact
title, structured body, and
diagnoseGitHubAuthFailurefor token problems. Itexports
fileFailureIssue,buildIssueTitle,renderIssueBody,findOpenIssueByTitleandcreateGitHubIssue, so the reusable surface isalready factored out.
Sketch:
executor-e2e/github-issue.tsinto a sharedmodule (or import it directly — it is test-harness code, not a shipped
bundle, so either is fine).
tests/smoke/orchestrator-steps.yml, gated onfailed()and on the run being scheduled, so PR and manual runs staysilent — their failures are already surfaced on the PR.
[smoke-failure] ado-aw candidate compiler smoke (candidate), so dedupecollapses a recurring nightly failure into one issue.
renderResultsTablealready produces, plus thebuild URL and the per-case ref names.
Why this is better than what was deleted
claude-sonnet-4.6agent, 20 minazure-devopsMCP toolsetCredential note
No smoke case currently files GitHub issues, so
ADO_AW_GITHUB_TOKENisprovisioned nowhere and the
agenticlane holds no GitHub PAT beyondGITHUB_TOKEN(Copilot CLI auth). This work re-introduces an issues PAT — putit on the orchestrator definitions, not the lane, so it stays outside every
agent's blast radius.
scripts/rotate-agentplayground-secrets.ps1has acomment marking where to add it.
Acceptance
References
45cc6059)scripts/ado-script/src/executor-e2e/github-issue.tstests/smoke/README.md— lanes and modes