Skip to content

test(smoke): report scheduled smoke failures as GitHub issues from the orchestrator #1796

Description

@jamesadevine

Context

PR #1791 removed the smoke-failure-reporter smoke case. It resolved its
targets by exact ADO definition name, which the lane model abolishes for
cases — a case is now a ref queued against a shared lane, not a definition —
and two of the three names it watched (Daily safe-output smoke canary 2545,
Daily smoke az CLI access 2546) are deleted at cutover. It could not be
repaired by editing names.

Its intent is still worth keeping:

A failed scheduled smoke run should become a GitHub issue, because nobody
watches ADO.

Right now nothing does this. A nightly candidate or daily released orchestrator
failure is visible only in ADO.

Proposal

File the issue from the orchestrator, deterministically, with no agent in
the loop.

scripts/ado-script/src/executor-e2e/github-issue.ts (279 lines) already does
exactly this job for the executor-e2e harness: scoped PAT, dedupe by exact
title, structured body, and diagnoseGitHubAuthFailure for token problems. It
exports fileFailureIssue, buildIssueTitle, renderIssueBody,
findOpenIssueByTitle and createGitHubIssue, so the reusable surface is
already factored out.

Sketch:

  1. Lift the generic parts of executor-e2e/github-issue.ts into a shared
    module (or import it directly — it is test-harness code, not a shipped
    bundle, so either is fine).
  2. Add a failure step to tests/smoke/orchestrator-steps.yml, gated on
    failed() and on the run being scheduled, so PR and manual runs stay
    silent — their failures are already surfaced on the PR.
  3. Title on the orchestrator + mode, e.g.
    [smoke-failure] ado-aw candidate compiler smoke (candidate), so dedupe
    collapses a recurring nightly failure into one issue.
  4. Body: the results table renderResultsTable already produces, plus the
    build URL and the per-case ref names.

Why this is better than what was deleted

old reporter proposed
Mechanism claude-sonnet-4.6 agent, 20 min deterministic step, seconds
Decides "did it fail?" by reading build logs via MCP the orchestrator's own exit code
Credentials GitHub PAT in an agent's reach PAT in a non-agent step
Tooling azure-devops MCP toolset none
Target discovery ADO definition names (now wrong) the orchestrator itself
Coverage 3 named definitions every case, via the run that owns them

Credential note

No smoke case currently files GitHub issues, so ADO_AW_GITHUB_TOKEN is
provisioned nowhere and the agentic lane holds no GitHub PAT beyond
GITHUB_TOKEN (Copilot CLI auth). This work re-introduces an issues PAT — put
it on the orchestrator definitions, not the lane, so it stays outside every
agent's blast radius. scripts/rotate-agentplayground-secrets.ps1 has a
comment marking where to add it.

Acceptance

  • A failed scheduled candidate run files exactly one issue
  • A failed scheduled released run files exactly one issue
  • A second consecutive failure updates/skips rather than filing a duplicate
  • PR and manual runs file nothing
  • A green run files nothing
  • The PAT never appears in an Agent or Detection step

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions