feat(smoke): replace per-case pipelines with lane-based smoke suite - #1791
Merged
Conversation
|
Azure Pipelines: Successfully started running 2 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run. |
6 tasks
Collaborator
Author
|
Removed the \smoke-failure-reporter\ case in \45cc6059; follow-up filed as #1796. It resolved targets by exact ADO definition name, which the lane model abolishes for cases — a case is a ref queued against a shared lane, not a definition — and two of the three names it watched (2545, 2546) are deleted at cutover. Not repairable by editing names. Two knock-on effects worth flagging for review: Released mode is green today, without waiting for a release. #1670 merged \create-github-issue, which v0.48.0 does not know, and released mode compiles \HEAD\ sources with the last released binary. Verified against the real v0.48.0 asset:
No smoke case files GitHub issues any more, so the \�gentic\ lane holds no GitHub PAT beyond \GITHUB_TOKEN\ (Copilot CLI auth). \ADO_AW_GITHUB_TOKEN\ is provisioned nowhere. #1796 re-introduces it on the orchestrator, not the lane. The version-skew constraint that caused this is documented in \ ests/smoke/README.md\ — it is inherent to dropping the bot-regenerated lock files and will recur for any released-mode case adopting unreleased front matter. |
Adding a smoke cost a manual ADO definition registration, secret provisioning, service-connection authorization, fork-hardening, an orchestrator variable, a placeholder commit, a committed lock file and a TypeScript change - ten definitions and five locks in total. An ADO definition binds (repo, yamlFilename) but the ref is supplied per queue, so every case now compiles to the same .smoke/pipeline.yml path and is pushed to its own per-case ref. The ref carries the test case; the definition carries only the credentials. - Cases are declared in tests/smoke/cases.json and loaded by cases.ts with strict fail-closed validation (case ids become git ref segments). - Three lane definitions replace ten per-case ones, cut by credential class: agentic, debug (ADO_AW_DEBUG_GITHUB_TOKEN), infra (reserved for AWF and the ado-proxy sidecar, and ready for kind: raw cases). - Two modes share one steps template: candidate (compiler built from the commit) and released (latest release asset, release URLs required). The latter replaces the five committed *.lock.yml files and the bot workflow that kept them fresh. - Ref cleanup is now per case: one unproven build no longer strands every other case's ref. Adding a smoke is now a markdown file plus one manifest entry. Two dependencies that would otherwise have broken silently: executor-e2e's queue-build scenario targeted the noop-target definition, so it gets a dedicated static queue-target.yml; and the weekly janitor becomes a released-mode case (daily, its 30-day prune window is idempotent). Trigger hardening is two steps rather than three. Since on: became the complete declaration of when a pipeline runs, stripping it makes the compiler emit an explicit trigger: none / pr: none, so the harness no longer patches those keys into the staged copy. assertNoTriggers still verifies the staged bytes before push: ADO reads a MISSING trigger: as "CI on every branch", so a compiler that regressed to omitting it would let a ref push queue the shared lane on top of the API-queued run. The staged copy is now byte-identical to the pristine lock committed beside it. kind: raw sources have no compiler in the loop and so must declare both keys themselves. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
…oring guide The list conflated checks the harness actually performs with conventions an author must follow. arget: standalone and the credential-subset rule are neither parsed nor asserted anywhere, so a case declaring a safe output whose token its lane does not carry compiles, stages and queues cleanly, then fails in Stage 3. That distinction gets load-bearing as GitHub issue filing becomes a public configured-only safe output rather than an \�do-aw-debug:\ one: the obvious fix for such a Stage 3 failure is to add the token to the \�gentic\ lane, which would dissolve the isolation the lanes exist to provide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
…defer the infra lane Follows #1670, which promotes GitHub issue filing from \�do-aw-debug\ to the public \create-github-issue\ safe output and renames the secret. Same token value, same scope, same single case - the debug lane is a credential boundary, so a rename does not move the boundary. Also records that only \�gentic\ and \debug\ need registering at cutover. \loadCases\ resolves a definition id per lane in play for the mode being run, and no case targets \infra\ yet, so registering it now would create a credentialed definition nothing queues. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
…abling them The disable-first plan bought a rollback nobody wants: re-enabling ten definitions still leaves them without the secrets, service-connection authorizations and fork hardening that a working smoke needs. Deleting makes the tracked ids matter for exactly one reason, now stated where it can be acted on. The trigger-policy audit fetches every id in scheduled_only_definition_ids with curl --fail-with-body, so a deleted definition 404s, exhausts its three retries and aborts the run with 'Unable to audit scheduled-only definition <id>'. It fails closed rather than passing silently, but it fails every smoke run until the policy file is corrected - so 2545-2549 must leave that file in the same commit that deletes them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
assertAdoTokenIsolation covered four ADO credentials but not the GitHub PAT, so nothing in the smoke suite would have caught a regression that projected it into Stage 1. It is the one credential here that grants write access outside the AgentPlayground project - Issues write on an external GitHub repo - so a leak into the agent is a reach-outside-ADO escape rather than a widening within a sandbox already scoped to the project. The compiler confines it to the Stage 3 executor env today (generate_executor_ado_env), but that is a per-workflow compile-time property with no regression guard. This asserts it on freshly compiled YAML, before push, so a regression fails the run rather than being contained by the lane split. GITHUB_TOKEN stays permitted: it is Copilot CLI auth and the agent legitimately receives it. Pinned by its own test so a later tightening cannot conflate the two. Mutation-checked: dropping the entry fails both new cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
The debug lane isolated ADO_AW_GITHUB_TOKEN for a single case, back when GitHub issue filing was a debug-only capability behind \�do-aw-debug:\. #1670 promotes it to the public create-github-issue safe output, so a lane per credential would fragment as more cases adopt it - each new issue-filing case would either need its own lane or quietly widen this one, which is the failure mode the split existed to prevent. The isolation that mattered is now enforced where it cannot drift. The compiler projects the token into the Stage 3 executor only, and assertAdoTokenIsolation fails the run on freshly compiled YAML if it appears in Agent or Detection. That prevents the leak rather than bounding its blast radius, which is all a separate definition bought. infra stays: it holds no credentials at all, so it remains a real boundary for the AWF and ado-proxy smokes. The manifest test that pinned the debug split now asserts infra carries no cases, so the boundary cannot be dissolved by quietly provisioning a secret onto it. Cutover now needs ONE lane definition registered, not three. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
Provisioned in AgentPlayground and recorded here: - .smoke/pipeline.yml added to refs/heads/ado-aw-smoke-candidate-base (commit 1d173bc), carrying inert-child.yml - lane definition 'ado-aw smoke lane - agentic' registered as 2567, no triggers, default branch = the inert base ref - agent-playground-read/write authorized on 2567 - SMOKE_LANE_AGENTIC_DEFINITION_ID=2567 set on orchestrator 2559 - 2567 added to scheduled_only_definition_ids so the policy audit covers it The legacy tests/**/*.lock.yml paths were deliberately LEFT on the base ref: the ten retired definitions still point at them, so removing them before cutover would break the smokes that are currently running. They go with the definitions at the end. Also records a break this runbook had missed. Definition 2559 points at /tests/compiler-smoke-e2e/azure-pipelines.yml, which this change deletes, so the candidate orchestrator breaks on its next run unless it is repointed at /tests/smoke/azure-pipelines-candidate.yml. It cannot be repointed in advance because the new path does not exist on main until merge, so it is now an explicit merge-time step rather than an assumption. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
…the lanes replace 'Candidate compiler smoke' named two different things: the orchestrator 2559, and the six per-case children 2554-2565. Only the children are replaced by the lane model. An orchestrator builds or downloads the compiler, publishes the candidate artifact, stages each case to its own ref and queues the lane - work that cannot live in a lane, because a lane runs a staged pipeline from the mirror while an orchestrator runs from GitHub. Also pairs two edits that must land together at the repoint. The old orchestrator YAML reads the six COMPILER_SMOKE_*_DEFINITION_ID variables on 2559 and the new one never does, so removing them before the repoint breaks the running smoke, and leaving them afterwards keeps live pointers to deleted definitions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
…d-mode version skew The rotation script still named only the retired per-case definitions, so running it would have rotated secrets onto definitions being deleted while leaving the new lane with none - the lane cannot run at all until GITHUB_TOKEN is present. Adds 2567, and sets the issues PAT under BOTH names during the cutover: ADO_AW_DEBUG_GITHUB_TOKEN for the committed release-owned locks that 2549/2558 still run, and ADO_AW_GITHUB_TOKEN for the lane once a release ships #1670. Both are dropped with those definitions at the end of the cutover. Also documents a constraint released mode inherits from dropping the committed locks. Those were regenerated by a bot after each release, so lock and binary always agreed; released mode instead compiles HEAD sources with the last released binary, so a source adopting unreleased front matter fails to compile - ado-aw rejects unknown safe-output keys outright. Verified against the real v0.48.0 asset: smoke-failure-reporter now fails ('unrecognised tool name: create-github-issue') because #1670 is merged but unreleased. canary, azure-cli, noop-target and janitor all still compile, so the blast radius is one case until the next release. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
The reporter resolved its targets by exact ADO definition NAME, which the lane model abolishes for cases: a case is a ref queued against a shared lane, not a definition, so there is no 'canary definition' to look up. Two of the three names it watched - 'Daily safe-output smoke canary' (2545) and 'Daily smoke az CLI access' (2546) - are deleted at cutover anyway. It could not be repaired by editing names. Deleting it also removes the only released-mode case that could not compile. Verified against the real v0.48.0 asset: all four remaining released cases now compile, where smoke-failure-reporter failed with 'unrecognised tool name: create-github-issue' because #1670 is merged but unreleased. Released mode is green today rather than after a release. Knock-on simplification: no smoke case files GitHub issues any more, so the lane needs no GitHub PAT beyond Copilot CLI auth. ADO_AW_GITHUB_TOKEN is provisioned nowhere. Its intent - turn a failed scheduled run into a GitHub issue, because nobody watches ADO - is worth keeping and is filed as a follow-up. It belongs in the orchestrator as a deterministic step reusing executor-e2e/github-issue.ts, which already does exactly this job with title-based dedupe and no agent in the loop. Removes test_smoke_failure_reporter_uses_registered_ado_names_and_staging_repo, whose assertions were entirely about the deleted fixture's contents. The general contracts it touched keep coverage elsewhere: ADO_MCP_AUTH_TOKEN at five other sites, assert_job_execution_env_excludes_ado_credentials at six other call sites. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
…n step Confirms the secrets are provisioned on 2567 and adds a step the runbook was missing. A new lane definition needs the agent POOL authorized in addition to the service connections, and its absence does not surface as an error: the build queues, sits at status notStarted indefinitely, and the timeline shows Checkpoint.Authorization inProgress. No failure, no timeout - it simply never starts. Found by queueing the lane and watching it hang for seven minutes; every pre-existing definition was already on the pool's explicit allowlist, so this only bites on newly registered ones. Also records a live verification. Build 629504 queued 2567 against the base ref and failed at 'Reject inert candidate-smoke base' with 'Candidate compiler smoke must be queued with an explicit generated ref.' That failure is the pass condition: it proves checkout, pool, YAML path and the inert guard all work, and that a lane cannot run without an explicitly supplied case ref. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
…file removal Rebasing onto #1670 surfaced that this branch had dropped the 'Adding a new safe output' section along with the lock-file machinery it sat next to. The guidance itself was never lock-specific - it routes a new tool to executor-e2e (ADO write path) or signals.ts (signal-only) - so it is restored rather than lost. Two corrections while restoring it. The old point 5 said debug-only tools 'currently only create-github-issue' are excluded from both suites and exercised by smoke-failure-reporter.md; #1670 made that tool a public safe output and this branch deleted that case, so both halves were wrong. It now points at #1798, which tracks the executor-e2e gap. The 'Running locally' section referenced ado-aw check against committed locks that no longer exist, and a manual handoff runbook that now lives in tests/smoke/REGISTERED.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
jamesadevine
force-pushed
the
jamesadevine/lane-based-smoke-suite
branch
from
August 3, 2026 15:19
4e463fb to
7b05345
Compare
jamesadevine
added a commit
that referenced
this pull request
Aug 4, 2026
The plan doc was committed at the repository root so the redesign could be reviewed alongside the change, and its own header says to delete it once the cutover completes and the content has landed in tests/smoke/README.md. Both have now happened. Everything durable lives in the suite's own docs: the lane/ref model, modes, trigger handling and fork boundary in tests/smoke/README.md, and the definition ids, authorization steps and cutover runbook in tests/smoke/REGISTERED.md. What remains in the plan is design history - the before/after comparison and decision log - which is preserved in git history and in the #1791 description. Leaving it would be worse than removing it: it is a root-level file describing finished work as outstanding, and a second place where the remaining-cutover steps have to be kept in sync with REGISTERED.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
jamesadevine
added a commit
that referenced
this pull request
Aug 4, 2026
…#1800) * docs(smoke): record the provisioned orchestrator and queue-target ids Cutover provisioning is complete; recording the real ids and marking the runbook steps done. - released orchestrator registered as 2568 - executor-e2e queue target registered as 2569, verified green (build 629513), and E2E_QUEUE_PIPELINE_ID on 2550 repointed 2547 -> 2569 before 2547 is deleted - 2559 repointed at tests/smoke/azure-pipelines-candidate.yml with its six dead COMPILER_SMOKE_*_DEFINITION_ID variables removed in the same edit - SMOKE_LANE_AGENTIC_DEFINITION_ID=2567 on both orchestrators - pool 1453 and the service connections authorized on both new definitions 2568 and 2569 join scheduled_only_definition_ids so the policy audit covers them: both are scheduled or API-queued and must never acquire a CI or PR trigger. Drops the 2559-repoint warning, which has been actioned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd * docs(smoke): record the repository-resource authorization a new lane needs Fourth of five cases in the first live candidate run sat at notStarted forever with Checkpoint.Authorization inProgress and no logs. The multi-repo case checks out ado-aw-e2e-fixture via its repos: block, and a repository resource needs authorizing on the definition that consumes it - a separate grant from the agent pool and from the service connections, on a different resource type. The old per-case definitions had it (2544, 2564, 2565); the new lane inherited nothing, so exactly the one case with an extra checkout hung while the other four went green. This is the one piece of per-case ADO setup the lane model does NOT remove: a case adding a new repos: entry needs its repo authorized on the lane once, before that case can run. Documented in the runbook and, more usefully, in the case-authoring guide next to the other author-responsibility items - with the symptom spelled out, because a build with no logs gives no clue where to look. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd * docs(smoke): record the green candidate run and the released-mode release gate Candidate mode is verified end to end. Build 629522 ran all five candidate cases on the single agentic lane, each on its own per-case ref, and reported Overall: PASSED. Afterwards the mirror holds only ado-aw-smoke-candidate-base and main, confirming per-case ref cleanup. That exercises the whole premise of the lane model: one definition, five refs, five distinct pipelines, no per-case registration. Released mode remains blocked, and not on anything configurable. It compiles with the last RELEASED binary, and v0.48.0 predates the change that made an absent on: emit explicit trigger: none / pr: none, so assertNoTriggers correctly rejects its output with 'staged pipeline must declare trigger: none, got null'. Reproduced locally against the real v0.48.0 asset. Cutting a release clears it; there is no fix to make here, which is why it is recorded as a gate rather than a bug. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd * docs(smoke): delete the transient smoke redesign plan The plan doc was committed at the repository root so the redesign could be reviewed alongside the change, and its own header says to delete it once the cutover completes and the content has landed in tests/smoke/README.md. Both have now happened. Everything durable lives in the suite's own docs: the lane/ref model, modes, trigger handling and fork boundary in tests/smoke/README.md, and the definition ids, authorization steps and cutover runbook in tests/smoke/REGISTERED.md. What remains in the plan is design history - the before/after comparison and decision log - which is preserved in git history and in the #1791 description. Leaving it would be worse than removing it: it is a root-level file describing finished work as outstanding, and a second place where the remaining-cutover steps have to be kept in sync with REGISTERED.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 323b70ff-d193-4c6e-b4c7-9ec6c3dc6ebd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding a smoke used to cost a manual ADO definition registration, secret provisioning, service-connection authorization, fork-hardening, an orchestrator variable, a placeholder commit, a committed lock file and a TypeScript change — ten definitions and five locks in total.
An ADO definition binds
(repo, yamlFilename), but the ref is supplied per queue. So every case now compiles to the same.smoke/pipeline.ymlpath and is pushed to its own per-case ref:Adding a smoke is now a markdown file plus one entry in
tests/smoke/cases.json.What changes
tests/smoke/cases.jsonand loaded bycases.tswith strict fail-closed validation (case ids become git ref segments, so they are allowlisted before anygit pushargument is built).agentic,debug(ADO_AW_DEBUG_GITHUB_TOKEN), andinfra(reserved for AWF and the ado-proxy sidecar, and ready forkind: rawcases).candidate(compiler built from the commit, pinned to this run's pipeline artifact) andreleased(latest release asset, release URLs required). The latter replaces the five committed*.lock.ymlfiles and therecompile-safe-output-fixturesbot workflow that kept them fresh.Notable dependencies that would otherwise have broken silently
executor-e2e'squeue-buildscenario targeted thenoop-targetdefinition, so it gets a dedicated statictests/executor-e2e/queue-target.yml.Why staged pipelines carry
trigger: none/pr: noneNow that
on:is the complete declaration of when a pipeline runs (#1786), stripping it makes the compiler emit an explicittrigger: none/pr: none, so the harness no longer patches those keys into the staged copy — the staged bytes are byte-identical to the pristine lock committed beside them, and the runtimeado-aw check <lock>integrity step still passes.assertNoTriggersremains as the fail-closed guard on the staged bytes before push: ADO reads a missingtrigger:as "CI on every branch", not "no CI", so a compiler that regressed to omitting it would silently let a ref push queue the shared lane on top of the API-queued run.kind: rawsources have no compiler in the loop and so must declare both keys themselves.Validation
cargo test— green (2779 + integration suites)npm run typecheck— greennpx vitest run src/compiler-smoke-e2e— 274 tests green (cases.test.tsandindex.test.tsboth load the realcases.json, so a mis-laned manifest fails locally rather than in ADO)npm run build:compiler-smoke-e2e— bundles cleanbash_lint_testsran with shellcheck presentRemaining work (ADO-side, cannot be done from a checkout)
Tracked in
SMOKE-REDESIGN-PLAN.mdand the runbook intests/smoke/REGISTERED.md: create the base ref, register the three lanes + released orchestrator + queue target, provision secrets, set theSMOKE_LANE_*_DEFINITION_IDvariables, record the ids, run both orchestrators, then disable (not delete) definitions2545–2549and2554–2565.SMOKE-REDESIGN-PLAN.mdis deliberately committed at the root so it can be reviewed alongside the change; it is deleted once the cutover completes.Rebase note
Rebased onto
main, which had landed themulti-reposmoke (#1735) against the old per-case model. It is ported here to the lane model: moved totests/smoke/multi-repo.md, declared as a candidate-onlyagenticcase incases.json, and itsLOCK_FILEassertion repointed attests/smoke/multi-repo.lock.yml. Its retired definition2565is recorded in the retirement table.