ci: fix demo/19 session expiry, split per-project workflows, add triage fixtures + Slack - #5
Merged
Merged
Conversation
Two fixtures that exercise the CI Failure Triage pipeline's two branches, built on the existing static.shiplight.ai failure-analysis pages: - 01-spec-drift (Type B): renamed-elements.html renamed "Generate Report" to "Create Report". The app is correct and the spec is stale, so triage should return spec_issue/fixable and open an autofix PR. - 02-app-regression (Type A): broken-assertion.html fires its click handler and logs to the click history, but never writes the new value to the DOM. The test is correct and the app is broken, so triage should return app_regression/not-fixable and open NO PR. Both assert with `js:` predicates on stable ids rather than natural-language clicks. A natural-language click would be self-healed onto the renamed button and the drift fixture would pass, leaving nothing to triage. Verified locally: both fail, and for the intended reasons. The drift fixture reports that the buttons now read "Create Report"; the regression fixture reports that the click registered but the display stayed at "Count: 0". Registered as a separate `triage-fixtures` project so the main suite never runs them, and `npm test` now names demo + showcase explicitly so a local run matches CI instead of picking up two guaranteed failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Runs the failing fixtures nightly and on demand, kept separate from "YAML Examples Tests" so the main suite stays a real signal — if the fixtures lived there, main would be permanently red and a genuine regression would be indistinguishable from them. No push/pull_request trigger, since every red run spends an agent run in triage. Skips `npx shiplight report`: these fail by design, and publishing them would drag down the pass-rate and flaky-test analytics the `cloud` subcommand reports. Triage reads the GitHub artifact, so it loses nothing. Adds triage-fixtures to allowed-paths so autofix can repair the drift fixture — that PR is the demonstration and should be left open, which also stops ci-triage opening a duplicate on the next run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test step used `continue-on-error: true` plus a trailing `exit 1` step to force the job red. That makes GitHub record the test step's *conclusion* as success, so `gh run view --log-failed` — the command ci-triage uses to gather evidence — returns only the five-line `exit 1` step. Observed on run 31560400823: triage received a 509-byte failed-logs.txt containing nothing but `Process completed with exit code 1`. ci-triage's `if [ ! -s ]` fallback to the full log did not rescue it, because 509 bytes is not empty. Screenshots and report-data.json still arrived via the artifact, but the agent lost the error text and stack entirely. Let the test step fail outright instead. The uploads still run on always()/!cancelled(), and the trailing exit-1 step is now redundant. The cloud upload gets continue-on-error so a publishing hiccup cannot mask a green suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ci-triage deliberately posts nowhere itself — it publishes artifacts and lets the consumer route them with its own credentials. This adds that caller-owned job. triage.md is already authored in Slack mrkdwn (single-asterisk bold, no headers, TL;DR inside a code fence, under 1500 chars), so it is posted verbatim rather than reformatted. Details worth keeping: - SLACK_RELEASE_BOT_TOKEN lives in this job, never in extra_env, so the autofix agent cannot read it. - continue-on-error at the job level: Slack being down must not mask a triage result. - Posts a short notice when triage.md is absent. Silence would read as "no failures", and a missing model credential fails every engine while still exiting green — exactly what happened on run 31560977596. - Gates on the `ok` field, since chat.postMessage returns HTTP 200 even when it refuses the message. - Payload built with jq so quotes and newlines in the report cannot break the JSON. - Channel comes from the SLACK_TRIAGE_CHANNEL repo variable, so changing it needs no workflow edit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SLACK_RELEASE_CHANNEL_ID is an existing org-level Actions *variable* (C0ALYPN2NAF), so it is read via vars, not secrets. Replaces the placeholder SLACK_TRIAGE_CHANNEL I had invented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes demo/19-wait-conditions and scopes each workflow to the tests it runs. Root cause of the intermittent demo failures: saucedemo's `session-username` cookie expires ~10 minutes after the `setup` project logs in, and Playwright drops expired cookies when restoring storage state. Any demo test whose context is created after that mark starts logged out and lands on the login page. Evidence — same code, different scheduling: PR #4 run: setup ~03:11, demo/19 at 03:21:34 (~10.5 min) -> failed main run: setup 03:34:21, demo/19 at 03:43:22 (~9.0 min) -> passed The nine green runs before this were demo-only and took 5-7 minutes end to end, entirely inside the window. Adding showcase pushed the suite to 13 minutes and the last-scheduled demo tests started crossing the boundary. Runner size and worker count were incidental; total elapsed time was the variable. Running demo alone on shiplight-large at 8 workers puts it at roughly 2-3 minutes, which is margin rather than a narrow escape. Not 16 workers: every demo test shares the single `standard_user` account, and a demo site is entitled to throttle that many concurrent sessions from one IP — that would swap a session failure for a rate-limit failure, which is harder to diagnose. showcase is unauthenticated (public sites), so it has no deadline, but it splits out anyway so the two suites run in parallel and a change to one does not run the other. Path filters now match each suite plus the shared config that affects it. triage-fixtures is scoped to its own directory only, and deliberately excludes playwright.config.ts and package.json: it is red by design, so firing it on shared-config edits would turn unrelated PRs red. Path lists are repeated rather than shared through a YAML anchor. The Actions workflow parser does not support anchors, and a workflow that fails to parse stops running silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
feng-shiplight
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the intermittent
demo/19-wait-conditionsfailure, splits the suites intoper-project workflows, adds two fixtures that exercise the CI Failure Triage
pipeline, and routes triage results to Slack.
demo/19 — root cause
saucedemo's
session-usernamecookie expires ~10 minutes after thesetupproject logs in, and Playwright drops expired cookies when restoring storage
state. Any demo test whose context is created after that starts logged out and
lands on the login page.
Measured to the moment each test's context was created:
The boundary falls exactly where a 10-minute expiry predicts. The nine green runs
before this were demo-only at 5–7 minutes, entirely inside the window; adding
showcase pushed the suite to 13 minutes and the last-scheduled demo tests began
crossing it. Runner size and worker count were incidental — total elapsed time
was the variable.
Fixed structurally rather than by extending the cookie:
demonow runs alone onshiplight-largeat 8 workers, finishing in 2–3 minutes. Not 16 workers — everydemo test shares the single
standard_useraccount, and a demo site is entitledto throttle that many concurrent sessions from one IP.
Per-project workflows
yaml-examples.ymlis replaced by three path-scoped workflows, so a change toone suite no longer runs the others:
demo/**+ shared configshiplight-large, 8 workersshowcase/**+ shared configshiplight-large, 8 workerstriage-fixtures/**only, plus nightly cronshiplight-smallTriage watches all three. Path lists are repeated rather than shared via a YAML
anchor: the Actions parser does not support anchors, and a workflow that fails to
parse stops running silently.
Triage fixtures
Built on the
failure-analysispages already published at static.shiplight.ai —a purpose-made corpus nothing in this repo was using.
01-spec-driftrenamed-elements.htmlspec_issue,fixable: true, opens an autofix PR02-app-regressionbroken-assertion.htmlapp_regression,fixable: false, no PRAn autofix PR on the second would be a pipeline failure — the agent rewriting a
correct test to accommodate a broken app, which the triage skill forbids.
Both assert with
js:predicates on stable ids. A natural-language click getsself-healed onto the renamed button, the drift fixture passes, and nothing is
left to triage. Verified locally: both fail, for the intended reasons.
They live in their own project and workflow so
mainstays a real signal, andskip
npx shiplight reportso guaranteed failures do not drag down the cloudpass-rate analytics.
Triage Fixtures is red on this PR by design. That is the fixture working.
Slack + evidence fix
triage.mdis already authored in Slack mrkdwn, so it is posted verbatim by acaller-owned job. The bot token stays out of
extra_envwhere the autofix agentcould read it; the job posts a notice when no report exists, since silence would
read as "no failures"; and it gates on the
okfield, becausechat.postMessagereturns HTTP 200 even when it refuses.Also removes
continue-on-errorfrom the test steps. It made GitHub record thestep's conclusion as success, so
gh run view --log-failed— how ci-triagegathers evidence — returned only the trailing
exit 1. On run 31560400823 triagereceived a 509-byte log containing nothing but
Process completed with exit code 1.Not covered
showcase/05-ai-extractionfailed once onmainand passed on two other runs.That is a separate flake in an AI extraction against Wikipedia, unrelated to the
session issue and not investigated here.
🤖 Generated with Claude Code