ci: run demo + showcase on shiplight-small and wire failure triage - #4
Merged
Conversation
Regenerates package-lock.json. Keeps `npm ci` in sync, which the E2E workflow and the ci-triage autofix job both rely on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
base_url was http://localhost:3000, a leftover from local development that nothing serves in CI. The same page is published at https://static.shiplight.ai/testing/interactions/non-standard-clickable.html, which every other showcase test already uses. The generated .yaml.spec.ts is gitignored and `shiplight test` transpiles on the fly, so only the source changes here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Test workflow: - shiplight-medium -> shiplight-small - run both projects at 2 workers (`setup` still runs as a dependency of `demo`); 24 + 18 = 42 tests - timeout 30 -> 60 min, since 30 was sized for the demo project alone and playwright.config.ts allows each test 10 minutes - npm install -> npm ci, matching the autofix job so lockfile drift breaks the test workflow first rather than only the repair path - replace the full e2e-report artifact with ci-triage/upload-report, which drops traces and videos. Keeping both would defeat the point: triage runs `gh run download` with no name filter, so it would pull the heavy copy anyway. Full traces and videos still reach the Shiplight cloud report. Triage workflow watches "YAML Examples Tests". The Shiplight project is not at the repo root, so it passes working-directory: yaml-examples; allowed-paths is relative to that and is scoped to what the fix agent may actually edit. No extra_env: the demo suite uses public saucedemo credentials and showcase hits public sites. Note that workflow_run only reads workflow files from the default branch, so triage stays dormant until this lands on main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Generated by `add-mcp` alongside the other agent tooling already ignored here. Leaving it untracked also keeps it out of the ci-triage write-scope guard, which only inspects `git diff --name-only`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
feng-shiplight
added a commit
that referenced
this pull request
Aug 12, 2026
Fixes demo/19-wait-conditions and scopes each workflow to the tests it runs. Root cause of the intermittent demo failures: saucedemo's `session-username` cookie expires ~10 minutes after the `setup` project logs in, and Playwright drops expired cookies when restoring storage state. Any demo test whose context is created after that mark starts logged out and lands on the login page. Evidence — same code, different scheduling: PR #4 run: setup ~03:11, demo/19 at 03:21:34 (~10.5 min) -> failed main run: setup 03:34:21, demo/19 at 03:43:22 (~9.0 min) -> passed The nine green runs before this were demo-only and took 5-7 minutes end to end, entirely inside the window. Adding showcase pushed the suite to 13 minutes and the last-scheduled demo tests started crossing the boundary. Runner size and worker count were incidental; total elapsed time was the variable. Running demo alone on shiplight-large at 8 workers puts it at roughly 2-3 minutes, which is margin rather than a narrow escape. Not 16 workers: every demo test shares the single `standard_user` account, and a demo site is entitled to throttle that many concurrent sessions from one IP — that would swap a session failure for a rate-limit failure, which is harder to diagnose. showcase is unauthenticated (public sites), so it has no deadline, but it splits out anyway so the two suites run in parallel and a change to one does not run the other. Path filters now match each suite plus the shared config that affects it. triage-fixtures is scoped to its own directory only, and deliberately excludes playwright.config.ts and package.json: it is red by design, so firing it on shared-config edits would turn unrelated PRs red. Path lists are repeated rather than shared through a YAML anchor. The Actions workflow parser does not support anchors, and a workflow that fails to parse stops running silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
feng-shiplight
added a commit
that referenced
this pull request
Aug 12, 2026
Fixes demo/19-wait-conditions and scopes each workflow to the tests it runs. Root cause of the intermittent demo failures: saucedemo's `session-username` cookie expires ~10 minutes after the `setup` project logs in, and Playwright drops expired cookies when restoring storage state. Any demo test whose context is created after that mark starts logged out and lands on the login page. Evidence — same code, different scheduling: PR #4 run: setup ~03:11, demo/19 at 03:21:34 (~10.5 min) -> failed main run: setup 03:34:21, demo/19 at 03:43:22 (~9.0 min) -> passed The nine green runs before this were demo-only and took 5-7 minutes end to end, entirely inside the window. Adding showcase pushed the suite to 13 minutes and the last-scheduled demo tests started crossing the boundary. Runner size and worker count were incidental; total elapsed time was the variable. Running demo alone on shiplight-large at 8 workers puts it at roughly 2-3 minutes, which is margin rather than a narrow escape. Not 16 workers: every demo test shares the single `standard_user` account, and a demo site is entitled to throttle that many concurrent sessions from one IP — that would swap a session failure for a rate-limit failure, which is harder to diagnose. showcase is unauthenticated (public sites), so it has no deadline, but it splits out anyway so the two suites run in parallel and a change to one does not run the other. Path filters now match each suite plus the shared config that affects it. triage-fixtures is scoped to its own directory only, and deliberately excludes playwright.config.ts and package.json: it is red by design, so firing it on shared-config edits would turn unrelated PRs red. Path lists are repeated rather than shared through a YAML anchor. The Actions workflow parser does not support anchors, and a workflow that fails to parse stops running silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs the showcase suite in CI alongside demo, moves to a smaller runner, and
wires up automated failure triage.
Test workflow (
yaml-examples.yml)shiplight-medium→shiplight-small--project demo --project showcase --workers=2.setupstill runsautomatically as a dependency of
demo. 24 + 18 = 42 tests.timeout-minutes30 → 60. The old value was sized for demo alone, andplaywright.config.tsallows each test 10 minutes.npm install→npm ci, matching what the triage autofix job runs, solockfile drift breaks this workflow first instead of only the repair path.
e2e-reportartifact withci-triage/upload-report, whichdrops traces (
*.zip) and videos (*.webm). Keeping both would defeat thepurpose: triage runs
gh run downloadwith no name filter, so it would pullthe heavy copy anyway. Full traces and videos still reach the Shiplight cloud
report via the retained
npx shiplight reportstep, but they are no longerdownloadable from the GitHub run.
Triage workflow (
ci-failure-triage.yml, new)Watches
YAML Examples Testsand callsShiplightAI/ci-triage@v1.2. On a redrun an agent diagnoses each failure from the logs and the uploaded report, and
for failures it classifies as fixable spec issues it applies the fix, re-runs
the test, and opens a PR. It never auto-merges.
The Shiplight project is not at the repository root, so this passes
working-directory: yaml-examples.allowed-pathsis relative to that and isscoped to
demo showcase templates— what the fix agent is actually permittedto edit, not
helpers/or the fixture directories. Noextra_env: the demosuite logs into saucedemo with public credentials and showcase hits public
sites, so the autofix re-run needs no app secrets.
Other
showcase/17-non-standard-clickablehadbase_url: http://localhost:3000, aleftover from local development that nothing serves in CI. The same page is
published at
https://static.shiplight.ai/testing/interactions/non-standard-clickable.html,which every other showcase test already uses.
shiplightaito^0.1.99and regenerates the lockfile..mcp.jsonalongside the other agent tooling.Not yet verified
The
shiplight-smallrunner, whether all 42 tests pass at 2 workers, andwhether 60 minutes is enough — this PR's own run is the first test of all three.
Triage will not run on this PR.
workflow_runonly reads workflow files fromthe default branch, so
ci-failure-triage.ymlstays dormant until this merges.🤖 Generated with Claude Code