Skip to content

ci: run demo + showcase on shiplight-small and wire failure triage - #4

Merged
feng-shiplight merged 5 commits into
mainfrom
ci/wire-failure-triage
Aug 12, 2026
Merged

ci: run demo + showcase on shiplight-small and wire failure triage#4
feng-shiplight merged 5 commits into
mainfrom
ci/wire-failure-triage

Conversation

@feng-shiplight

Copy link
Copy Markdown
Contributor

Runs the showcase suite in CI alongside demo, moves to a smaller runner, and
wires up automated failure triage.

Test workflow (yaml-examples.yml)

  • shiplight-mediumshiplight-small
  • Runs --project demo --project showcase --workers=2. setup still runs
    automatically as a dependency of demo. 24 + 18 = 42 tests.
  • timeout-minutes 30 → 60. The old value was sized for demo alone, and
    playwright.config.ts allows each test 10 minutes.
  • npm installnpm ci, matching what the triage autofix job runs, so
    lockfile drift breaks this workflow first instead of only the repair path.
  • Replaces the full e2e-report artifact with ci-triage/upload-report, which
    drops traces (*.zip) and videos (*.webm). Keeping both would defeat the
    purpose: triage runs gh run download with no name filter, so it would pull
    the heavy copy anyway. Full traces and videos still reach the Shiplight cloud
    report via the retained npx shiplight report step, but they are no longer
    downloadable from the GitHub run.

Triage workflow (ci-failure-triage.yml, new)

Watches YAML Examples Tests and calls ShiplightAI/ci-triage@v1.2. On a red
run an agent diagnoses each failure from the logs and the uploaded report, and
for failures it classifies as fixable spec issues it applies the fix, re-runs
the test, and opens a PR. It never auto-merges.

The Shiplight project is not at the repository root, so this passes
working-directory: yaml-examples. allowed-paths is relative to that and is
scoped to demo showcase templates — what the fix agent is actually permitted
to edit, not helpers/ or the fixture directories. No extra_env: the demo
suite logs into saucedemo with public credentials and showcase hits public
sites, so the autofix re-run needs no app secrets.

Other

  • showcase/17-non-standard-clickable had base_url: http://localhost:3000, a
    leftover from local development that nothing serves in CI. The same page is
    published at https://static.shiplight.ai/testing/interactions/non-standard-clickable.html,
    which every other showcase test already uses.
  • Removes the two Playwright example workflows.
  • Bumps shiplightai to ^0.1.99 and regenerates the lockfile.
  • Gitignores .mcp.json alongside the other agent tooling.

Not yet verified

The shiplight-small runner, whether all 42 tests pass at 2 workers, and
whether 60 minutes is enough — this PR's own run is the first test of all three.

Triage will not run on this PR. workflow_run only reads workflow files from
the default branch, so ci-failure-triage.yml stays dormant until this merges.

🤖 Generated with Claude Code

feng-shiplight and others added 5 commits August 11, 2026 20:07
Regenerates package-lock.json. Keeps `npm ci` in sync, which the E2E
workflow and the ci-triage autofix job both rely on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
base_url was http://localhost:3000, a leftover from local development that
nothing serves in CI. The same page is published at
https://static.shiplight.ai/testing/interactions/non-standard-clickable.html,
which every other showcase test already uses.

The generated .yaml.spec.ts is gitignored and `shiplight test` transpiles
on the fly, so only the source changes here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Test workflow:
- shiplight-medium -> shiplight-small
- run both projects at 2 workers (`setup` still runs as a dependency of
  `demo`); 24 + 18 = 42 tests
- timeout 30 -> 60 min, since 30 was sized for the demo project alone and
  playwright.config.ts allows each test 10 minutes
- npm install -> npm ci, matching the autofix job so lockfile drift breaks
  the test workflow first rather than only the repair path
- replace the full e2e-report artifact with ci-triage/upload-report, which
  drops traces and videos. Keeping both would defeat the point: triage runs
  `gh run download` with no name filter, so it would pull the heavy copy
  anyway. Full traces and videos still reach the Shiplight cloud report.

Triage workflow watches "YAML Examples Tests". The Shiplight project is not
at the repo root, so it passes working-directory: yaml-examples; allowed-paths
is relative to that and is scoped to what the fix agent may actually edit.
No extra_env: the demo suite uses public saucedemo credentials and showcase
hits public sites.

Note that workflow_run only reads workflow files from the default branch, so
triage stays dormant until this lands on main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Generated by `add-mcp` alongside the other agent tooling already ignored
here. Leaving it untracked also keeps it out of the ci-triage write-scope
guard, which only inspects `git diff --name-only`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@feng-shiplight
feng-shiplight merged commit dcaf5f5 into main Aug 12, 2026
1 check failed
feng-shiplight added a commit that referenced this pull request Aug 12, 2026
Fixes demo/19-wait-conditions and scopes each workflow to the tests it runs.

Root cause of the intermittent demo failures: saucedemo's `session-username`
cookie expires ~10 minutes after the `setup` project logs in, and Playwright
drops expired cookies when restoring storage state. Any demo test whose context
is created after that mark starts logged out and lands on the login page.

Evidence — same code, different scheduling:

  PR #4 run: setup ~03:11, demo/19 at 03:21:34 (~10.5 min) -> failed
  main run:  setup 03:34:21, demo/19 at 03:43:22 (~9.0 min) -> passed

The nine green runs before this were demo-only and took 5-7 minutes end to end,
entirely inside the window. Adding showcase pushed the suite to 13 minutes and
the last-scheduled demo tests started crossing the boundary. Runner size and
worker count were incidental; total elapsed time was the variable.

Running demo alone on shiplight-large at 8 workers puts it at roughly 2-3
minutes, which is margin rather than a narrow escape. Not 16 workers: every demo
test shares the single `standard_user` account, and a demo site is entitled to
throttle that many concurrent sessions from one IP — that would swap a session
failure for a rate-limit failure, which is harder to diagnose.

showcase is unauthenticated (public sites), so it has no deadline, but it splits
out anyway so the two suites run in parallel and a change to one does not run
the other.

Path filters now match each suite plus the shared config that affects it.
triage-fixtures is scoped to its own directory only, and deliberately excludes
playwright.config.ts and package.json: it is red by design, so firing it on
shared-config edits would turn unrelated PRs red.

Path lists are repeated rather than shared through a YAML anchor. The Actions
workflow parser does not support anchors, and a workflow that fails to parse
stops running silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
feng-shiplight added a commit that referenced this pull request Aug 12, 2026
Fixes demo/19-wait-conditions and scopes each workflow to the tests it runs.

Root cause of the intermittent demo failures: saucedemo's `session-username`
cookie expires ~10 minutes after the `setup` project logs in, and Playwright
drops expired cookies when restoring storage state. Any demo test whose context
is created after that mark starts logged out and lands on the login page.

Evidence — same code, different scheduling:

  PR #4 run: setup ~03:11, demo/19 at 03:21:34 (~10.5 min) -> failed
  main run:  setup 03:34:21, demo/19 at 03:43:22 (~9.0 min) -> passed

The nine green runs before this were demo-only and took 5-7 minutes end to end,
entirely inside the window. Adding showcase pushed the suite to 13 minutes and
the last-scheduled demo tests started crossing the boundary. Runner size and
worker count were incidental; total elapsed time was the variable.

Running demo alone on shiplight-large at 8 workers puts it at roughly 2-3
minutes, which is margin rather than a narrow escape. Not 16 workers: every demo
test shares the single `standard_user` account, and a demo site is entitled to
throttle that many concurrent sessions from one IP — that would swap a session
failure for a rate-limit failure, which is harder to diagnose.

showcase is unauthenticated (public sites), so it has no deadline, but it splits
out anyway so the two suites run in parallel and a change to one does not run
the other.

Path filters now match each suite plus the shared config that affects it.
triage-fixtures is scoped to its own directory only, and deliberately excludes
playwright.config.ts and package.json: it is red by design, so firing it on
shared-config edits would turn unrelated PRs red.

Path lists are repeated rather than shared through a YAML anchor. The Actions
workflow parser does not support anchors, and a workflow that fails to parse
stops running silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant