Skip to content

feat(record): per-test retries + scene-failure hint + slow-scene warning#13

Merged
shreyaskarnik merged 1 commit into
mainfrom
feat/scene-retries-v1
May 5, 2026
Merged

feat(record): per-test retries + scene-failure hint + slow-scene warning#13
shreyaskarnik merged 1 commit into
mainfrom
feat/scene-retries-v1

Conversation

@shreyaskarnik
Copy link
Copy Markdown
Owner

@shreyaskarnik shreyaskarnik commented May 5, 2026

Summary

First half of the long-demo flake mitigation roadmap. Pure additive — no behavior change unless callers opt in.

  • video.retries: N config + --retries N CLI flag — embeds Playwright's built-in retries in the generated playwright.record.config.mjs so a transient failure (browser hiccup, race in showOverlay, intermittent network) gets a free retry without manual intervention.
  • Failing scene hint — when the recording subprocess fails, the error message appends the last scene name from .scene-progress.jsonl so users can map the failure to a scene without reading the Playwright stack trace.
  • Slow-scene warning — after a successful pipeline, scenes whose duration is more than 1.75× the median are surfaced as a console warning. Useful intermittent-flake signal — a scene that drifts from 6s → 18s on retries is the most likely culprit.

Scope

This is v1 of the long-demo flake roadmap. v2 (scene-isolated recording with state checkpoints + --resume-from) is deliberately deferred — it's ~5× the scope and the design will be better informed by what failure modes v1's diagnostics surface in production. v1 alone catches a lot of transient failures via the Playwright retry plumbing.

Test plan

  • npm test — 639 tests pass (3 new tests covering retries embedding: default 0, explicit value, clamp/floor of fractional values)
  • Local npx argo pipeline ci-smoke --retries 1 runs cleanly; slow-scene warning fires correctly on the trailing-wait scene
  • No behavior change for callers that don't set retries (default remains 0, current behavior)
  • CI matrix passes (chromium / firefox / webkit) — should be untouched, but verifying

…ing (v1)

First half of the long-demo flake mitigation roadmap. Pure additive — no
behavior change unless callers opt in.

* `video.retries: N` config + `--retries N` CLI flag — embeds Playwright's
  built-in `retries` in the generated playwright.record.config.mjs so a
  transient failure (browser hiccup, race in showOverlay) gets a free
  retry without manual intervention.
* When the recording subprocess fails, append the last scene name from
  `.scene-progress.jsonl` to the error so users can map the failure to a
  scene without reading the Playwright stack trace.
* After a successful pipeline, surface scenes whose duration is more than
  1.75× the median as a console warning. Useful intermittent-flake signal
  — a scene that drifts from 6s → 18s on retries is the most likely
  culprit.

Doesn't fix the "10-minute demo flakes on scene 9, re-records all 10"
case — that's v2 (scene-isolated recording + checkpoints), tracked
separately. v1 ships first because it's universal and zero-configuration
for existing demos.

3 new tests covering the retries embedding (default 0, explicit value,
clamp/floor of fractional values).
@shreyaskarnik shreyaskarnik merged commit b501b98 into main May 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant