Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ coverage
# nyc test coverage
.nyc_output

# twd-cli recording artifacts
twd-artifacts

# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
.grunt

Expand Down Expand Up @@ -137,3 +140,6 @@ dist
# Vite logs files
vite.config.js.timestamp-*
vite.config.ts.timestamp-*

# Superpowers SDD scratch workspace
.superpowers/
51 changes: 37 additions & 14 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,30 +16,53 @@ twd-cli is a CLI tool for running TWD (Test While Developing) browser-based test

## Architecture

The codebase is a small ESM-only Node.js CLI with two core source files:
The codebase is a small ESM-only Node.js CLI. `bin/twd-cli.js` and `src/index.js` are the spine; every other file in `src/` is a single-purpose helper with a matching `tests/*.test.js`.

**`bin/twd-cli.js`**CLI entry point. Parses `process.argv` for the `run` command, calls `runTests()`, and exits with code 0 (pass) or 1 (failure).
**`bin/twd-cli.js`**: CLI entry point. Parses `process.argv` for the `run` command via `src/parseArgs.js`, calls `runTests()`, and exits with code 0 (pass) or 1 (failure).

**`src/config.js`** — `loadConfig()` reads `twd.config.json` from `process.cwd()`, merges it with defaults (url, timeout, coverage, headless, puppeteerArgs, retryCount, protocolTimeout, maxFailures, chunkSize), and returns the merged config. Falls back to defaults if the file is missing or unparseable.
**`src/parseArgs.js`**: `parseRunArgs(argv)` returns `{ testFilters, record }`. Supports `--test` (repeatable substring filter) and the recording flags `--record`, `--record-dir`, `--record-speed`. Each accepts both `--flag value` and `--flag=value`. The returned `record` object is passed to `runTests()` as `recordOverrides` and wins over the config file.

**`src/config.js`**: `loadConfig()` reads `twd.config.json` from `process.cwd()`, merges it with defaults (url, timeout, coverage, coverageDir, nycOutputDir, headless, puppeteerArgs, retryCount, protocolTimeout, maxFailures, chunkSize, record), and returns the merged config. Falls back to defaults if the file is missing or unparseable.

`protocolTimeout` (default `300000`, 5 min) is passed to `puppeteer.launch` and bounds each chunk's CDP call. `maxFailures` (default `10`) stops the run after that many cumulative test failures; set to `0` to disable. `chunkSize` (default `10`) controls how many tests run per browser call.

**`src/index.js`** — `runTests()` is the main orchestrator:
1. Loads config via `loadConfig()`
2. Launches Puppeteer with configured headless mode and args
3. Navigates to the configured URL (default: `http://localhost:5173`)
4. Waits for `#twd-sidebar-root` selector (indicates app + TWD are ready)
5. Enumerates all registered test handlers and computes pre-order execution order
6. Runs tests in ordered chunks via `runByIds(chunkIds)`, with chunk size controlled by config; accumulates results in Node so the run can stop after `maxFailures` failures and partial results survive a timeout or crash
7. Prints a relay-style summary block (`formatRunComplete` in `src/testSummary.js`) as the last output: passed/failed/skipped counts, duration, failed tests with `suite > test` paths and error messages, retried tests, and "Not run" count if stopped early. Known infrastructure errors (dev server down, sidebar missing, protocol timeout, Chrome launch failure) get actionable diagnostics from `src/diagnostics.js`.
8. Optionally collects `window.__coverage__` and writes to `.nyc_output/out.json` (skipped whenever the run has failures, including an early bail)
9. Returns boolean `hasFailures`
`record` (`DEFAULT_RECORD`) is the only **nested** config key, so the merge goes two levels deep: `record` merges over `DEFAULT_RECORD`, and `record.viewport` merges over the default viewport. A flat spread would wipe sibling defaults. Recording is off by default and never runs unless explicitly requested.

**`src/index.js`**: `runTests({ testFilters, recordOverrides })` is the main orchestrator:
1. Loads config via `loadConfig()`, then overlays `recordOverrides` onto a **copy** of `config.record` (never mutate it, it can be the shared `DEFAULT_RECORD` object)
2. Probes for ffmpeg via `assertFfmpegAvailable()` when recording, before anything expensive, so a missing binary fails fast instead of after launch and navigation
3. Launches Puppeteer with configured headless mode and args
4. `page.setViewport(record.viewport)` when recording (a normal run keeps Puppeteer's implicit 800x600)
5. Navigates to the configured URL (default: `http://localhost:5173`)
6. Waits for `#twd-sidebar-root` selector (indicates app + TWD are ready)
7. Injects the framing stylesheet when recording, hiding the sidebar and resetting the html margin twd-js sets inline
8. Enumerates all registered test handlers and computes pre-order execution order
9. Resolves `--test` filters into the id list to run
10. Starts the screencast when recording. This happens **after** filter resolution, because `page.screencast()` fixes the output path up front and the filename is derived from the tests that survived the filter (`src/recordFilename.js`)
11. Runs tests in ordered chunks via `runByIds(chunkIds)`, with chunk size controlled by config; accumulates results in Node so the run can stop after `maxFailures` failures and partial results survive a timeout or crash
12. Stops the recorder, then reports the artifact, but only after checking the file has bytes on disk. A resolved `stop()` is not evidence of a usable video (see the recording notes below)
13. Prints a relay-style summary block (`formatRunComplete` in `src/testSummary.js`) as the last output: passed/failed/skipped counts, duration, failed tests with `suite > test` paths and error messages, retried tests, and "Not run" count if stopped early. Known infrastructure errors (dev server down, sidebar missing, protocol timeout, Chrome launch failure) get actionable diagnostics from `src/diagnostics.js`.
14. Optionally collects `window.__coverage__` and writes to `.nyc_output/out.json` (skipped whenever the run has failures, including an early bail)
15. Returns boolean `hasFailures`

**`src/recorder.js`** holds the screencast wrapper: `assertFfmpegAvailable()` (pre-flight `spawnSync(ffmpegPath, ['-version'])` probe), `FRAMING_CSS` / `applyRecordingFraming()`, and `startRecording()` which creates the output dir and calls `page.screencast()`.

### Recording gotchas

These are load-bearing and easy to undo by accident:

- **`stopRecorder()` must run before `browser.close()` on both the success and `catch` paths, and at most once.** If the browser closes first, ffmpeg is orphaned and the file is truncated. The closure nulls `recorder` before awaiting, so a throw between the success-path stop and `browser.close()` cannot double-stop.
- **`record.ffmpegPath` has to reach `page.screencast()`, not just the probe.** Puppeteer spawns its own ffmpeg and defaults to a bare `ffmpeg` on PATH, so forwarding only to the probe produces a passing pre-flight followed by a raw `spawnSync ffmpeg ENOENT`.
- **A 0-byte output is a normal outcome, not a crash.** Puppeteer's frame pipeline buffers with `bufferCount(2, 1)` and Chrome only emits screencast frames on a compositor update, so a suite that never repaints (or an empty run) finishes cleanly with an empty file. `recordedFileSize()` gates the success line on real bytes.
- **`viewport.deviceScaleFactor` does not affect the video.** Puppeteer measures the recording with `deviceScaleFactor` forced to 0, so the emulated factor never reaches the encoder, but it *is* live on the page during the run. Default is `1`. Puppeteer's actual output-size knob is `scale`, which this feature does not expose.

**`test-example-app/`** — A React demo app with TWD tests integrated, used for manual testing/demonstration. Not part of the published package or test suite.

## Testing

Tests are in `tests/` and use vitest. The test suite mocks `fs` to test config loading and mocks Puppeteer to test the run flow. Coverage is configured for `src/**/*.js` only.
Tests are in `tests/` and use vitest, one file per `src/` module. The suite mocks `fs` to test config loading and mocks Puppeteer to test the run flow. Coverage is configured for `src/**/*.js` only.

No test may require a real ffmpeg binary or a real browser: `node:child_process` and `page.screencast` are always mocked. Note that `vi.mock('fs')` auto-mocks `fs.statSync` to return `undefined`, so anything reading a `Stats` has to tolerate that.

## Key Dependencies

Expand Down
83 changes: 83 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,40 @@ Notes:
- Code coverage collection is skipped while a `--test` filter is active, since a
filtered run is a partial (debug) run.

### Recording a run

Record a test run to a video file, for a PR attachment, a docs clip, or a demo:

```bash
# Record one flow
npx twd-cli run --record --test "checkout flow"

# Record at half speed, into a custom directory
npx twd-cli run --record --record-speed 0.5 --record-dir ./clips
```

Requires ffmpeg. See [Requirements](#requirements).

The run produces a single video containing every matched test, back to back, in
declaration order. Note that `--test` matches a substring of the full
`"suite > test"` path, so one filter can match several tests.

The file is named after what is in it: a single recorded test gets a slug of its
full path (`login-shows-error-on-bad-password.mp4`), and anything else gets
`run.<ext>`, where `<ext>` comes from `format` (`mp4` by default, or `webm`/`gif`
if you set that). Re-running overwrites the file.

The TWD sidebar is hidden during recording so the frame is just your app.

Chrome only emits video frames when the page repaints, so a suite that only
asserts and never changes anything on screen can finish with an empty file. When
that happens the run says so rather than reporting a video you cannot play.

**A recorded run is a demo artifact, not a substitute for a CI run.** Recording
sets its own viewport (1280x720 by default, versus the 800x600 a normal run
uses) and reflows the app to full width, so a recorded run can pass or fail
differently. Run CI normally and record separately.

### Configuration

Create a `twd.config.json` file in your project root:
Expand Down Expand Up @@ -86,9 +120,55 @@ Create a `twd.config.json` file in your project root:
| `chunkSize` | number | `10` | How many tests run per browser call. Smaller values make the failure limit and timeouts more granular (less work lost if one chunk hangs); larger values reduce overhead. `0` runs everything in one call |
| `contracts` | array | — | OpenAPI contract validation specs (see [Contract Validation](#contract-validation)) |
| `contractReportPath` | string | — | Path to write a markdown report for CI/PR integration |
| `record` | object | see below | Video recording settings (see [Recording a run](#recording-a-run)) |

**Partial Results on Timeout or Crash:** Tests run in chunks (controlled by `chunkSize`), so on a `protocolTimeout` or unexpected crash mid-run, results from completed chunks are printed instead of being lost entirely.

#### Recording Options

All keys live under `record` in `twd.config.json`.

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `enabled` | boolean | `false` | Turn recording on. Equivalent to passing `--record` |
| `dir` | string | `"./twd-artifacts"` | Directory the video is written to |
| `filename` | string \| null | `null` | Explicit output filename. When `null`, the name is derived from the recorded tests. A known extension (`.mp4`, `.webm`, `.gif`) is respected, otherwise `format` supplies it |
| `format` | string | `"mp4"` | `"mp4"`, `"webm"` or `"gif"`. All three are encoded natively, no conversion step |
| `viewport` | object | `{ "width": 1280, "height": 720, "deviceScaleFactor": 1 }` | Applied only when recording. `width` and `height` set the video dimensions. `deviceScaleFactor` does **not** change the output resolution (Puppeteer measures the recording in CSS pixels), it only changes the page environment under test: raising it makes `srcset` and `image-set` pick 2x assets and sends dpr-branching code down a different path |
| `fps` | number | `30` | Capture frame rate |
| `speed` | number | `1` | Playback speed, e.g. `0.5` for half speed. This is a **uniform stretch of the whole timeline**, not per-command pacing: it slows the fast parts and the already-slow parts equally and cannot hold on a just-clicked element |
| `preRoll` | number | `0` | Milliseconds to hold the opening state before the first test runs. Purely cosmetic |
| `postRoll` | number | `500` | Milliseconds to hold the final state after the last test. **Not cosmetic:** without it the last thing your test did never appears in the video at all. See [Why the ending needs a hold](#why-the-ending-needs-a-hold). Set `0` only if you do not care about the ending |
| `hideSidebar` | boolean | `true` | Hide the TWD sidebar during capture so the frame is just your app |
| `ffmpegPath` | string | `"ffmpeg"` | Path to the ffmpeg binary if it is not on your `PATH` |

#### Why the ending needs a hold

Chrome only sends a video frame when the page repaints, and Puppeteer holds each
frame until the *next* one arrives, because the next frame's timestamp is what
says how long to display the current one. The newest frame is therefore never
written, and stopping the recorder pads the tail by repeating the one before it.

A settled page produces no more repaints, so simply waiting does not help.
Measured against real Chrome: stopping immediately ended two states early, and a
400ms plain wait still ended one state early.

`postRoll` fixes this by briefly repainting the whole viewport with an invisible
overlay after the last test, which forces the real final frame through and then
holds it. This is why it defaults to on.

#### Making the video longer

`postRoll` fixes the *ending*, not the *pace*. Tests run in milliseconds, so a
two-test run is around a second of video. Two things help today:

- `record.speed` (or `--record-speed 0.5`) stretches the whole timeline
- `record.preRoll` and `record.postRoll` stop it starting and ending abruptly

Both are blunt. Per-command pacing, where the video dwells on each click and
assertion, has to happen inside `twd-js` because that is where the command loop
lives, and it is not part of this feature yet.

## How It Works

**Important**: Puppeteer is **not** used as a testing framework here. It simply provides a headless browser to load your application — the same way a user would open Chrome. Once the page loads, all test execution happens inside the real browser context through the [TWD runner](https://brikev.github.io/twd/). Your tests interact with real DOM, real components, and real browser APIs — Puppeteer just opens the door and gets out of the way.
Expand Down Expand Up @@ -302,3 +382,6 @@ Failed validations are included in a collapsible details section with a link to

- Node.js >= 20.19.x
- A running development server with TWD tests
- ffmpeg, only for `--record`. Install with `brew install ffmpeg` (macOS),
`sudo apt-get install ffmpeg` (Linux), or `winget install ffmpeg` (Windows).
Set `record.ffmpegPath` if it is not on your `PATH`.
13 changes: 11 additions & 2 deletions bin/twd-cli.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ const command = process.argv[2];

if (command === 'run') {
try {
const { testFilters } = parseRunArgs(process.argv.slice(3));
const hasFailures = await runTests({ testFilters });
const { testFilters, record } = parseRunArgs(process.argv.slice(3));
const hasFailures = await runTests({ testFilters, recordOverrides: record });
process.exit(hasFailures ? 1 : 0);
} catch (error) {
if (!error?.reported) {
Expand All @@ -25,12 +25,21 @@ Usage:
npx twd-cli run --test "<name>" Run only tests whose "suite > test" path
contains <name> (case-insensitive).
Repeatable; multiple --test values are OR'd.
npx twd-cli run --record Record the run to a video file

Examples:
npx twd-cli run --test "shows error"
npx twd-cli run --test "Login" --test "Signup"

Options:
--test "<name>" Filter tests by "suite > test" path (repeatable, OR'd)
--record Record the run to a video file (requires ffmpeg)
--record-dir <path> Output directory (default ./twd-artifacts)
--record-speed <n> Playback speed, e.g. 0.5 for half speed

--record-dir and --record-speed only set values. Recording still has to be
turned on with --record or "record": { "enabled": true } in twd.config.json.

Create a twd.config.json file in your project root to customize settings.
`);
process.exit(command ? 1 : 0);
Expand Down
Loading
Loading