Spike: Playwright Test Agents (Planner / Generator / Healer)

[christian-byrne]

TL;DR

Playwright v1.56+ ships three LLM-driven "agents" — Planner, Generator, and Healer — that can scaffold test plans, generate test files, and auto-repair failing tests. We evaluated them against our codebase. Recommendation: wait.

---

Feature Status

Available?	Yes, GA since Playwright v1.56
Our version	1.58.x ✅
npx playwright init-agents	Works in our repo

What Are They?

Three prompt templates + MCP tool definitions that guide an external LLM (VS Code Copilot, Claude Code, etc.) to interact with a live browser via Playwright MCP:

Agent	Purpose
🎭 Planner	Explores the running app → produces a Markdown test plan
🎭 Generator	Transforms the plan into Playwright test files
🎭 Healer	Replays failing tests, inspects UI, patches locators/waits, re-runs

Key insight: These are not standalone CLI tools or CI automation. They require an interactive LLM session to operate.

Why Wait?

1. Not CI-integrated. The agents need a live LLM session — can't run them in a pipeline.
2. Our skipped tests aren't fixable by a healer. They're skipped due to missing backend features or environment constraints, not stale selectors.
3. Custom fixture complexity. Our tests use ComfyPage, ComfyMouse, VueNodeHelpers, and page objects. The generic agent prompts may produce incorrect patches.
4. Community feedback is mixed. Works for simple apps, struggles with complex SPAs (which ComfyUI is).
5. Planner + Generator overlap with our existing writing-playwright-tests skill and established patterns.

When to Revisit

• When Playwright ships a CI-integrated autonomous healer
• When agents support custom fixture awareness / project context injection
• When community feedback improves for complex SPAs

Full Evaluation

Detailed findings are in temp/plans/playwright-agents-evaluation.md (local to the spike branch).

---

Thoughts? Anyone tried these on a different project?

[christian-byrne]

Follow-up: Integration with our Agent Skills

I dug into what npx playwright init-agents --loop=claude actually generates. They are literally Claude Code agent definitions — the exact same .claude/agents/*.md format we already use.

What gets generated

.claude/agents/playwright-test-healer.md
.claude/agents/playwright-test-generator.md
.claude/agents/playwright-test-planner.md
.mcp.json   (adds playwright-test MCP server)
seed.spec.ts
specs/README.md

Each agent uses YAML frontmatter with tools: listing MCP tools like mcp__playwright-test__browser_snapshot, mcp__playwright-test__test_debug, etc. The MCP server is Playwright's built-in npx playwright run-test-mcp-server.

The interesting integration angle

Our writing-playwright-tests skill teaches agents our conventions (fixtures, page objects, tags, anti-patterns). Playwright's agents teach them how to interact with a live browser via MCP tools. These are complementary:

	Our skill	Playwright agents
Format	.claude/skills/*.md	.claude/agents/*.md
Knowledge	Project conventions, fixture APIs	Generic Playwright MCP tool usage
Browser access	None	Full (snapshot, click, evaluate)
Project awareness	Deep (ComfyPage, VueNodeHelpers)	None

What a combined approach could look like

A merged healer agent that:

1. Has Playwright MCP tools for live browser interaction (from their agent)
2. Knows our custom fixtures and patterns (from our skill)
3. Understands our test conventions (tags, asset loading, nextFrame patterns)

This would be taking their .claude/agents/playwright-test-healer.md, injecting our project-specific context from writing-playwright-tests/SKILL.md, and pointing the seed test at our fixture setup.

Practical concern

The Playwright MCP server runs a real browser — needs a display or headless Chromium. Works locally but adds complexity for automated/CI use.

To experiment

npx playwright init-agents --loop=claude --config playwright.config.ts

Then customize the generated .claude/agents/playwright-test-healer.md to reference our fixtures and conventions.

[christian-byrne]

Experiment: Codegen → Convention Transform Pipeline

Tried combining Playwright codegen with an agent/skill to auto-rewrite into our conventions.

Pipeline

npx playwright codegen http://localhost:5173 -o raw.spec.ts
          ↓
   Agent/Skill transforms raw output
          ↓
   Convention-compliant test using ComfyPage fixtures

What codegen produces (raw)

import { test, expect } from '@playwright/test';

test('test', async ({ page }) => {
  await page.goto('http://localhost:5173/');
  await page.locator('canvas').dblclick({ position: { x: 500, y: 400 } });
  await page.getByPlaceholder('Search Nodes...').fill('KSampler');
  await page.getByPlaceholder('Search Nodes...').press('Enter');
  await page.getByRole('button', { name: 'Queue' }).click();
  await page.waitForTimeout(1000);  // anti-pattern
});

What a transform skill would produce

import {
  comfyPageFixture as test,
  comfyExpect as expect
} from '../fixtures/ComfyPage'

test.describe('Queue workflow with KSampler', { tag: ['@canvas'] }, () => {
  test.afterEach(async ({ comfyPage }) => {
    await comfyPage.canvasOps.resetView()
  })

  test('should add KSampler and queue', async ({ comfyPage }) => {
    await comfyPage.canvas.dblclick({ position: { x: 500, y: 400 } })
    await comfyPage.searchBox.fillAndSelectFirstNode('KSampler')
    await comfyPage.nextFrame()
    await comfyPage.menu.queueButton.click()
  })
})

Key transforms needed

Raw codegen	Convention
import { test } from '@playwright/test'	comfyPageFixture as test from fixtures
page.goto(url)	Removed — fixture handles navigation
page.locator('canvas')	comfyPage.canvas
page.waitForTimeout(n)	comfyPage.nextFrame() or retrying assertion
page.getByPlaceholder('Search...')	comfyPage.searchBox.*
No tags, no cleanup	{ tag: ['@canvas'] }, resetView() in afterEach

Verdict

Useful as onboarding tool, not a productivity win for experienced contributors.

• ✅ Captures correct selectors and interaction sequences quickly
• ✅ Good for new contributors who don't know the fixture API yet
• ❌ Canvas interactions record pixel coordinates (fragile, env-dependent)
• ❌ Can't record ComfyPage helper usage — only raw Playwright APIs
• ❌ Transform gap is large enough that writing directly with our skill is often faster

The sweet spot: new contributor records a flow with codegen → pastes output → asks agent to rewrite using our conventions. This could be documented in the writing-playwright-tests skill as an alternative starting point.

Experiment files in temp/codegen-experiment/.