Skip to content

coeus-ventures/epic-test

BranchesTags

Repository files navigation

epic-test

A collection of AI-powered testing utilities for modern applications.

Libraries

This package includes three testing libraries:

Library Purpose
spec-test Specification-driven browser testing with AI-powered automation
b-test LLM-powered browser assertions with HTML snapshot diffing
db-test Database state management for deterministic testing

Installation

npm install epic-test
# or
pnpm add epic-test
# or
bun add epic-test

Peer Dependencies

Depending on which libraries you use, you may need:

# For spec-test and b-test (browser testing)
npm install playwright

# For db-test (database testing)
npm install @libsql/client

Quick Start

spec-test - Specification-Driven Browser Testing

Write behavior specifications in Epic format markdown and execute them as browser tests:

Note: spec-test only supports the Epic specification format. If you use a different format, you'll need to write an adapter (see spec-test documentation below).

import { SpecTestRunner } from 'epic-test';

const runner = new SpecTestRunner({
  baseUrl: 'http://localhost:3000',
  headless: true,
});

// Run tests from a markdown spec file (must be in Epic format)
const result = await runner.runFromFile('./specs/login.md');

if (result.success) {
  console.log('All tests passed!');
} else {
  console.log('Tests failed:', result.failedAt?.context.error);
}

await runner.close();

Epic format spec file (login.md):

# Login

## Examples

### Login with valid credentials

#### Steps
* Act: User enters "user@example.com" in email field
* Act: User enters "password123" in password field
* Act: User clicks Login button
* Check: URL contains /dashboard
* Check: Welcome message is displayed

b-test - LLM-Powered Browser Assertions

Use natural language to assert page conditions:

import { chromium } from 'playwright';
import { Tester } from 'epic-test/b-test';

const browser = await chromium.launch();
const page = await browser.newPage();
const tester = new Tester(page);

await page.goto('https://example.com');

// Take snapshots and assert with natural language
await tester.snapshot();
await page.click('button.submit');
await tester.snapshot();

const hasSuccessMessage = await tester.assert('Success message is displayed');

db-test - Database State Management

Set up and verify database state for deterministic tests:

import { PreDB, PostDB } from 'epic-test/db-test';
import { db } from './db';
import * as schema from './schema';

// Setup: Set initial database state
await PreDB(db, schema, {
  users: [{ id: 1, name: 'Alice', email: 'alice@example.com' }]
});

// Act: Run the code being tested
await createUser({ name: 'Bob', email: 'bob@example.com' });

// Assert: Verify final database state
await PostDB(db, schema, {
  users: [
    { id: 1, name: 'Alice', email: 'alice@example.com' },
    { id: 2, name: 'Bob', email: 'bob@example.com' }
  ]
});

spec-test

A specification-driven testing library that parses behavior specifications and executes them as browser tests using AI-powered automation.

⚠️ IMPORTANT: Specification Format Requirement

spec-test only accepts specifications in the Epic format (see below). If your specifications use a different format (Gherkin, Cucumber, custom markdown, etc.), you MUST write an adapter to transform them into the Epic format before using spec-test.

The built-in parser (parseSpecFile) will fail silently or incorrectly parse specifications that don't follow the exact Epic format structure.

See Adapting Other Formats for guidance on writing adapters.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        spec-test Library                        │
├─────────────────────────────────────────────────────────────────┤
│  parseSpecFile  →  SpecTestRunner  →  ExampleResult             │
│                                                                  │
│              ┌──────────────┴──────────────┐                    │
│              │                             │                    │
│       ┌──────▼──────┐           ┌──────────▼────────┐           │
│       │  Stagehand  │           │      B-Test       │           │
│       │ (Act steps) │           │  (Check steps)    │           │
│       │ AI browser  │           │  LLM assertions   │           │
│       └─────────────┘           └───────────────────┘           │
└─────────────────────────────────────────────────────────────────┘

Epic Specification Format (Required)

IMPORTANT: spec-test requires specifications to follow the Epic format exactly. If you have specifications in a different format, you must either:

  1. Convert your specifications to match the Epic format
  2. Write an adapter that transforms your format to Epic format before passing to SpecTestRunner

The parser expects this exact structure:

# Behavior Name

Description of the behavior.

Directory: `app/features/my-feature/`

## Examples

### Example scenario name

#### Steps
* Act: User performs some action
* Act: User fills in "value" in field name
* Check: URL contains /expected-path
* Check: Success message is displayed

Format Requirements

Element Required Format Notes
Behavior name # Title (H1) First H1 in file
Examples section ## Examples (H2) Exact text, case-sensitive
Example name ### Name (H3) Under Examples section
Steps marker #### Steps (H4) Required - steps are only parsed after this marker
Act step * Act: instruction Must be after #### Steps
Check step * Check: instruction Must be after #### Steps

Adapting Other Formats

If your specifications use ANY format other than Epic format, you MUST write an adapter.

Common scenarios requiring an adapter:

  • Different heading levels (e.g., #### Examples instead of ## Examples)
  • Gherkin/Cucumber format (Given, When, Then)
  • Custom markdown structures
  • Different keywords (e.g., Action: instead of Act:)
  • Specifications stored in YAML, JSON, or other formats

Adapter Pattern

Create a function that transforms your format into the TestableSpec interface:

import { SpecTestRunner } from 'epic-test';
import type { TestableSpec, SpecExample, SpecStep } from 'epic-test';

/**
 * Adapter for Gherkin/Cucumber format
 */
function parseGherkinToEpic(gherkinContent: string): TestableSpec {
  // Parse your Gherkin content
  const feature = parseGherkin(gherkinContent);

  return {
    name: feature.name,
    examples: feature.scenarios.map(scenario => ({
      name: scenario.name,
      steps: scenario.steps.map(step => {
        // Map Given/When to Act, Then to Check
        const type = (step.keyword === 'Then') ? 'check' : 'act';
        return {
          type,
          instruction: step.text,
          ...(type === 'check' ? { checkType: 'semantic' } : {})
        } as SpecStep;
      })
    }))
  };
}

// Use with SpecTestRunner
const runner = new SpecTestRunner({ baseUrl: 'http://localhost:3000' });
const spec = parseGherkinToEpic(gherkinContent);
const result = await runner.runFromSpec(spec);
console.log(result.success ? 'PASS' : 'FAIL', spec.name);

TypeScript Interfaces

Your adapter must return objects matching these interfaces:

interface TestableSpec {
  name: string;
  directory?: string;
  examples: SpecExample[];
}

interface SpecExample {
  name: string;
  steps: SpecStep[];
}

type SpecStep =
  | { type: 'act'; instruction: string }
  | { type: 'check'; instruction: string; checkType?: 'deterministic' | 'semantic' };

Do not attempt to modify the built-in parser. Write an adapter instead.

Step Types

Act Steps - User actions executed with Stagehand AI:

* Act: User clicks the Login button
* Act: User enters "user@example.com" in email field
* Act: User selects "Admin" from the role dropdown

Check Steps - Verifications (deterministic or semantic):

# Deterministic (fast, no AI):
* Check: URL contains /dashboard
* Check: Page title is "Dashboard"

# Semantic (LLM-powered):
* Check: Success notification is displayed
* Check: Error message shows "Invalid credentials"

Configuration

interface SpecTestConfig {
  baseUrl: string;              // Required - app URL
  headless?: boolean;           // Default: true
  aiModel?: LanguageModelV2;    // Override AI model
  browserbaseApiKey?: string;   // Cloud browser execution
  cacheDir?: string;            // Enable action caching
  cachePerSpec?: boolean;       // Per-spec cache directories
}

Caching

Enable caching for faster subsequent runs:

const runner = new SpecTestRunner({
  baseUrl: 'http://localhost:3000',
  cacheDir: './cache/e2e-tests',
});

// First run: ~30s per action (LLM inference)
// Subsequent runs: ~3s per action (cached)

b-test

LLM-powered browser testing utilities with HTML snapshot capture and natural language assertions.

Features

  • HTML Snapshots: Capture full page HTML with timestamps
  • LLM Assertions: Use natural language to assert conditions
  • Snapshot Comparison: Generate structured diffs between states
  • Polling: Wait for conditions with intelligent polling

API

const tester = new Tester(page);

// Capture snapshots
await tester.snapshot();

// Assert conditions
const result = await tester.assert('Login form is visible');

// Compare page states
const diff = await tester.diff();

// Wait for conditions
await tester.waitFor('Loading spinner is gone', 10000);

Error Codes

Code Description
NO_PAGE No page provided
NO_SNAPSHOT No snapshot for assertion
SNAPSHOT_FAILED Failed to capture page
ASSERTION_FAILED LLM service error
DIFF_FAILED Comparison error
WAIT_TIMEOUT Condition timeout

db-test

Database testing utilities for deterministic state management with Drizzle ORM.

PreDB - Setup Initial State

await PreDB(db, schema, {
  users: [
    { id: 1, name: 'Alice' },
    { id: 2, name: 'Bob' }
  ],
  posts: [
    { id: 1, userId: 1, title: 'First Post' }
  ]
});

Options:

{
  wipe?: boolean;           // Delete existing rows (default: true)
  resetSequences?: boolean; // Reset auto-increment (default: true)
  only?: string[];          // Target specific tables
}

PostDB - Verify Final State

await PostDB(db, schema, {
  users: [
    { id: 1, name: 'Alice' },
    { id: 2, name: 'Bob' },
    { id: 3, name: 'Charlie' }  // New user from test
  ]
});

Options:

{
  only?: string[];         // Target specific tables
  allowExtraRows?: boolean; // Allow extra DB rows (default: false)
  loose?: boolean;         // Loose comparison (default: false)
}

Load from JSON Files

await PreDBFromFile(db, schema, './fixtures/initial-state.json');
// ... run test ...
await PostDBFromFile(db, schema, './fixtures/expected-state.json');

Environment Variables

Variable Library Description
OPENAI_API_KEY spec-test, b-test Required for AI-powered features
BROWSERBASE_API_KEY spec-test Cloud browser execution

Development

# Install dependencies
npm install

# Build
npm run build

# Run tests
npm test

# Run integration tests
npm run test:integration

License

MIT

About

No description, website, or topics provided.

Resources

Readme

License

MIT license
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages