Arena

Framework-agnostic multi-agent scenario simulation and evaluation.

Arena is a measurement layer for multi-agent systems. Define a scenario suite once, run it under any orchestration strategy, and get comparable quality / safety / autonomy / cost / latency metrics — with statistical significance tests and CI regression gates. It is deliberately framework-agnostic: no LangGraph, no Vercel AI SDK, no SaaS platform required. Bring your own agents through a thin adapter, or use the built-in scripted adapters for fully reproducible runs.

Same suite, many orchestration strategies, comparable numbers.

Why Arena?

The existing JS/TS evaluation tools all lock you into one framework or platform:

@langwatch/scenario — bound to LangWatch SaaS and the LangGraph ecosystem.
reddial — tied to the Vercel AI SDK.
@perceo/perceo — coupled to Temporal + Supabase infrastructure.
@agentesting/agentest — built around a specific agent runtime.

None of them are a framework-agnostic measurement layer. None of them let you run the same scenario suite under two orchestration strategies and compare the results head-to-head. And none of them measure autonomy — the human-in-the-loop ratio that matters for agent systems that collaborate with people.

Arena fills that gap:

Framework-agnostic — AgentAdapter is the only contract between your agents and the engine.
Deterministic — seeded PRNG and a message-bus scheduler mean the same input always produces the same event stream.
Comparable — run one suite under strategy A and strategy B; get a side-by-side table with paired t-test and Wilcoxon signed-rank significance.
Autonomy-aware — measures the human intervention rate and autonomous decision share out of the box.
CI-ready — RegressionGate thresholds with JSON report and non-zero exit codes, plus a ready-made GitHub Action.

Features

Area	What you get
Scenario	Roles (agents + human roles), task goal, environment state, initial messages, max turns, timeout, per-scenario seed.
ScenarioSuite	Write suites in YAML, JSON or TypeScript. Seeded for reproducibility.
SimulationRunner	Message-bus-driven deterministic scheduler: turn-based execution, full event log, timeout and deadlock detection, scripted customer turns.
AgentAdapter	Framework-agnostic contract. Adapters for OpenAI Agents SDK, LangGraph.js, Vercel AI SDK, plus `createScriptedAdapter` / `createFunctionAdapter` / `createHumanAdapter`.
MetricsEngine	Five dimensions: `quality`, `safety`, `autonomy`, `cost`, `latency`. Autonomy = autonomous decisions / (autonomous + human interventions).
ComparisonReport	Same suite under two strategies, side-by-side table with Δ, Δ%, p-values (paired t-test + Wilcoxon) and significance verdicts.
RegressionGate	Threshold assertions over metrics, JSON-serializable results, non-zero exit code for CI. Bundled GitHub Action.
CLI	`arena run`, `arena compare`, `arena report` (bin name `arena`).
Packaging	ESM + CJS, full type declarations, zero runtime dependencies (optional adapter deps are peer + optional).

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        Scenario Suite (YAML / TS)                   │
│              scenarios: roles, goal, env, messages, turns           │
└────────────────────────────────┬────────────────────────────────────┘
                                 │ loads
┌────────────────────────────────▼────────────────────────────────────┐
│                        SimulationRunner                             │
│   message bus ──► turn scheduler ──► timeouts ──► deadlock detect   │
│   emits SimulationEvent[] ──► ScenarioRunResult[]                   │
└────────────────────────────────┬────────────────────────────────────┘
        ┌────────────────────────┴────────────────────────┐
        ▼                                                  ▼
┌───────────────┐                                  ┌───────────────┐
│ Strategy A    │                                  │ Strategy B    │
│ AgentAdapter  │                                  │ AgentAdapter  │
│ (scripted /   │                                  │ (scripted /   │
│  OpenAI SDK / │                                  │  OpenAI SDK / │
│  LangGraph /  │                                  │  LangGraph /  │
│  AI SDK /     │                                  │  AI SDK /     │
│  custom)      │                                  │  custom)      │
└───────────────┘                                  └───────────────┘
        └────────────────────────┬────────────────────────┘
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                        MetricsEngine                                 │
│   quality │ safety │ autonomy │ cost │ latency                       │
│   (rubric / PII / human-ratio / token price / latency quantiles)     │
└────────────────────────────────┬────────────────────────────────────┘
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│   ComparisonReport      RegressionGate          CLI + GitHub Action │
│   Δ, Δ%, p-values       min/max thresholds      arena run/compare/   │
│   t-test + Wilcoxon     exit 0/1 for CI         report               │
└─────────────────────────────────────────────────────────────────────┘

Installation

npm install @noahisarider/arena

The core runtime has zero dependencies. Optional adapters pull their framework lazily — install them only if you use them:

npm install agents              # OpenAI Agents SDK adapter
npm install ai @ai-sdk/openai   # Vercel AI SDK adapter
npm install @langchain/langgraph

Quick start

1. Write a scenario suite

suite.yaml:

id: customer-service
seed: 7
scenarios:
  - id: billing-refund
    goal: Resolve the duplicate charge and issue a refund.
    maxTurns: 8
    timeoutMs: 30000
    roles:
      - id: tier1
        name: Tier 1 Support
        systemPrompt: First line of support for billing inquiries.
      - id: manager
        name: Manager
        isHuman: true
        systemPrompt: Authorized to approve refunds.
    initialMessages:
      - id: m1
        role: user
        from: customer
        to: tier1
        content: "I was charged twice for my subscription."

2. Define a strategy

strategy-autonomous.ts:

import { createScriptedAdapter } from '@noahisarider/arena';

export const strategy = {
  name: 'autonomous',
  adapters: {
    tier1: createScriptedAdapter('tier1', [
      { match: /billing|charge|double/i, reply: 'Refund issued.', done: true },
    ]),
    manager: createScriptedAdapter('manager', [
      { match: /.*/, reply: 'Standing by.' },
    ]),
  },
  runner: { turnOrder: 'conversational' },
  metrics: { pricing: { inputPerMTok: 0.5, outputPerMTok: 1.5 } },
};

3. Run it

arena run suite.yaml --strategy strategy-autonomous.ts

Scenario       | Status    | Quality | Safety | Autonomy | Interv. | Cost($)   | Latency(ms)
---------------|-----------|---------|--------|----------|---------|-----------|------------
billing-refund | completed |   1.000 |  1.000 |    1.000 |   0.000 |   0.00000 |            0

Suite "customer-service" (seed 7) — 1 scenario
  quality   mean 1.000 (p95 1.000)
  safety    mean 1.000 (p95 1.000)
  autonomy  mean 1.000 (intervention rate 0.000)
  cost      mean $0.00000 total (p95 $0.00000)
  latency   mean 0ms per run (p95 0ms)

4. Compare two strategies

arena compare suite.yaml \
  --a strategy-human-first.ts \
  --b strategy-autonomous.ts --md

Metric	human-first	autonomous	Δ	Δ %	p (t)	p (w)	Better
Quality	1.0000	1.0000	0.0000	+0.0%	1.0000	1.0000	TIE
Safety	1.0000	1.0000	0.0000	+0.0%	1.0000	1.0000	TIE
Autonomy	0.8333	1.0000	+0.1667	+20.0%	0.4226	1.0000	TIE
Cost	0.0000	0.0000	0.0000	+0.0%	1.0000	1.0000	TIE
Latency	0.3333	0.3333	0.0000	+0.0%	1.0000	1.0000	TIE

5. Enforce it in CI

gate.yaml:

thresholds:
  aggregates.quality.mean:
    min: 0.8
  aggregates.autonomy.mean:
    min: 0.4

arena run suite.yaml --strategy strategy.ts --gate gate.yaml && echo "gate passed"

Failing gates exit with code 1. See the bundled GitHub Action below for a turnkey CI step.

CLI

arena run <suite> [options]                    Run a suite with one strategy.
arena compare <suite> --a <a.ts> --b <b.ts>    Run one suite under two strategies.
arena report <results.json> [options]          Render a saved results file.

Option	Meaning
`--strategy <file>`	Strategy config (TS/JS/YAML/JSON) with adapters.
`--a <file>` / `--b <file>`	Strategy configs A and B for `compare`.
`--out <file>`	Write full report data as JSON.
`--json`	Print report data as JSON to stdout.
`--md`	Print a Markdown table (`compare`, `report`).
`--gate <file>`	Regression gate thresholds (YAML/JSON/TS). Exit 1 on failure.
`--seed <n>`	Override the simulation seed.
`--alpha <n>`	Significance level for compare (default 0.05).
`--metric <k,...>`	Limit comparison to `quality,safety,autonomy,cost,latency`.

Scenario DSL

Suite

Field	Type	Description
`id`	`string`	Required. Unique suite id.
`name` / `description`	`string`	Optional metadata.
`seed`	`number`	Default seed for scenarios without their own.
`scenarios`	`Scenario[]`	Required, non-empty.

Scenario

Field	Type	Description
`id`	`string`	Required. Unique within the suite.
`goal`	`string`	Required. What the agents should accomplish.
`roles`	`Role[]`	Required, non-empty.
`environment`	`object`	Initial shared state, merged into every adapter's view.
`initialMessages`	`Message[]`	Messages present at simulation start.
`maxTurns`	`number`	Hard cap on turns before the run stops.
`timeoutMs`	`number`	Wall-clock timeout in ms.
`seed`	`number`	Per-scenario seed (falls back to suite seed).
`metadata`	`object`	Free-form. Used by default scorers (see below).

Role

Field	Type	Description
`id`	`string`	Required. Role/agent id.
`name` / `description`	`string`	Optional.
`systemPrompt`	`string`	Optional system instructions.
`isHuman`	`boolean`	When `true`, the role is played by a human and drives the autonomy metric.
`capabilities`	`string[]`	Optional, for rubric scorers and adapters.

Message

Field	Type	Description
`id`	`string`	Stable id.
`role`	`'system' \| 'user' \| 'assistant' \| 'tool' \| 'human'`	Message role.
`from`	`AgentId \| 'system' \| 'customer'`	Sender.
`to`	`AgentId \| 'customer' \| 'human' \| null`	Recipient; `null` broadcasts.
`content`	`string`	Message body.
`metadata`	`object`	Optional payload.

Scenario metadata hooks

The default scorers read two metadata keys:

qualityChecks — rubric items evaluated against all message text:

metadata:
  qualityChecks:
    - label: issued-refund
      match: "refund"        # substring, RegExp, or function
      weight: 2

blockedPhrases — safety violations (alongside default PII patterns for email, phone, SSN, credit card):
```
metadata:
  blockedPhrases:
    - "share your password"
    - "credit card number"
```

customerScript — deterministic scripted customer turns injected by turn:

metadata:
  customerScript:
    - atTurn: 2
      content: "I still get 'invalid credentials'."
      to: tier2

Turn scheduling

turnOrder: 'conversational' (default) — follow output.to handoffs, fall back to round-robin.
turnOrder: 'round-robin' — cycle through roles in declaration order.

Handoff to 'human' requires a role with isHuman: true; otherwise the run ends as a deadlock.

Metrics

Autonomy

Defined as the share of decisions made autonomously:

autonomy = autonomous decisions / (autonomous decisions + human interventions)
interventionRate = human interventions / (autonomous decisions + human interventions)

A decision is any assistant message; an intervention is any human message. A strategy that escalates everything to a human manager scores low on autonomy; a strategy that resolves everything itself scores 1.0.

Other dimensions

Dimension	Meaning
`quality`	Weighted rubric score. Defaults to completion + agent-reply + no-error checks when no `qualityChecks` metadata is present.
`safety`	Fraction of agent/human messages that avoid PII and `blockedPhrases`.
`cost`	USD from token usage, either adapter-reported or computed via `metrics.pricing`.
`latency`	Total run time plus per-turn mean and p50/p90/p95/p99 from `agent-turn-end` events.

Significance tests

ComparisonReport runs a paired t-test and a Wilcoxon signed-rank test between the two strategies' per-scenario metric vectors, and marks a metric as A, B or TIE at the configured alpha.

Programmatic API

import {
  SimulationRunner,
  MetricsEngine,
  RegressionGate,
  createScriptedAdapter,
} from '@noahisarider/arena';

const runner = new SimulationRunner({ turnOrder: 'conversational' });
const run = await runner.runSuite(suite, adapters, { seed: 7 });

const engine = new MetricsEngine({ pricing: { inputPerMTok: 0.5, outputPerMTok: 1.5 } });
const metrics = engine.metricsForSuite(run, suite);

const gate = RegressionGate.fromYaml(`
thresholds:
  aggregates.quality.mean: { min: 0.8 }
`);
const result = gate.evaluate(metrics);
console.log(result.passed); // boolean

Adapters

import { createOpenAIAgentsAdapter } from '@noahisarider/arena/adapters';
import { createLangGraphAdapter } from '@noahisarider/arena/adapters';
import { createAiSdkAdapter } from '@noahisarider/arena/adapters';

createOpenAIAgentsAdapter({ role, agent }) — wraps an OpenAI Agents SDK Agent (lazy agents import).
createLangGraphAdapter({ role, graph | invoke, formatResult? }) — wraps a compiled LangGraph graph or an invoke function.
createAiSdkAdapter({ role, model, system?, tools? }) — wraps a Vercel AI SDK language model (lazy ai import).
createScriptedAdapter(role, rules, opts) — deterministic rule-driven adapter for reproducible runs.
createFunctionAdapter(role, fn) — wrap any function as an adapter.
createHumanAdapter(role, reply?) — a human role that emits human-intervention events.
defineAdapter(adapter) — identity helper with full type checking for hand-written adapters.

GitHub Action

The repo ships a ready-made composite action for CI gating.

steps:
  - uses: noahisarider/arena/action@v1
    with:
      suite: demo/customer-service/suite.yaml
      strategy: demo/customer-service/strategies/human-first.ts
      gate: demo/customer-service/gate.yaml

For a two-strategy comparison:

steps:
  - uses: noahisarider/arena/action@v1
    with:
      suite: demo/customer-service/suite.yaml
      a: demo/customer-service/strategies/human-first.ts
      b: demo/customer-service/strategies/autonomous.ts
      gate: demo/customer-service/gate.yaml

Inputs: suite (required), strategy (for run), a + b (for compare), gate (required), version (npm package version, default latest).

Comparison with existing tools

	@noahisarider/arena	@langwatch/scenario	reddial	@perceo/perceo	@agentesting/agentest
Framework-agnostic	Yes — `AgentAdapter`	No — LangGraph ecosystem	No — Vercel AI SDK	No — Temporal + Supabase	No — specific agent runtime
No SaaS/platform dependency	Yes	No — LangWatch SaaS	Yes	No — infra required	Yes
Same suite across strategies	Yes — `arena compare`	Partial	No	No	No
Autonomy metric	Yes — intervention rate / decision share	No	No	No	No
Deterministic & seeded	Yes — seeded PRNG + message bus	Partial	Partial	Partial	Partial
Regression gate + CI	Yes — `RegressionGate` + GitHub Action	Partial	No	No	No
ESM + CJS + types	Yes	—	—	—	—
Zero runtime deps	Yes	No	No	No	No

Demo

A complete, runnable example lives in demo/customer-service: a customer service center with 3 roles (Tier 1, Tier 2, human Manager) and an A/B comparison between a human-first escalation strategy and a fully autonomous strategy.

arena run demo/customer-service/suite.yaml \
  --strategy demo/customer-service/strategies/human-first.ts

arena compare demo/customer-service/suite.yaml \
  --a demo/customer-service/strategies/human-first.ts \
  --b demo/customer-service/strategies/autonomous.ts --md

Development

npm install
npm run check     # lint + typecheck + test + build
npm run coverage  # test with coverage thresholds (90%+)
npm run attw      # verify ESM/CJS/type exports

CI runs lint, typecheck, coverage, build and export verification on every push/PR. Publishing is automatic when a v* tag is pushed (with npm provenance).

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
.github/workflows		.github/workflows
action		action
demo/customer-service		demo/customer-service
docs		docs
src		src
test		test
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
eslint.config.js		eslint.config.js
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json
tsup.config.ts		tsup.config.ts
vitest.config.ts		vitest.config.ts

Navigation Menu

Uh oh!

Repository files navigation

Arena

Why Arena?

Features

Architecture

Installation

Quick start

1. Write a scenario suite

2. Define a strategy

3. Run it

4. Compare two strategies

5. Enforce it in CI

CLI

Scenario DSL

Suite

Scenario

Role

Message

Scenario metadata hooks

Turn scheduling

Metrics

Autonomy

Other dimensions

Significance tests

Programmatic API

Adapters

GitHub Action

Comparison with existing tools

Demo

Development

License

About

Releases

Packages

Contributors

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Arena

Why Arena?

Features

Architecture

Installation

Quick start

1. Write a scenario suite

2. Define a strategy

3. Run it

4. Compare two strategies

5. Enforce it in CI

CLI

Scenario DSL

Suite

Scenario

Role

Message

Scenario metadata hooks

Turn scheduling

Metrics

Autonomy

Other dimensions

Significance tests

Programmatic API

Adapters

GitHub Action

Comparison with existing tools

Demo

Development

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages