LoopForge

A local-first AI agent observability and evaluation workbench. LoopForge takes the loop at the heart of every modern AI agent — observe → reason → act → verify — and makes it a first-class, observable object instead of a black box: every model turn, thinking block, tool call, and result streams live to a trace dashboard as a structured event. The same loop runs interchangeably against four model providers (a scripted mock, a local Ollama model, the logged-in Claude Code CLI, or the Claude API) and three pluggable environments (a sandboxed coding project, a Sokoban game arena, and a Playwright-driven web-QA browser hunting a planted bug in a seeded shop), and a deterministic eval harness runs whole task suites as scored runs and ranks providers — down to the specific model — on a live leaderboard.

The agent loop is the heart of every modern AI agent. Most frameworks hide it behind a final answer. LoopForge inverts that: the loop emits a typed TraceEvent for every meaningful moment, and everything else — the server, the dashboard, the scorer — is just a consumer of that one event stream.

Trust boundary: LoopForge is for trusted local/development-team use. Do not expose it as an unauthenticated service or accept anonymous coding tasks. See ADR 0001 and Security and isolation.

A coding run streaming live — the model's thinking, each tool call, and the file diff it produced, side by side.

Sokoban arena — the agent solves a puzzle on a live, animated board.

Web QA — a Playwright agent drives a real browser and finds a planted checkout bug.

Eval harness — a suite run as scored pass/fail with live aggregate metrics.

_{📖 Documentation · Architecture · Design notes & interview prep}

What it is

• A provider-agnostic agent loop (@loopforge/core) — AgentLoop runs observe → reason → act → verify, owns the conversation and tool execution, and emits a TraceEvent for every step (iteration start, model request/response with thinking, tool started/finished, env state, run finished).
• A run + eval orchestrator (@loopforge/server) — an Express + WebSocket server (port 8787) that creates runs, records each event log, broadcasts every event live, and drives batch evals.
• A live trace dashboard (@loopforge/web) — a React + Vite app that folds the event stream into per-iteration cards, an animated Sokoban board, live file diffs for coding runs, a browser screenshot panel for web-QA runs, and an eval leaderboard.
• A deterministic evaluation harness — task suites run as real scored runs, judged pass/fail purely from their recorded events, aggregated into pass-rate and cost metrics.
• A portfolio-grade codebase — TypeScript throughout, npm workspaces, 53 node:test tests, and an adversarial audit-hardening pass that fixed 11 confirmed bugs, each locked down with a regression test.

Feature highlights

• Four interchangeable providers, one loop. Swap the model backing the exact same loop with no code changes:

  • Run a real frontier model with no API key — claude-cli drives your locally-installed, logged-in Claude Code CLI as a single-turn model (its own tools disabled, our loop runs the tools).
  • Run a real model for free — ollama drives a local model (e.g. llama3) through a ReAct JSON adapter — no key, no per-token cost.
  • Zero-setup demo — mock replays deterministic scripts whose tool calls execute for real, so the demo genuinely runs without any key.
  • Live API — anthropic calls the Claude API directly with adaptive extended thinking.

• Three pluggable environments. A whole new domain plugs in behind one RunEnvironment interface without touching the loop: a sandboxed coding project (four path-confined tools + a planted bug to fix, with live file diffs in the dashboard), a sokoban game arena (an in-memory engine with a live, animated board), and a browser web-QA arena — a Playwright-driven headless Chromium the agent uses to test the seeded LoopMart demo shop, whose checkout carries a planted bug (POST /order always 500s) for the agent to find and report.

• Per-provider model selection. Every run and eval takes an optional model override (the forms show each provider's default as the placeholder), so the same suite can pit specific models — not just providers — against each other.

• Deterministic eval harness with a live leaderboard. Run a suite × N repeats as concurrency-capped real runs; a deterministic, event-based scorer marks each pass/fail (with an anti-cheat so echoing a test file can't false-pass); pass rate, mean iterations, tokens, and duration aggregate live, and a leaderboard keyed by (provider, model) ranks backends — and specific models — head-to-head.

  
  _{The demo suite run through the Ollama provider at two model sizes — no API key, no cost. The 8B model solves the coding task but not the spatial puzzle; the 4.5B model clears neither. That honest capability gradient is exactly what a deterministic harness exists to surface.}

• Live trace dashboard. Every run streams over WebSocket — thinking, tool inputs/outputs, token usage, and status — and any eval result row drills into the identical trace + board UI, because an eval run is a real run under the hood.

Architecture

Three npm-workspace packages, layered so dependencies only ever point inward toward the pure core.

flowchart LR
  subgraph browser["Browser"]
    web["@loopforge/web<br/>React + Vite<br/>live trace dashboard"]
  end

  subgraph node["Node process — port 8787"]
    server["@loopforge/server<br/>Express + WebSocket<br/>RunManager · EvalManager"]
    core["@loopforge/core<br/>AgentLoop<br/>observe → reason → act → verify"]
    providers["Providers<br/>mock · ollama · claude-cli · anthropic"]
    envs["Environments<br/>coding · sokoban · browser"]
  end

  web -->|"REST /api"| server
  server -.->|"WebSocket /ws — TraceEvents"| web
  server -->|"creates & runs"| core
  core -->|"complete(request)"| providers
  core -->|"executes tools"| envs

  classDef pkg fill:#1e293b,stroke:#475569,color:#e2e8f0;
  class web,server,core,providers,envs pkg;

Loading

loopforge/
├── packages/core      Agent-loop engine: loop, providers, tools, trace events
├── packages/server    Run + eval orchestration: REST + WebSocket streaming
├── packages/web       Live trace dashboard (React + Vite)
├── sandbox/           Seeded coding project the agent operates on
└── docs/              Full documentation set (see below)

For the deep dive — the loop lifecycle, the TraceEvent model, and the extensibility seams — see docs/ARCHITECTURE.md.

Providers

The same agent loop runs against all four providers; the two local ones (Ollama, Claude CLI) share the packages/core/src/providers/react.ts adapter.

Provider	What it is	API key?	Cost	Native tool-calling?
mock	Scripted, deterministic steps — but its tool calls execute for real	No	None	n/a (script emits calls)
ollama	A local model via Ollama (llama3:latest default)	No	None (local compute)	No — uses the ReAct adapter
claude-cli	The logged-in Claude Code CLI (claude -p) driven as a single-turn model	No — uses the CLI's account	Real per-iteration account usage	No — CLI tools disabled on purpose
anthropic	The Claude API via @anthropic-ai/sdk (claude-opus-4-8 default)	Yes — ANTHROPIC_API_KEY	Paid API tokens	Yes — the only native-tools provider

Every provider works in both the Runs view and the Eval harness. Running the demo suite under a real local model produces an honest capability profile — e.g. llama3 typically solves the coding bug-fix but not Sokoban, landing the eval near 50%, right next to mock's designed 2-pass/2-fail. Full contract, per-provider internals, and the ReAct adapter: docs/PROVIDERS.md.

Quickstart

No API key needed for the default mock provider.

npm install

# Terminal 1 — REST API + WebSocket trace stream (http://localhost:8787)
npm run dev:server

# Terminal 2 — React dashboard (http://localhost:5173)
npm run dev:web

Open http://localhost:5173 and start a mock run — no setup required. The scripted agent finds and fixes a real bug in a seeded calculator project: its tool calls actually execute — it lists files, reads the failing test, runs node test.js red, patches calc.js (which ships with add returning a - b), and re-runs the test green. Then try a sokoban run to watch the agent push boxes on a live board, or open the Eval tab and run the demo suite (deliberately 2 pass / 2 fail under mock) to see the scorer and leaderboard in action.

To try the Web QA (browser) environment — the agent drives a real headless Chromium against the seeded LoopMart shop (which the server hosts on :8788) and hunts its planted checkout bug — install the browser binary once: npx playwright install chromium (~95 MB). This step is optional: the server and the other two environments run fine without it (browser tools just error with that install hint). With it installed, the web-qa eval suite (deliberately 1 pass / 1 fail under mock) shows the scorer separating a QA run that finds the bug from one that merely browses.

Vite proxies /api and /ws to :8787, so the dashboard talks to the server transparently. To try a real model with no API key, install Ollama (ollama pull llama3) and pick Local (Ollama · llama3), or use the logged-in Claude CLI (local account) provider. Full setup — including each provider's prerequisites and the .env for the live API — is in docs/DEVELOPMENT.md.

Documentation

Start with the documentation index, or jump straight in:

• Architecture — the observable loop, monorepo layering, the TraceEvent model, and the extensibility seams.
• Providers — the four model backends and the shared ReAct adapter.
• Environments — the RunEnvironment seam, the coding sandbox, and the Sokoban arena.
• Eval Harness — scored suites, the deterministic scorer, and the live leaderboard.
• Development — build, run, and test locally; per-provider setup; the roadmap.

Roadmap

Phase	Scope	Status
1	Core agent loop engine + live trace dashboard + coding tools + mock mode	✅ Done
2	Sokoban game-arena environment via pluggable environments	✅ Done
3	Eval harness — task suites, parallel scored runs, pass-rate aggregation, leaderboard, per-run sandbox isolation	✅ Done
—	Local providers: Ollama (no-key local model) + Claude CLI (local account)	✅ Done
—	Audit hardening + node:test suites (11 bugs fixed, 28 tests)	✅ Done
4	Per-provider model selection (leaderboard compares specific models head-to-head) + coding file-diff view	✅ Done
5	Autonomous web-QA agent environment (Playwright) vs the seeded LoopMart shop	✅ Done
—	Multi-file coding tasks	⏳ Next
—	More eval suites	⏳ Next
—	CI	⏳ Next

Testing

npm run typecheck               # tsc --noEmit across all workspaces
npm test -w @loopforge/core     # 17 tests
npm test -w @loopforge/server   # 36 tests

53 deterministic node:test tests (run through tsx, no network, no model): the ReAct JSON adapter, the sandboxed coding tools, the Sokoban engine, the coding file-diff snapshots, the provider model overrides, the LoopMart target site, the browser environment, the eval suites, and the deterministic scorer — each regression test naming the bug it locks down.

Security note

run_command executes real shell commands, confined to the per-run sandbox directory (os.tmpdir()/loopforge-run-<runId>) with a 30-second timeout, and the file tools are path-confined to that sandbox via realpath (so a symlink inside the sandbox can't escape it). The browser environment is origin-allowlisted to the seeded local shop (http://localhost:8788) — any other URL is rejected before Chromium is even launched. This is a local development / portfolio tool: there is no syscall or network isolation, a command still has shell access within the sandbox dir, and runs only ever start when you start them. Run LoopForge on your own machine, not as a service exposed to untrusted input. Details in docs/ENVIRONMENTS.md.