Forge — LLM Evaluation & Prompt Engineering Console

Forge is a self-hostable, production-ready prompt engineering and scientific LLM evaluation dashboard designed for building robust, enterprise-grade LLM applications.

---

Key Features

• 🎭 Prompt Playground: Multi-panel playground using dynamic Monaco Editor with custom Variable highlighting ({{variables}}) and real-time streaming tokens, latency tracking, and cost indicators.
• 📚 Prompt Library: Complete CRUD prompt management with strict schema versioning and side-by-side Monaco Diff compare visualizations.
• 🧪 Scientific Evaluations: Set up dataset suites (CSV/JSONL) and evaluate LLM responses using 10 Core Judges:
  • EXACT_MATCH, CONTAINS, REGEX, JSON_SCHEMA
  • BLEU (translation accuracy), ROUGE-L (summarization recall)
  • SEMANTIC_SIMILARITY (cosine similarity of frequency vectors)
  • LATENCY_THRESHOLD & COST_THRESHOLD
  • LLM_JUDGE (structured validation rubrics generated by Claude)
• 📊 Metrics Dashboard: Full analytics overview utilizing theme-synced Recharts (Stacked Area Token trends, Multi-line Costs, Latency distribution histograms, and Model comparison Radars).
• 🔒 Enterprise API Access: Workspace-scoped tokens protected by hashed SHA-256 matching for programmatic REST integrations (GET /api/v1/prompts, GET /api/v1/traces, POST /api/v1/evaluations/run).
• ⚡ Webhook Subscriptions: Register webhook handlers to trigger on evaluation outcomes (EVAL_RUN_COMPLETED, etc.) signed with secure HMAC-SHA256 headers.
• 🔔 Real-Time Notification Center: Radix UI Popover alert bell fetching user event notifications.

---

Tech Stack

• Core Framework: Next.js 15 (App Router, React 19, TypeScript)
• Styling: Tailwind CSS, Obsidian CSS Variables (Light / Dark responsive)
• Database & ORM: PostgreSQL, Prisma ORM
• Cache & Queueing: Redis, ioredis, Memory Queue fallback
• Graphics: Three.js, React Three Fiber (rotating neural networks, dynamic waveforms)
• Animations: Framer Motion, GSAP (respects prefers-reduced-motion profiles)
• Testing: Vitest, Playwright (E2E)

---

Getting Started

Prerequisites

• Node.js >= 20.x
• pnpm >= 9.x
• Docker & Docker Compose (optional, for Postgres + Redis)

Environment Setup

Create a .env file in the root directory:

# Database
DATABASE_URL="postgresql://postgres:postgres@localhost:5432/forge?schema=public"

# Redis
REDIS_URL="redis://localhost:6379"

# Auth.js Secrets
AUTH_SECRET="your-auth-secret-here"
NEXTAUTH_URL="http://localhost:3000"

# LLM Providers (Required for Live API Queries)
OPENAI_API_KEY="sk-..."
ANTHROPIC_API_KEY="sk-ant-..."
GOOGLE_GENERATIVE_AI_API_KEY="AIzaSy..."

# Transactional Invites (Optional, fallbacks to dry-run mocks)
RESEND_API_KEY="re_..."

Local Installation

1. Install package dependencies:

   pnpm install

2. Start PostgreSQL and Redis via Docker Compose:

   docker compose up -d

3. Sync database models:

   pnpm db:push

4. Seed the database with default workspaces and mock traces:

   pnpm db:seed

5. Launch the development server:

   pnpm dev

---

Running Tests

Execute the unit testing suite (Vitest):

pnpm test

Execute end-to-end user journey tests (Playwright):

pnpm test:e2e

---

Deployment (Docker Standalone)

Forge is configured for multi-stage Docker builds.

1. Build the production image:

   docker build -t forge-app .

2. Run the application:

   docker run -p 3000:3000 --env-file .env forge-app

---

📚 Strategic Documentation & Vision

To aid the Anthropic OSS Grant committee and open-source contributors, we have documented our core engine and our roadmap for the future:

• 🔬 Architecture Review: A deep dive into the engineering of Forge 1.0. Covers our zero-allocation WebGL memory safety, local implementations of the 10 core evaluation algorithms (including BLEU, ROUGE-L, and Semantic Similarity), and our secure HMAC-SHA256 webhook implementations.
• 🚀 Future Roadmap (v2.0 & v3.0): Our strategic vision for the grant. Outlines the evolution of Forge into a multimodal, WebXR-enabled immersive evaluation environment with autonomous research agents and a decentralized community judge marketplace.