ModelForge

ModelForge is a portfolio-grade benchmark workbench for comparing local and OpenAI-compatible LLM endpoints. It combines a React/Vite frontend, an ASP.NET Core API, a Python worker, PostgreSQL persistence, live run updates over SignalR, seeded demo data, exportable reports, Docker packaging, and CI validation.

Stack

• Frontend: React 19, TypeScript, Vite, React Query, React Hook Form, Zod, Recharts, SignalR
• API: ASP.NET Core 8, EF Core, Npgsql, FluentValidation, JWT cookie auth, Serilog
• Worker: Python 3.12+, FastAPI, httpx, psycopg, cryptography
• Data: PostgreSQL 16

Features

• Authentication with access and refresh cookies
• Model catalog with encrypted API keys and connection checks
• Benchmark suite and case management
• Benchmark run creation, retry, cancellation, live progress, and comparison views
• Playground for ad-hoc completions and side-by-side model checks
• Report export in JSON, CSV, Markdown, and print-friendly HTML
• Demo seed with four deterministic mock models and three benchmark collections

Quickstart

1. Copy .env.example to .env.
2. Start the full stack:

docker compose up --build

3. Open:

• Frontend: http://localhost:3000
• API: http://localhost:8080
• Worker health: http://localhost:8090/health

Demo credentials:

• Email: demo@modelforge.local
• Password: ModelForge123!

Local development

Frontend:

cd frontend
npm ci
npm run dev

API:

dotnet tool restore
dotnet build api/ModelForge.Api/ModelForge.Api.csproj
dotnet test api/ModelForge.Api.Tests/ModelForge.Api.Tests.csproj
dotnet run --project api/ModelForge.Api/ModelForge.Api.csproj

Worker:

python3 -m pip install -e 'worker[dev]'
python3 -m pytest worker/tests
uvicorn main:app --app-dir worker/app --host 0.0.0.0 --port 8090

Database migrations:

dotnet tool restore
dotnet tool run dotnet-ef database update --project api/ModelForge.Api/ModelForge.Api.csproj --startup-project api/ModelForge.Api/ModelForge.Api.csproj

Documentation

• Architecture
• Development
• Deployment