A playground to compare search + answer setups side by side. Pick embedding, optional rerank, and generation models, ask the same question, and compare answers, evidence, cost, and speed. Each setup runs as a durable Render Workflows task and is scored by a shared judge model.
This repo demonstrates how to run a multi-model RAG bake-off on Render:
| Platform | Role |
|---|---|
| Render Workflows | Fan-out durable tasks for ingest, embed, retrieve, generate, and judge with retries and leases |
| OpenRouter | One API key for embeddings, optional rerank, chat, and the judge model |
| Render Postgres | Corpus chunks, pgvector embeddings, runs, trials, and cost receipts |
| Render Web Services | Hosts the API and Compare / Configure / Inspect UI |
Configure setups, run them in parallel, then inspect evidence and scores. These are screenshots from the live deploy.
Pick a question and compose one or more setups (embedding + optional rerank + generation).
Answers appear side by side. The live execution timeline shows retrieve / rerank / generation / judge as clock-time bars, including retries and failures.
Select an answer to see retrieved passages, stage receipts, and judge dimensions. Failed setups show the provider reason after retries.
- Browser starts a run from Configure
- Web service admits the run, persists the plan, and dispatches
run_bakeoff - Render Workflows executes the pipeline:
| Workflow task | What it does |
|---|---|
ingest_document / corpus prep |
Chunks demo or uploaded text into Postgres |
embed_batch |
Embeds missing chunks per embedding model into pgvector |
run_trial |
Retrieve → optional rerank → generate → judge for one setup × question |
run_bakeoff |
Orchestrates fan-out, aggregation, and run status |
- Stage events stream into Compare so you can watch concurrency, retries, and cost as they happen
- Render account
- OpenRouter API key
- Render API key (to start and cancel workflow tasks)
- Click Deploy to Render above (or create a Blueprint from
render.yaml) - Set secrets when prompted:
OPENROUTER_API_KEYRENDER_API_KEYJUDGE_MODEL(any chat slug you can call, e.g.openai/gpt-4o-mini)
- Create the Workflow service manually (Blueprints do not create Workflows yet):
- Dashboard → New → Workflow
- Same repo, root directory empty (pnpm workspace needs the repo root)
- Build:
pnpm install && pnpm build:workflows - Start:
node apps/workflows/dist/index.js - Same region as the web service and Postgres
- Slug should match
WORKFLOW_SLUG(Blueprint default:ragtime-workflows) - Wire
DATABASE_URL,OPENROUTER_API_KEY,JUDGE_MODEL, andAPP_URL
- Open your web service URL (or try the live demo), load the demo library, compose two setups, and click Run
A small smoke run with budget-tier models usually lands in low single-digit USD, always bounded by MAX_RUN_BUDGET_USD (default $5).
| Feature | Description |
|---|---|
| Side-by-side setups | Compare embedding / rerank / generation stacks on the same question |
| Live execution timeline | Gantt-style bars for retrieve, rerank, generation, and judge |
| Provider failure reasons | After workflow retries, failed setups show the OpenRouter / dispatcher message |
| Resumable stages | Completed retrieve / generate / judge work is checkpointed so retries skip finished work |
| Per-run budget | Pre-call reservations and an idempotent cost ledger |
| Shared-deploy safe | Session-scoped runs with admission limits per session and deployment-wide |
| Live model catalog | Pickers come from OpenRouter; no hardcoded model slug list |
| Variable | Where | Description |
|---|---|---|
DATABASE_URL |
Web + Workflow | Render Postgres (pgvector) connection string |
OPENROUTER_API_KEY |
Web + Workflow | Model calls and the live catalog |
RENDER_API_KEY |
Web | Start and cancel workflow tasks |
JUDGE_MODEL |
Web + Workflow | Default judge chat slug; runs are rejected without one |
WORKFLOW_SLUG |
Web | Must match the Dashboard workflow slug (ragtime-workflows) |
APP_URL |
Workflow | Public web URL for OpenRouter HTTP-Referer |
OPENROUTER_APP_TITLE |
Both | Defaults to Answer Arena |
MAX_RUN_BUDGET_USD |
Both | Hard per-run ceiling (default 5) |
MAX_PROVIDER_CALL_USD |
Both | Max reserved for one provider call (default 0.5) |
MAX_ACTIVE_RUNS_PER_SESSION |
Web | Concurrent runs per browser session (default 1; 0 disables) |
MAX_ACTIVE_RUNS_TOTAL |
Web | Concurrent runs deployment-wide (default 5; 0 disables) |
CHAOS_FAILURE_RATE |
Workflow | Injected pre-spend failures for resilience demos (default 0) |
Runs are scoped to an anonymous session cookie. Over the admission limit, POST /api/runs returns 429 with session_run_limit or global_run_limit. Reloading reattaches via GET /api/runs/active. Worst-case concurrent spend is MAX_ACTIVE_RUNS_TOTAL × MAX_RUN_BUDGET_USD ($25 with Blueprint defaults).
apps/web/ Fastify API + Vite React SPA
apps/workflows/ Render Workflow tasks (ingest, embed, trial, bake-off)
packages/core/ Ports, pipeline stages, prompts, schemas
packages/composition/ Env-selected gateway + workflow wiring
packages/db/ Drizzle schema, migrations, seed, PgVectorStore
packages/gateway-openrouter/
render.yaml Blueprint (web + Postgres)
static/images/ README screenshots from the live deploy
Internal npm packages still use the @ragtime/* scope; that is an implementation detail and does not change the product name.
| Problem | Solution |
|---|---|
| Run stays in draft / 502 on start | Check RENDER_API_KEY, WORKFLOW_SLUG, and that the Workflow service is live in the same region |
| "Judge model required" | Set JUDGE_MODEL on web and workflows to a chat slug you can call |
| Setup shows FAILED after retries | Open Inspect / the answer card for the provider reason (rate limit, missing model, credits) |
sorry, too many clients already |
Lower TRIAL_FANOUT_BATCH / EMBED_FANOUT_BATCH, or raise DB_POOL_MAX on a larger Postgres plan |
429 with an active run |
Cancel the current run, or raise MAX_ACTIVE_RUNS_PER_SESSION / MAX_ACTIVE_RUNS_TOTAL |
| Empty model pickers | Set OPENROUTER_API_KEY on the web service; check /api/models |
Logs: web service and Workflow service logs in the Render Dashboard.
pnpm testBuilds workspace packages, then runs core / db / gateway / web unit tests. Set TEST_DATABASE_URL to also run the Postgres budget-recovery suite.
Render:
OpenRouter:
Issues and PRs are welcome on github.com/render-examples/answer-arena. Keep changes small and focused; match existing module boundaries (packages/composition for wiring, packages/core for pipeline logic, apps/workflows for durable tasks).