feat: LLM-as-Judge evaluator + timeline scoring

[risjai]

Summary

• Add llm_judge evaluator type that uses LLM function calling (Braintrust pattern) to score agent outputs on criteria like correctness, coherence, safety, relevance, and task_completion
• Add rewind eval score CLI command that scores session timelines and compares original vs. forked timelines after a replay
• Wire scoring into the store with a new timeline_scores table, enabling cached re-scoring and persistence

What's included

Phase 1 — Rust evaluator type

• llm_judge match arm in evaluator.rs dispatching to Python subprocess
• run_llm_judge_evaluator() with 120s timeout, sends {input, output, expected, config} on stdin
• builtin_types() updated, CLI config shorthand (-c correctness → {"criteria": "correctness"})

Phase 2 — Python LLM judge module

• llm_judge.py with 5 criteria presets and full prompt templates (correctness, coherence, relevance, safety, task_completion)
• Function calling pattern for structured choice+reasoning extraction (OpenAI SDK compatible)
• Lazy openai import with clear error, retry on 429/5xx, temperature=0 default
• Validates expected is required for reference-based criteria (correctness)
• llm_judge_evaluator() factory + registered in _BUILTIN_EVALUATORS
• Subprocess entry point: python3 -m rewind_agent.llm_judge

Phase 3 — Fork-score integration

• timeline_scores table with UNIQUE(timeline_id, evaluator_id) upsert constraint
• extract_timeline_output() — finds first/last LlmCall, skips ToolCall/ToolResult steps
• rewind eval score <session> -e <evaluator> [--compare-timelines] [--expected <json>] [--json]
• Score caching, pretty table output with delta and verdict

Stats

• 11 files changed, ~1,435 lines added
• 42 new tests (9 Rust + 33 Python), all passing
• Clippy clean, zero regressions on full workspace (202 Rust + 225 Python tests)

Usage

# Create an LLM judge evaluator
rewind eval evaluator create "quality" -t llm_judge -c correctness

# Score a session's main timeline
rewind eval score latest -e quality

# Compare scores across original and forked timelines
rewind replay latest --from 5
rewind eval score latest -e quality --compare-timelines

# With expected output for correctness
rewind eval score latest -e quality --expected '{"answer": "Tokyo"}'

# JSON output for CI
rewind eval score latest -e quality --json

# Python SDK
result = rewind_agent.evaluate(
    dataset=ds,
    target_fn=my_agent,
    evaluators=["llm_judge"],  # default: correctness + gpt-4o-mini
)

# Or with config
result = rewind_agent.evaluate(
    dataset=ds,
    target_fn=my_agent,
    evaluators=[
        rewind_agent.llm_judge_evaluator(criteria="correctness", model="gpt-4o"),
        rewind_agent.llm_judge_evaluator(criteria="safety"),
    ],
)

Test plan

• Rust: builtin_types() includes llm_judge, is_valid_type() validates it
• Rust: extract_timeline_output() finds first/last LlmCall, skips tool steps
• Rust: validate_session_for_scoring() errors on missing session
• Python: All 5 criteria presets have correct templates and scores
• Python: Template rendering handles strings, dicts, None, lists
• Python: Correctness without expected raises clear ValueError
• Python: Safety/coherence work without expected
• Python: Mocked LLM calls return correct scores for each choice
• Python: Missing openai dep, API errors, and unknown choices handled gracefully
• Python: llm_judge registered in _BUILTIN_EVALUATORS and resolvable by string
• Python: Subprocess entry point handles valid/invalid stdin JSON
• Full workspace: 202 Rust + 225 Python tests pass, clippy clean
• Manual: rewind eval evaluator create + rewind eval score e2e flow
• Manual: --compare-timelines with fork/replay produces delta table

🤖 Generated with Claude Code

[risjai]

Code Review: LLM-as-Judge Evaluator + Timeline Scoring

Overall: Good implementation that closely follows the plan. 2 bugs, 1 security concern, 2 behavioral issues to address.

---

Bugs (P0)

1. Retry loop matches "429" as substring, not HTTP status code

python/rewind_agent/llm_judge.py:282

err_str = str(e)
if "429" in err_str or "500" in err_str or "502" in err_str or "503" in err_str:

This matches "429" anywhere in the error string. An error like "Token limit 500 exceeded" or "Connection to 10.0.0.429" would trigger retries incorrectly. The OpenAI SDK raises typed exceptions (openai.RateLimitError, openai.APIStatusError with .status_code) — check those instead of string matching. Also missing: a test for the retry path.

2. Score cache has no bypass — stale scores returned after code changes

crates/rewind-cli/src/main.rs:2277

Once scored, rewind eval score always returns the cached value from timeline_scores. If the user changes their agent, replays to a new fork, then re-scores the same timeline — the stale score is returned. The upsert (ON CONFLICT UPDATE) in create_timeline_score exists but is unreachable because the cache check short-circuits first.

Add a --force flag that skips the cache check.

---

Security (P1)

3. API key may leak into stored reasoning field

crates/rewind-eval/src/evaluator.rs:106

If the Python subprocess crashes during client construction, stderr may contain the API key (e.g., openai.AuthenticationError: Incorrect API key provided: sk-proj-abc...). This stderr is stored in timeline_scores.reasoning (persisted to SQLite) and printed to the terminal. Consider sanitizing stderr before storage — redact strings matching sk-[a-zA-Z0-9-]+.

---

Behavioral Issues (P2)

4. _session_id parameter unused in public API

crates/rewind-eval/src/timeline_scoring.rs:16

extract_timeline_output() accepts _session_id but never uses it (underscore-prefixed to suppress warnings). Either use it for validation or remove it — a dead parameter in a public API is confusing for future callers.

5. Multi-fork delta compares first/last timeline, not main vs. each fork

crates/rewind-cli/src/main.rs:2330

Delta is last - first in timeline order. For 3+ timelines (main, fork-1, fork-2), the comparison is fork-2 vs. main — silently ignoring fork-1. Consider comparing each fork against main, or documenting this as first-vs-last only.

---

Nits (P3)

6. ValueError exits with code 0 in subprocess, masking config errors as zero scores

python/rewind_agent/llm_judge.py:285 — When run_judge raises ValueError ("correctness requires expected"), the subprocess catches it, outputs score: 0.0, and exits 0. The Rust side sees success. The user gets a zero score that looks like the LLM judged poorly, not a config error. Exit with code 1 for ValueError so Rust can surface it as an error.

7. resolve_timeline_ref called but not in the diff

crates/rewind-cli/src/main.rs:2228 — Verify this existing function handles label-based lookup (not just ID/prefix).

---

Tests: Solid ✓

42 tests (9 Rust + 33 Python), good coverage of presets, template rendering, expected validation, choice→score mappings, error paths, SDK integration, and subprocess entry point. Missing: a test for the retry logic (429 → retry → succeed on attempt 2).

---

Recommend

Fix #1 (retry by exception type) and #2 (add --force flag) before merge. #3 (API key leak) is worth a fast-follow. Rest are non-blocking.