v0.2.0

What's New

42 Datasets (was 7)

• QA: HotpotQA, Natural Questions, MuSiQue, NarrativeQA, TriviaQA, FRAMES, QuALITY, QASPer
• Knowledge/MC: MMLU, MMLU-Pro (10-choice), ARC-Challenge, TruthfulQA, GPQA, HellaSwag, WinoGrande
• Reasoning/Math: GSM8K, DROP, MATH, MGSM (multilingual, configurable), BIG-Bench Hard
• Code: HumanEval, MBPP (both with execution-based pass@1)
• Summarization: Multi-News, DialogSum, QMSum, SummScreenFD, MeetingBank, GovReport
• NLI: ContractNLI, SciFact
• Instruction Following: IFEval (19 programmatic checks), AlpacaEval
• Multi-Turn: MT-Bench (2-turn conversations)
• Long Context: LongBench, LongBench v2, InfiniteBench, NoLiMa
• Agent Traces: BFCL, APIGen, SWE-bench (full/verified/lite)

8 Evaluators (was 1)

All auto-wired based on dataset selection — no manual configuration needed.

• AnswerQuality — F1, exact match, recall, contains (all datasets)
• SummarizationQuality — ROUGE-L (summarization datasets)
• MultipleChoiceAccuracy — letter extraction A-J (MC datasets)
• CodeExecution — subprocess pass@1 (HumanEval, MBPP)
• MathEquivalence — LaTeX normalization + numeric comparison (MATH, GSM8K, MGSM)
• NLILabelMatch — label extraction with alias mapping (ContractNLI, SciFact)
• IFEvalChecker — 19 programmatic instruction checks (IFEval)
• LLMJudge — external LLM scoring via --judge-url

7 Metrics (was 4)

• MeanScore, PassRate, CompressionRatio, CostOfPass
• New: Latency (mean/median/p95/p99), PerDatasetBreakdown, ParetoRank

Infrastructure

• Concurrent execution — --max-workers N for parallel example processing
• Result caching — --cache-dir saves progress; resume interrupted runs
• HTML reporter — --output html with dark mode, interactive filtering
• Multi-turn support — process_conversation() protocol for MT-Bench and custom chatbots
• API cost tracking — extracts prompt/completion tokens from API responses
• Retry logic — exponential backoff on 429/5xx/connection errors (configurable)
• Rich progress bars — auto-detected when rich is installed

Documentation

• Per-evaluator usage examples with code snippets
• Cookbook with 10 workflow recipes
• Full dataset tables organized by category

v0.1.0

Initial release of context-bench.

• OpenAIProxy system for benchmarking any OpenAI-compatible endpoint
• Built-in metrics: MeanScore, PassRate, CompressionRatio, CostOfPass, ParetoRank
• Dataset loaders: HotpotQA, GSM8K, BFCL, APIGen, SWE-agent, local JSONL
• Markdown and JSON reporters
• Works out of the box with Headroom and Compresr