EvalKit

Command-line tool for evaluating and comparing LLM performance on sports news classification. Given a text article, each model must decide whether it belongs to the sports or not sports category. EvalKit runs the same task across multiple backends, computes standard binary classification metrics, and generates reproducible comparison reports.

---

Table of contents

• Features
• Prerequisites
• Installation
• Quick start
• Backends and models
• Dataset format
• Usage
  • evalkit run
  • evalkit compare
  • evalkit list-backends
  • evalkit list-tasks
• Metrics
• Reports
• Project structure
• License

---

Features

• 5 ready-to-use backends: OpenAI, Anthropic Claude, Google Gemini, Ollama (local), and rule-based classifier
• Standard metrics: accuracy, precision, recall, specificity, and confusion matrix
• JSON reports with timestamp, backend, and evaluated model
• Comparison table across all generated reports sorted by accuracy
• No heavy dependencies: no ML frameworks or GPU required

---

Prerequisites

• Python 3.10 or higher
• For the Ollama backend: Ollama installed and running (ollama serve)
• For the OpenAI, Anthropic, and Gemini backends: a valid API key for each

---

Installation

# 1. Clone the repository
git clone https://github.com/yourusername/evalkit.git
cd evalkit

# 2. Create and activate a virtual environment
python -m venv venv
source venv/bin/activate      # Linux/Mac
# venv\Scripts\activate       # Windows

# 3. Install EvalKit
pip install -e .

# 4. Configure credentials
cp .env.example .env

Edit .env with the required keys:

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=...
OLLAMA_MODEL=llama3.2        # optional, defaults to llama3.2

Only the keys for the backends you intend to use need to be set.

---

Quick start

# Rule-based classifier — no API key or external model required
evalkit run --backend rules

# Local model with Ollama (requires ollama serve)
evalkit run --backend ollama --model llama3.2:3b

# OpenAI
evalkit run --backend openai --model gpt-4o

# Compare all results in a table
evalkit compare

---

Backends and models

OpenAI

Model	Notes
gpt-4o	Default
gpt-4o-mini
gpt-4-turbo
gpt-4
gpt-3.5-turbo
o1-preview	Reasoning model
o1-mini	Reasoning model
o3-mini	Reasoning model

Anthropic Claude

Model	Notes
claude-sonnet-4-6	Default
claude-opus-4-6	Highest capability
claude-haiku-4-5-20251001	Fastest
claude-3-5-sonnet-20241022
claude-3-5-haiku-20241022
claude-3-opus-20240229

Google Gemini

Model	Notes
gemini-2.0-flash	Default
gemini-2.0-flash-thinking
gemini-1.5-pro
gemini-1.5-flash
gemini-1.5-flash-8b

Ollama (local)

Model	Notes
llama3.2:3b	Default
llama3.1
mistral
phi3
qwen2.5
gemma2
deepseek-r1

Any model available in your local Ollama installation can be used by passing it with --model.

Rule-based

Lexical classifier using sports-related keywords in both Spanish and English. Requires no API key or external connection. Useful as a baseline for comparison.

---

Dataset format

EvalKit expects files in JSONL format (one example per line):

{"texto": "Real Madrid wins 3-2 against Barcelona", "es_deporte": true}
{"noticia": "The economy grows 2% this quarter", "es_deporte": false}

Field	Values	Description
texto or noticia	string	Article text to classify
es_deporte	true / false	Ground truth label

Dataset files go in the data/ directory.

---

Usage

evalkit run

Runs a full evaluation on a dataset.

evalkit run [OPTIONS]

Option	Default	Description
--backend	ollama	Backend to use: ollama, openai, anthropic, gemini, rules
--model	backend default	Model name (overrides the backend's default model)
--data	data/sports_or_not.jsonl	Path to the JSONL dataset file
--limit	no limit	Maximum number of examples to evaluate
--summary	off	Include a summary section in the JSON report
--examples	off	Include the first 20 evaluated examples in the JSON report

Examples:

# Basic evaluation
evalkit run --backend anthropic --model claude-sonnet-4-6

# Quick test with 20 examples
evalkit run --backend openai --model gpt-4o --limit 20

# Full report with summary and examples
evalkit run --backend gemini --summary --examples

# Custom dataset
evalkit run --backend rules --data data/my_dataset.jsonl

Console output:

Evaluating task : sports_or_not
  Backend        : anthropic
  Model          : claude-sonnet-4-6
  Loading data from data/sports_or_not.jsonl...
  200 examples loaded
  Evaluation complete

  RESULTS:
  Accuracy    : 91.50%
  Precision   : 93.20%
  Recall      : 89.80%
  Specificity : 93.40%
  Correct     : 183/200

  Confusion Matrix:
                Predicted +   Predicted -
  Actual +  |   TP=98          FN=11
  Actual -  |   FP=6           TN=85

  Report saved to: reports/eval_20260321_103000.json

---

evalkit compare

Loads all generated JSON reports and displays a comparison table sorted by accuracy from highest to lowest.

evalkit compare [OPTIONS]

Option	Default	Description
--dir	reports/	Directory to search for JSON report files

Example output:

  Backend     Model               Date              Total  Accuracy  Precision    Recall  Specificity
  --------------------------------------------------------------------------------------------------------
  anthropic   claude-sonnet-4-6   2026-03-21 10:30    200    91.50%     93.20%    89.80%       93.40%
  openai      gpt-4o              2026-03-21 11:00    200    89.20%     90.10%    87.50%       91.00%
  ollama      llama3.2            2026-03-21 09:00    200    78.30%     81.40%    74.00%       82.50%
  rules       -                   2026-03-21 08:30    200    72.10%     68.50%    80.20%       65.30%

---

evalkit list-backends

Lists the available backends and their supported models.

evalkit list-backends

---

evalkit list-tasks

Lists the available evaluation tasks.

evalkit list-tasks

---

Metrics

EvalKit computes the following standard binary classification metrics with respect to the positive class (sports):

Metric	Formula	Description
Accuracy	(TP + TN) / total	Overall proportion of correct predictions
Precision	TP / (TP + FP)	Of all articles classified as sports, how many actually are
Recall	TP / (TP + FN)	Of all actual sports articles, how many the model detected
Specificity	TN / (TN + FP)	Of all non-sports articles, how many the model correctly identified

TP = true positive · TN = true negative · FP = false positive · FN = false negative

---

Reports

Each evalkit run execution generates a JSON file in reports/ with the following structure:

{
  "backend": "anthropic",
  "model": "claude-sonnet-4-6",
  "date": "2026-03-21T10:30:00.000000",
  "metrics": {
    "total": 200,
    "correct": 183,
    "accuracy": 0.915,
    "precision": 0.932,
    "recall": 0.898,
    "specificity": 0.934,
    "confusion_matrix": { "tp": 98, "tn": 85, "fp": 6, "fn": 11 }
  }
}

reports/latest.json is always overwritten with the most recent report.

---

Project structure

evalkit/
├── src/evalkit/
│   ├── cli.py                # Command-line interface
│   ├── backends/             # Backend implementations
│   │   ├── rules.py          # Keyword-based lexical classifier
│   │   ├── local_ollama.py   # Local models via Ollama
│   │   ├── api_openai.py     # OpenAI API
│   │   ├── api_anthropic.py  # Anthropic Claude API
│   │   └── api_gemini.py     # Google Gemini API
│   ├── eval/
│   │   ├── run_eval.py       # Evaluation engine
│   │   └── metrics.py        # Metrics calculation and report generation
│   └── tasks/
│       └── sports_or_not.py  # Sports news classification task
├── data/                     # JSONL dataset files
├── reports/                  # Generated evaluation reports
├── .env.example              # Environment variables template
└── pyproject.toml

---

License

MIT