Skip to content

v2.5.0 — Text-to-SQL Evaluation (Spider 1.0 / BIRD / Spider 2.0-lite)

Choose a tag to compare

View all tags
@lianghsun lianghsun released this 27 Mar 03:33
· 57 commits to main since this release
v2.5.0
9f812c9

Text-to-SQL Evaluation

Unified text2sql evaluation method supporting three major text-to-SQL benchmarks:

New Features

  • SQL Extractor: extracts SQL from LLM responses (```sql blocks, plain SELECT, mixed text)
  • SQL Scorer: two scoring modes
    • Execution Accuracy (EX): executes predicted + gold SQL against SQLite, compares result sets (default)
    • Exact Match (EM): normalized SQL string comparison
  • Read-only SQLite execution: mode=ro + PRAGMA query_only = ON

Supported Benchmarks

Benchmark Example Size Databases Notes
Spider 1.0 10 rows concert_singer, pets_1 Cross-domain text-to-SQL
BIRD 10 rows california_schools, financial With external knowledge (evidence)
Spider 2.0-lite 10 rows book_store SQLite-only subset (85 questions in full)

Spider 2.0: Only lite version supported — full version requires BigQuery/Snowflake cloud credentials.

Config Example

evaluation:
  evaluation_method: "text2sql"
  strategy_config:
    text2sql_scoring_mode: "exec"
    text2sql_db_base_path: "datasets/example/spider/databases"

Test Results (Devstral-Small-2-24B-Instruct-2512, EX mode)

Dataset Accuracy
Spider 1.0 90%
BIRD 60%
Spider 2.0-lite 80%

Full Changelog: v2.4.0...v2.5.0

Assets 2
Loading