Skip to content

v0.1.0

Choose a tag to compare

View all tags
@YatootaY YatootaY released this 17 Oct 10:56
· 9 commits to main since this release
v0.1.0
786a6db

First public release of the open-source engine behind PaperLab.space: evaluate LLMs on your own spreadsheet data and get paper-ready results.

pip install 'paperlab-space[cli]'
paperlab-space init my_data.xlsx
paperlab-space run run.yaml

What's included

  • Data: .xlsx (any sheet), .csv, .tsv; encoding detection and Unicode-safe handling for non-Latin scripts (tested with Burmese, Thai, Chinese, Japanese, Korean); validation for empty cells, duplicate IDs, and garbled text
  • Models: OpenRouter (including free models), Gemini, OpenAI, Anthropic, and anything else LiteLLM supports, side by side, with your own API keys
  • Runs: per-model rate limits tuned for free tiers, retries with backoff, resume after interruption, and a cost and time estimate before anything is sent
  • Metrics: exact match, accuracy (with label extraction), precision/recall/F1, token F1
  • Output: Excel workbook, booktabs LaTeX table, reproducibility YAML (model versions, prompt, seed, metric settings, data hash), full results JSON
  • CLI: paperlab-space init | validate | estimate | run [--resume] | metrics

See the README for the quickstart and PLAN.md for what's next (BLEU/chrF/ROUGE, LLM-as-judge, significance tests).

Assets 2
Loading