Added
- Added LLM-assisted evaluation dataset generation.
- integration: Haystack integration and Elasticsearch retriever support.
- example: Streamlit evaluation dashboard example.
- Improved retriever, dataset loading, and metric logging.
- Updated token-overlap judge defaults
Evaluation Dataset Improvements
- QueryExample now supports expected_doc_ids.
- When expected_doc_ids are present, Evaluator compares retrieved doc_ids directly instead of relying on answer-text judging. JSON and CSV dataset loading both support this field.
Aliases supported:
- expected_doc_ids
- relevant_doc_ids
Examples And UI
Added a Streamlit dashboard example for running and visualizing evaluations:
- examples/evals-streamlit-dashboard/index.py
- examples/evals-streamlit-dashboard/run_evals_ui.py
Dependencies
Updated pyproject.toml:
-
Version bumped from 0.0.2 to 0.0.3.
-
Added runtime dependency:
- tqdm>=4.67.0
-
Added optional extras:
- elasticsearch
- haystack
Full Changelog: v0.0.2...v0.0.3