v0.1.0 — Initial release

First public release of padosoft/eval-harness — a standalone Laravel evaluation framework for RAG / LLM applications.

Highlights

• Pluggable golden datasets — YAML-loaded strict-schema datasets with per-metric scoring; extend with custom metrics via Eval::extend(...).
• Three production-grade metrics — ExactMatchMetric (byte-equality), CosineEmbeddingMetric (laravel/ai SDK provider, configurable timeout via TimeoutNormalizer), LlmAsJudgeMetric (deterministic seed + strict-JSON validation).
• Two report formats — Markdown (human) + canonical JSON (machine-readable, byte-identity for diffing across runs).
• Artisan CLI — eval-harness:run <dataset> with strict failure handling.
• Standalone agnostic — zero coupling to AskMyDocs or any sister Padosoft package; works on any Laravel 12/13 project.
• CI matrix — PHP 8.3 / 8.4 / 8.5 × Laravel 12 / 13. 109 Unit + 3 Architecture tests, ~600 assertions. Pint + PHPStan level 6 clean.
• 🚀 AI vibe-coding pack included — full Padosoft .claude/ skills + rules + agents.

Installation

composer require padosoft/eval-harness

Roadmap (v0.2)

• Parallel batch evals via Laravel queues.
• Cohort metrics aggregating by metadata.tags.
• Histogram view + ROUGE-L + BERTScore + refusal-quality + citation-groundedness metrics.
• Adversarial harness (prompt injection / jailbreak datasets).
• Regression detection (compare against last N runs).
• Minimal HTML web dashboard.

Comparison

vs OpenAI Evals + LangSmith + Ragas + Promptfoo + DeepEval — see README "Comparison vs alternatives" for the side-by-side. Differentiators: native Laravel integration, deterministic seed throughout, the .claude/ pack ships in the box.

License: Apache-2.0.