v1.6.0

What's Changed

• feat(sampling): repeat rows and report the difference a run could actually detect (P1) by @lopadova in #49
• feat(regression): baselines and per-row regression gating (P2) by @lopadova in #50
• feat(online): production failures become regression tests, safely (P6) by @lopadova in #54
• feat: land P3a, P4, P5 and P6 on main by @lopadova in #55
• docs: a README opening that says what this is, and the companion cross-links by @lopadova in #56

Full Changelog: v1.5.0...v1.6.0

v1.4.0 — WOW enterprise documentation site

📚 WOW enterprise documentation site

The docmd documentation site has been rewritten to be immediately compelling to a developer who has never seen the package:

• Hero banner + admin dashboard screenshot, badges and a one-line value proposition
• "What it is in a minute" — a Laravel-native evaluation framework for RAG & LLM apps that gives the non-deterministic part of your stack the same regression protection your business logic has
• Problem → solution table, who it's for and the differentiating moats (Laravel-native, retrieval metrics, CI regression gate, deterministic evaluation, dataset management)
• A head-to-head competitor matrix (vs Python eval libs, manual spot-checks, no eval)
• The architecture diagram, a 30-second quickstart and clear next steps

Semantic search and all docmd plugins verified active; the site builds cleanly (32 pages) on Cloudflare Pages.

Full changelog: v1.3.0...v1.4.0

v1.3.0 — retrieval metrics, judge calibration, online monitoring

Additive, backward-compatible release. All v1 contracts are preserved.

Added

• Retrieval-ranking metrics (domain-agnostic): retrieval-hit-at-k, retrieval-recall-at-k, retrieval-mrr, retrieval-ndcg-at-k (binary or graded gains), answer-containment-at-k, on a shared RankedRetrieval parser + AbstractRetrievalRankingMetric base. metrics.retrieval.default_k config + per-sample metadata.k.
• ordinal-distance metric — partial credit for ordered labels (exact 1.0 / off-by-one 0.5 / further 0.0).
• Judge calibration — eval-harness:calibrate-judge: verdict agreement rate, confusion matrix, length-bias signal, self-preference guard, CI gating.
• Online / production monitoring (off by default) — OnlineMonitor::capture(), queued JudgeLiveSampleJob, first package migration (eval_harness_online_scores), OnlinePassRateDropped drift event, read-only GET /{prefix}/online/{dataset}/trend endpoint, eval-harness-migrations publish tag.
• RuntimeOptions::normalizeUnitInterval() helper.

Companion admin UI screen: padosoft/eval-harness-admin#6 (requires this release).

🤖 Generated with Claude Code

v1.2.0 - Report API completeness for companion UI

Highlights

• Report diff endpoint: GET /<configured-prefix>/reports/{id}/diff/{otherId} with metric, cohort, sample-count, failure-count, and adversarial deltas.
• Adversarial manifest discovery endpoints for configured manifest storage.
• Live batch registry and progress endpoints backed by cache, with best-effort observability registration/cleanup.
• Dataset trend endpoint with chronological points, metrics, cohorts, usage summaries, arbitrary report-prefix discovery, and configurable scan cap.
• Companion UI package spec in docs/UI_PACKAGE_SPEC.md for the future optional padosoft/eval-harness-ui package.
• README updated with the Eval Harness banner, v1.2 feature coverage, endpoint docs, and companion UI link.

Compatibility

• All changes are additive for v1 consumers.
• ReportApiSchema::VERSION remains eval-harness.report-api.v1.
• New v1.2 endpoint payloads include per-payload schema discriminators.

Quality Gates

• Local release gate on d848e3f: composer validate --strict, PHPUnit OK (778 tests, 2165 assertions), PHPStan no errors, Pint passed.
• CI matrix green across PHP 8.3 / 8.4 / 8.5 x Laravel 12 / 13.

Merged PRs

• #40 report diff endpoint
• #41 adversarial manifest discovery
• #42 batch live registry and progress endpoints
• #43 dataset trend endpoint
• #44 macro PR into main

v1.1.0 — Macro 8: Enterprise Operations and Scalability Add-On

Builds on v1.0 (tag 1.0) with the v1.x enterprise operations and scalability add-on. All v1 contracts are preserved.

Highlights

Operational batch profiles

--batch-profile=ci|smoke|nightly with explicit-CLI-wins precedence and host-app overrides under eval-harness.batches.profiles.* (see config/eval-harness.php). Built-in profile defaults are conservative (CI runs do not exceed 4 in-flight samples by default).

Producer-side backpressure

--chunk-size, --rate-limit, --rate-window-seconds. RateLimitWindow uses a monotonic clock (hrtime) and amortized O(1) sliding-window math via head-offset compaction.

Progress checkpoints

--checkpoint-every with the optional BatchProgressReporter container binding (default NullBatchProgressReporter) plus an optional BatchTerminalProgressReporter sub-contract for terminal status (SUCCESS / FAILURE / EMPTY) with partial-wins tolerance on the failure path.

TTL controls

--result-ttl-seconds operator override with separate run/dispatch TTL math (run uses chunk-deadline-bounded windows; dispatch uses concurrency-keyed drain time with a fixed 60s per-drain-batch fallback decoupled from the wait-timeout config).

Operator guidance

Horizon supervisor sizing guidance in docs/HORIZON_BATCH_QUEUES.md (multi-producer caveat, chunk-size vs concurrency, rate-limit producer-side behavior).

Compatibility

• No hard Horizon dependency added. The package still uses sync queues and queue fakes in tests; Horizon is recommended but optional in host apps.
• No breaking changes to v1 CLI flags or report JSON shape. The new flags are additive.

Quality gates (merge commit 3e202ce)

• PHPUnit: OK (699 tests, 1956 assertions)
• PHPStan: [OK] No errors
• Pint: passed
• CI matrix green across PHP 8.3 / 8.4 / 8.5 × Laravel 12 / 13.

Sources

• Macro PR: #38 (merged 2026-05-05T18:05:07Z, merge commit 3e202ce)
• Sub-task PR: #37 (47 commits across 12 productive Copilot review rounds)
• Roadmap entry: docs/ROADMAP_IMPLEMENTATION_PLAN.md § Macro Task 8

🤖 Generated with Claude Code

1.0

padosoft/eval-harness 1.0

Roadmap is now fully complete for the v1.0 slice. This release includes:

• Stable core contracts for Metric, EvalReport, JSON report schema, batch execution, command surface, queue jobs, and report API resources.
• Parallel batch execution (SerialBatch, LazyParallelBatch) with --batch=serial and --batch=lazy-parallel.
• Adversarial lane with regression manifests, eval-harness:adversarial, --regression-gate, and --promote-failures.
• Read-only report API for list/show, cohorts, histograms, CSV export, and artifact download.
• Contract stability and migration docs for pre-1.0 users.

For package compatibility details and usage, see:

• docs/CONTRACT_STABILITY.md
• docs/MIGRATION_FROM_PRE_1_0.md
• docs/REPORT_API_CONTRACT.md
• README.md

v0.1.0 — Initial release

First public release of padosoft/eval-harness — a standalone Laravel evaluation framework for RAG / LLM applications.

Highlights

• Pluggable golden datasets — YAML-loaded strict-schema datasets with per-metric scoring; extend with custom metrics via Eval::extend(...).
• Three production-grade metrics — ExactMatchMetric (byte-equality), CosineEmbeddingMetric (laravel/ai SDK provider, configurable timeout via TimeoutNormalizer), LlmAsJudgeMetric (deterministic seed + strict-JSON validation).
• Two report formats — Markdown (human) + canonical JSON (machine-readable, byte-identity for diffing across runs).
• Artisan CLI — eval-harness:run <dataset> with strict failure handling.
• Standalone agnostic — zero coupling to AskMyDocs or any sister Padosoft package; works on any Laravel 12/13 project.
• CI matrix — PHP 8.3 / 8.4 / 8.5 × Laravel 12 / 13. 109 Unit + 3 Architecture tests, ~600 assertions. Pint + PHPStan level 6 clean.
• 🚀 AI vibe-coding pack included — full Padosoft .claude/ skills + rules + agents.

Installation

composer require padosoft/eval-harness

Roadmap (v0.2)

• Parallel batch evals via Laravel queues.
• Cohort metrics aggregating by metadata.tags.
• Histogram view + ROUGE-L + BERTScore + refusal-quality + citation-groundedness metrics.
• Adversarial harness (prompt injection / jailbreak datasets).
• Regression detection (compare against last N runs).
• Minimal HTML web dashboard.

Comparison

vs OpenAI Evals + LangSmith + Ragas + Promptfoo + DeepEval — see README "Comparison vs alternatives" for the side-by-side. Differentiators: native Laravel integration, deterministic seed throughout, the .claude/ pack ships in the box.

License: Apache-2.0.