Skip to content

v0.6.0 — Reliable evidence and resilient evaluation

Latest
Latest

Choose a tag to compare

View all tags
@jsdhwfmax jsdhwfmax released this 02 Oct 10:49
v0.6.0
54c4667
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

EvalForge 0.6.0 strengthens evaluation-based release decisions and makes the built-in RAG runner more resilient.

Changes

  • Gates fail safely on invalid numeric evidence; malformed numeric API requests return 422.
  • Comparisons require matching dataset and metric-version identities. Incompatible runs cannot claim improvements or regressions.
  • Batch execution returns persisted failures alongside successful runs, continues independent configurations, and never automatically retries failures. The dashboard clearly shows partial failures.
  • CLI imports protect source exports, gates clear selected old reports, and promptfoo/DeepEval means preserve threshold boundaries.
  • Production API/dashboard access keys, non-root containers, a deployment guide, reusable BM25 statistics, and duplicate-ID handling are included.

Verification

578 tests passed on both Python 3.9 and 3.12; coverage 93.87% and 93.96%. CodeQL, Docker integration, package validation, and isolated wheel checks passed.

Upgrade

pip install -U evalforge-ci==0.6.0

Use evalforge-ci[rag,dashboard]==0.6.0 for the optional API/dashboard. Python 3.9+, artifact 1.0, policy 1, and deterministic-v2 remain supported. No database migration is needed. Production deployments must configure EVALFORGE_ACCESS_KEY. Inspect each experiment status in batch HTTP 200 responses; rerun baselines with missing or incompatible metric identities.

Deployment guide · Changelog · Implementation PR

The release workflow publishes the same verified wheel/sdist pair to GitHub and PyPI and attaches SHA256SUMS.

Full changes: v0.5.0...v0.6.0

Assets 5
Loading