EvalForge v0.4.0 — comparable evidence across evaluators

EvalForge v0.4.0 makes existing evaluator results usable as traceable CI release evidence. A numerically better score can now be blocked when its required dataset or metric identity no longer matches the baseline.

New capabilities

• Offline Ragas import with explicit score columns and complete-row validation.
• DeepEval 3.8.1 and 4.2.2 imports with verified fixtures and separate names for metrics whose score semantics reversed across major versions.
• Opt-in dataset, producer, and metric-version comparison requirements.
• Deterministic SHA-256 input/policy digests and source revisions in JSON, JUnit, SARIF, and Markdown reports.
• GitHub Action job summaries on pass and failure, with stable failure exit codes.
• A complete CI cookbook, strict-comparison demonstration, and dated OSS evidence map.

Verification

240 tests pass locally on Python 3.9.6 with 89.78% branch-aware coverage. Ruff, mypy, package builds, and Twine checks pass. A fresh base-only wheel installation runs all three imports and gates without Ragas, DeepEval, database, or dashboard dependencies.

Upgrade PR and CI evidence · CI cookbook · Full changelog

Install

pip install --upgrade evalforge-ci==0.4.0

Python 3.9+ remains supported. Existing flat-summary policies retain their behavior unless strict comparison is enabled. Canonical numeric fields now reject booleans and numeric strings; overflowing deltas fail safely.

The project remains alpha. Fixtures use synthetic scores and establish format compatibility, not model quality or downstream adoption. Digests identify inputs and are not signatures. The release workflow attaches checked distributions and SHA-256 checksums and publishes the same distributions to PyPI through Trusted Publishing.

EvalForge v0.3.1 — promptfoo interoperability and trusted releases

Highlights

• Adds an explicit promptfoo results-schema-v3 adapter that converts upstream output into portable EvalForge evidence while preserving producer provenance.
• Adds upstream-format synthetic fixtures and compatibility coverage for unknown fields, missing provenance, non-finite values, inconsistent counts, and the CLI import path.
• Hardens the supply chain with CodeQL, commit-pinned Actions, single-build release artifacts, SHA-256 checksums, and PyPI Trusted Publishing through GitHub OIDC.

Compatibility

• Python 3.9+
• Distribution: evalforge-ci; import package and CLI: evalforge
• EvalForge artifact schema 1.0 and policy schema 1 are unchanged; no migration is required from v0.3.0.
• The promptfoo adapter accepts OutputFile documents using results schema 3 and is verified against promptfoo 0.122.2.

Known limitations

EvalForge remains alpha software. The promptfoo adapter intentionally keeps aggregate evaluation evidence only: prompts, responses, grading traces, arbitrary metadata, and unregistered named scores are excluded. Compatibility with other promptfoo versions is not claimed until covered by upstream-format fixtures. Stars indicate interest, not confirmed production adoption.

Install

python -m pip install "evalforge-ci==0.3.1"
evalforge --help

See the changelog and release checklist for details.

EvalForge v0.3.0 — reproducible AI quality gates

EvalForge v0.3.0 is an alpha release of the evaluator-neutral evidence and policy-gate layer for RAG applications and AI assistants.

Highlights

• Compare baseline and candidate runs with direction-aware metric verdicts.
• Preserve dataset fingerprints, full configuration snapshots, and metric-version metadata.
• Run stored-experiment release gates through the API, dashboard, or the new end-to-end evalforge check command.
• Emit portable JSON evidence plus JSON, JUnit, and SARIF reports for existing CI systems.
• Exercise a seeded, reproducible RAG release gate in GitHub Actions.
• Verify Plotly 7 dashboard compatibility and harden Python 3.9 development tooling.

Compatibility

• Python 3.9 or newer.
• Evaluation artifact schema 1 and gate policy schema 1 remain compatible.
• Reusable Action reference: jsdhwfmax/EvalForge@v0.3.0.

Verification

The release commit passed Python 3.9 and 3.12 CI, Docker build, 41 tests with 87.68% coverage, checked wheel/sdist builds, and a fresh wheel-installed quality gate. The release workflow independently builds the distributions, verifies their metadata and SHA-256 checksums, and attaches them below.

Known limitations

• The project remains pre-1.0 alpha software and the interoperability schema is intentionally small.
• Built-in deterministic metrics are transparent CI signals, not a replacement for semantic or human review.
• The API/dashboard do not provide authentication or tenant isolation; protect deployments with an authenticated gateway.
• PyPI publication is a separate step. Until its project page is publicly verified, use the attached wheel or install from source.

See CHANGELOG.md for the full history.

EvalForge v0.2.1 — Evidence integrity and release hardening

EvalForge v0.2.1 hardens the evaluator-neutral evidence path without changing artifact schema 1.0 or gate policy schema 1.

Highlights

• Reject non-finite metric values and policy thresholds.
• Reject duplicate policy check IDs.
• Fail delta checks when candidate and baseline units or metric directions differ.
• Isolate test databases so concurrent test sessions are reliable.
• Update maintained dependencies and pin third-party GitHub Actions to full commit SHAs.
• Attach checked wheel/source distributions and SHA-256 checksums to future GitHub Releases.
• Add a PEP 561 typing marker and a public release checklist.

Verification

• 34 tests passing
• 87.97% branch coverage
• Ruff and mypy passing
• Two complete pytest sessions passing concurrently
• Workflow files passing actionlint
• Fresh-environment wheel install and gate demo passing
• Wheel and source distribution passing twine check

Compatibility

Artifact schema 1.0 and policy schema 1 remain unchanged. Existing finite, semantically compatible evidence continues to work. Inputs containing NaN/infinity, duplicate check IDs, or incompatible baseline metadata now fail explicitly instead of producing ambiguous evidence.

EvalForge remains alpha software. The repository received 41 GitHub Stars in its first six days, which is an early-interest signal rather than a claim of production adoption.

EvalForge v0.2.0 — Portable evaluation evidence gates

EvalForge v0.2.0 introduces an evaluator-neutral evidence and policy layer for AI systems.

Highlights

• Versioned evaluation artifact and policy JSON Schemas
• Absolute thresholds and baseline-delta quality gates
• JSON, JUnit XML, and SARIF 2.1.0 reports
• CI-safe exit codes and a reusable composite GitHub Action
• A lightweight evalforge-ci base install with optional RAG platform dependencies
• Public governance, maintainer responsibilities, security response targets, and honest ecosystem success measures

The built-in zero-key RAG evaluator remains available as a reference producer. This release is alpha software: metric semantics must be validated by each producer, and independent adoption is still early.

See CHANGELOG.md and docs/INTEROPERABILITY.md for details.