Skip to content

proofrag v0.9.0

Choose a tag to compare

View all tags
@unshDee unshDee released this 04 Oct 09:20
v0.9.0
7c944e1
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

proofrag 0.9.0 adds an optional typed Jev judge and tightens the path from saved RAG predictions to a trustworthy CI result.

Added

  • proofrag evaluate --backend jev through the official TypeSafe API, with no extra dependency. Set TYPESAFE_API_KEY to use the pinned jev-1.13.0 model
  • Four ordered generation rubrics, normalized scores, raw probabilities, confidence, resolved model, usage and rubric fingerprints
  • Research notes and an outreach packet with a runnable demo, specific channels and editable announcement drafts

Reliability fixes

  • Full precision for native, DeepEval and Ragas score gates
  • Invalid scores, malformed JSON, duplicate keys, truncated or filtered completions and model drift invalidate evaluation runs
  • Full retrieved context in native judging, with evaluated text encoded as data
  • Failed comparisons no longer count as ties, and requested reports show incomplete runs
  • Validated adapter inputs, atomic JSONL writes, readable input errors and nested corpus ignore rules
  • CI uploads only fresh JSON and HTML artifacts, supports backend/model selection, installs locked dependencies and uses pinned Actions

Upgrade

Native and comparison judge fingerprints now use v3. DeepEval and Ragas use v2. Regenerate existing baselines with the same package version, backend and judge before gating new results.

uvx proofrag==0.9.0 demo --out scorecard.html

Jev scoring uses TYPESAFE_API_KEY and --backend jev. Generation and blind comparisons still use Anthropic or an OpenAI-compatible provider.

Verification and limits

188 tests pass locally, with clean Ruff, formatting, codespell and basedpyright checks. Exact 0.9.0 wheel and source builds, a fresh dependency-free install and both keyless demos pass. Offline SDK and external metric construction checks pass with refreshed dependencies.

Jev hosted RAG accuracy and calibration have not been benchmarked. Confidence is retained as model uncertainty, not verified correctness on your dataset.

Two distinct unpatched advisories remain in optional Ragas/DiskCache dependencies. The current text-only, uncached proofrag adapter does not invoke their affected paths. See SECURITY.md for the reviewed scope.

See research notes, changelog, and launch packet.

Assets 4
Loading