Skip to content

v0.2.0

Choose a tag to compare

View all tags
@ssf0409 ssf0409 released this 30 May 03:07
· 77 commits to main since this release
v0.2.0
a045136
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

First feature release since the docs-only v0.1.1. Published to PyPI: pip install tracelens==0.2.0.

Highlights

  • Human-eval calibration loop — tracelens sample selects trials for human review (diverse / boundary / failures / random) and emits a self-contained worksheet; tracelens reconcile (alias of calibrate) pairs grader vs. human per row (carries trial_id, so no separate results file and multi-run trials stay distinct). Backed by sample_for_review() + CalibrationAnalyzer.analyze_worksheet().
  • Infra-noise differentiator — DecisionSpec.InfraConfig, TrialStatus.INFRA_ERROR + InfraError, RegressionDetector.compare_with_specs() (3pp noise band), infra metrics in reports, and a flagship benchmark pack reproducing Anthropic's infra-noise finding.
  • Adapters / graders / examples — HTTPAPIAdapter, contract graders, and four runnable examples including human_eval_calibration.py.
  • Onboarding docs — human-eval guide, baseline-regression tutorial, evaluation-recipes.

Breaking (0.x)

  • Removed WorkflowTask/WorkflowRunner/WorkflowAdapter and LiteLLMProvider; create_provider() supports only "in-memory" (subclass LLMProvider for real vendors).

Full notes: CHANGELOG.md

Assets 2
Loading