Skip to content

Releases

v0.1.0

Choose a tag to compare

@fabian-lu fabian-lu released this 22 Oct 19:04

The first feature release since the initial upload, ahead of the EMNLP 2026 System Demonstrations.

pip install "cafe-core[all]"     # plus R with the `ordinal` and `lme4` packages

New

  • Per-factor tests for the ordinal and binary models. The CLMM and the logistic GLMM now report a Type-II likelihood-ratio test per factor and interaction, answering "does this factor matter at all?" in one line, next to the per-level coefficients.
  • The linear layer accounts for question difficulty. Its F-tests are now blocked by question (verdict ~ question + factors), so differences between questions no longer inflate the residual. Its variance components are fitted with an optimizer fallback, which fixes a spurious "between-question variance ≈ 0".
  • More from the R models: the random-intercept variance and the latent ICC for the CLMM and GLMM, and the CLMM's thresholds in report().
  • Energy tracking (opt-in): declare Wh per call and get totals and CO₂e.
  • Several rubrics per study (dimensions).
  • Fractional designs now use minimum-aberration generators.

Fixes

Most of these come from an external code review.

  • Judging: reads Markdown, decimal-comma and "out of" grade markers; rejects an ambiguous GRADE line instead of taking its first number; asks for JSON in structured mode; rejects unrenderable templates and rubrics with fewer than two levels.
  • Statistics: p-values are withheld when R's optimizer did not converge (the GLMM is first refitted with bobyqa); exact fits are scored instead of reported as NaN; pareto() counts each answer once and treats unreported cost as missing, not zero; coefficients go to the longest matching factor name; a : inside a factor level no longer reads as an interaction.
  • Core: reserved factor names are rejected; a measured cost of 0.0 is kept in estimate(); resuming a checkpoint keeps only the current answers' verdicts; ctx.add_cost is attributed per step.

Upgrading from 0.0.1

This is a minor release rather than a patch (0.0.2) because it adds features and changes some results and some behaviour. No function was removed or renamed, so existing code keeps running, but:

  • Different numbers in the linear layer. Because of the blocking by question, F and partial η² are larger than in 0.0.1 (question difficulty is no longer counted as noise). On the paper's HotpotQA study the conclusions are unchanged.
  • Stricter checks. Some inputs that 0.0.1 accepted, and then analysed wrongly, now raise an error or are reported as unparseable:
    • factor names that collide with CAFE's own result columns (question, output, verdict, rep, cost_usd, …);
    • rubrics with fewer than two levels;
    • judge templates that can't be rendered;
    • judge replies with a fractional grade, or with an ambiguous GRADE line (such as GRADE: 3 or 4).
  • p-values can be missing. The CLMM and GLMM now withhold their p-values when R's optimizer did not converge, instead of printing unreliable ones. The report says why.

v0.0.1: Initial Release on PyPi

Choose a tag to compare

@fabian-lu fabian-lu released this 20 Aug 13:39
add pypi metadata for pip installs