Repository navigation
Releases
Releases · fabian-lu/Cafe
v0.1.0
The first feature release since the initial upload, ahead of the EMNLP 2026 System Demonstrations.
pip install "cafe-core[all]" # plus R with the `ordinal` and `lme4` packagesNew
- Per-factor tests for the ordinal and binary models. The CLMM and the logistic GLMM now report a Type-II likelihood-ratio test per factor and interaction, answering "does this factor matter at all?" in one line, next to the per-level coefficients.
- The linear layer accounts for question difficulty. Its F-tests are now blocked by question (
verdict ~ question + factors), so differences between questions no longer inflate the residual. Its variance components are fitted with an optimizer fallback, which fixes a spurious "between-question variance ≈ 0". - More from the R models: the random-intercept variance and the latent ICC for the CLMM and GLMM, and the CLMM's thresholds in
report(). - Energy tracking (opt-in): declare Wh per call and get totals and CO₂e.
- Several rubrics per study (dimensions).
- Fractional designs now use minimum-aberration generators.
Fixes
Most of these come from an external code review.
- Judging: reads Markdown, decimal-comma and "out of" grade markers; rejects an ambiguous GRADE line instead of taking its first number; asks for JSON in structured mode; rejects unrenderable templates and rubrics with fewer than two levels.
- Statistics: p-values are withheld when R's optimizer did not converge (the GLMM is first refitted with bobyqa); exact fits are scored instead of reported as NaN;
pareto()counts each answer once and treats unreported cost as missing, not zero; coefficients go to the longest matching factor name; a:inside a factor level no longer reads as an interaction. - Core: reserved factor names are rejected; a measured cost of 0.0 is kept in
estimate(); resuming a checkpoint keeps only the current answers' verdicts;ctx.add_costis attributed per step.
Upgrading from 0.0.1
This is a minor release rather than a patch (0.0.2) because it adds features and changes some results and some behaviour. No function was removed or renamed, so existing code keeps running, but:
- Different numbers in the linear layer. Because of the blocking by question, F and partial η² are larger than in 0.0.1 (question difficulty is no longer counted as noise). On the paper's HotpotQA study the conclusions are unchanged.
- Stricter checks. Some inputs that 0.0.1 accepted, and then analysed wrongly, now raise an error or are reported as unparseable:
- factor names that collide with CAFE's own result columns (
question,output,verdict,rep,cost_usd, …); - rubrics with fewer than two levels;
- judge templates that can't be rendered;
- judge replies with a fractional grade, or with an ambiguous GRADE line (such as
GRADE: 3 or 4).
- factor names that collide with CAFE's own result columns (
- p-values can be missing. The CLMM and GLMM now withhold their p-values when R's optimizer did not converge, instead of printing unreliable ones. The report says why.
v0.0.1: Initial Release on PyPi
add pypi metadata for pip installs