Skip to content

v0.15.4: graders for the owner rules, Opus as judge

Choose a tag to compare

@kaveraa kaveraa released this 13 Nov 16:05
· 29 commits to main since this release

Added

  • Eval graders for the rules that had none: plain text in the answer and in
    the edited file, no "crewcut" in the code, comments on one line, comments
    that give a reason. Twenty-four graders over the seven cases, twenty of
    them regular expressions.

Changed

  • Opus 5.5 is the judge of the eval suite. Measured on four models, seven
    cases, three runs: Fable 5.2 0.98 against 0.87 without the plugin, Opus
    5.5 1.00 against 0.80, Sonnet 5.5 0.99 against 0.88, Haiku 5.5 0.98
    against 0.87. Without the plugin Opus writes comments that restate the
    code in 3 runs out of 6, and three models put a banned character in an
    answer.
  • The eval tables give the cost of the session alone. The figure published
    until now included the judge, which was most of it on Haiku.

Full Changelog: v0.15.3...v0.15.4