Skip to content

Releases

v0.17.1: eval tables on six runs

Choose a tag to compare

@kaveraa kaveraa released this 13 Nov 20:10

Changed

  • Eval tables on at least six runs per case instead of three, since three
    did not separate 0.97 from 1.00: Fable 5.2 1.00 against 0.87 without the
    plugin, Opus 5.5 0.99 against 0.85, Sonnet 5.5 0.98 against 0.87, Haiku
    5.5 0.99 against 0.84. The notebook lists every grader that still fails
    with the plugin.

Full Changelog: v0.17.0...v0.17.1

v0.17.0: the cause of a bug comes first in the answer

Choose a tag to compare

@kaveraa kaveraa released this 13 Nov 18:54

Changed

  • The answer to a bug fix names the cause first. The Output rule now reads
    "what changed (a bug: its cause first), where". Fable 5.2 states the
    cause on shared-bug in 5 runs out of 5, against 1 out of 3, and Opus
    5.5 on csv-export in 5 out of 5, against 2 out of 3.
  • The ruleset ceiling goes from 630 to 640 estimated tokens: the clause
    takes it to 634 at its heaviest level.
  • Eval suite: the short-answer grader of date-picker accepts an
    answer that says the field was made required and can be made optional.
  • Eval suite, nine cases: Fable 5.2 1.00 against 0.88 without the plugin,
    Opus 5.5 1.00 against 0.84, Sonnet 5.5 0.97 against 0.86, Haiku 5.5 0.97
    against 0.84. Sonnet and Haiku move between 0.97 and 1.00 from one suite
    to the next under either wording; the notebook has both.

Full Changelog: v0.16.0...v0.17.0

v0.16.0: plain text and comment rules that hold on long answers

Choose a tag to compare

@kaveraa kaveraa released this 13 Nov 17:57

Changed

  • Two rules reworded after explain-retry showed them giving way. Plain
    text now says "long answers too" and names the range of numbers, where
    Sonnet 5.5 kept an en dash: 5 plain answers out of 5, against 1 out of 3.
    Comments now ask for "only a reason the code cannot show", "never a
    summary of a function or a line", "in doubt, none": Sonnet 5 files out
    of 5 and Haiku 4 out of 5 pass the judge, against 1 out of 3 and 0 out
    of 3.
  • The ruleset ceiling goes from 600 to 630 estimated tokens: the two lines
    take it to 628 at its heaviest level.
  • Eval suite, nine cases, with the reworded rules: Fable 5.2 0.99 against
    0.88 without the plugin, Opus 5.5 1.00 against 0.85, Sonnet 5.5 1.00
    against 0.86, Haiku 5.5 0.98 against 0.84.

Full Changelog: v0.15.8...v0.16.0

v0.15.8: an eval case for long text and comments

Choose a tag to compare

@kaveraa kaveraa released this 13 Nov 17:30

Added

  • A ninth eval case, explain-retry: a long explanation and comments to
    write, where a model reaches for long dashes, arrows and comment blocks.
    Without the plugin no model writes a plain answer or one-line comments,
    0 run out of 12 each. With it, 10 and 11 out of 12, and the explanation
    stays complete in every run. Two weak spots: Sonnet 5.5 keeps a long
    dash in 2 answers out of 3, and one comment per file often names the
    mechanism instead of the reason.

Changed

  • The eval tables and the eval card count nine cases: Fable 5.2 0.98
    against 0.88 without the plugin, Opus 5.5 0.99 against 0.85, Sonnet 5.5
    0.98 against 0.86, Haiku 5.5 0.97 against 0.84.

Full Changelog: v0.15.7...v0.15.8

v0.15.7: eval tables on eight cases

Choose a tag to compare

@kaveraa kaveraa released this 13 Nov 17:13

Changed

  • The eval tables and the eval card count eight cases: the seven measured
    with Opus as judge and zod-email, same day, same plugin code. Fable 5.2
    0.98 against 0.89 without the plugin, Opus 5.5 0.99 against 0.86, Sonnet
    5.5 0.99 against 0.87, Haiku 5.5 0.98 against 0.86.

Full Changelog: v0.15.6...v0.15.7

v0.15.6: an eval case for current practice

Choose a tag to compare

@kaveraa kaveraa released this 13 Nov 16:28

Added

  • An eighth eval case, zod-email, for the rule on current practice: the
    manifest declares zod 4, where z.email() replaces the chained
    z.string().email(). Without the plugin Sonnet 5.5 and Haiku 5.5 write
    the deprecated form in 3 runs out of 3; with it, the current one in 3 out
    of 3. Fable 5.2 and Opus 5.5 write the current form either way. With the
    versions line taken out, the plugin gets 2 out of 3 on both: the rule
    does most of it, the line closes the gap.

Full Changelog: v0.15.5...v0.15.6

v0.15.5: examples recorded again

Choose a tag to compare

@kaveraa kaveraa released this 13 Nov 16:19

Changed

  • The five examples are recorded again with crewcut 0.15.4 through
    ponytail's harness, unchanged: no crewcut: comment is left in them.
    The date picker run is now 399 lines against 10; the before and after
    card and its figures follow.

Full Changelog: v0.15.4...v0.15.5

v0.15.4: graders for the owner rules, Opus as judge

Choose a tag to compare

@kaveraa kaveraa released this 13 Nov 16:05

Added

  • Eval graders for the rules that had none: plain text in the answer and in
    the edited file, no "crewcut" in the code, comments on one line, comments
    that give a reason. Twenty-four graders over the seven cases, twenty of
    them regular expressions.

Changed

  • Opus 5.5 is the judge of the eval suite. Measured on four models, seven
    cases, three runs: Fable 5.2 0.98 against 0.87 without the plugin, Opus
    5.5 1.00 against 0.80, Sonnet 5.5 0.99 against 0.88, Haiku 5.5 0.98
    against 0.87. Without the plugin Opus writes comments that restate the
    code in 3 runs out of 6, and three models put a banned character in an
    answer.
  • The eval tables give the cost of the session alone. The figure published
    until now included the judge, which was most of it on Haiku.

Full Changelog: v0.15.3...v0.15.4

v0.15.3: new banner, terminal theme, four more cards

Choose a tag to compare

@kaveraa kaveraa released this 13 Nov 15:27

Changed

  • New banner and one theme for every README image: near-black, a
    fixed-width font throughout, red as the only bright colour, and a glowing
    cut line in place of the barber pole. The banner is now a PNG rendered
    from assets/banner.html, like the cards.
  • README: the install, the settings, the FAQ and the limitations are cards
    too. The two install commands and the settings table stay as text.

Full Changelog: v0.15.2...v0.15.3

v0.15.2: descriptions aligned with the benchmark

Choose a tag to compare

@kaveraa kaveraa released this 13 Nov 14:58

Changed

  • The plugin, marketplace and package descriptions give the figure of
    ponytail 5's benchmark, 58 % less code on Opus, in place of the best cell
    of the October harness, and say "every security run passed", which is
    what was measured. No change to the plugin.

Full Changelog: v0.15.1...v0.15.2