Repository navigation
Releases
Releases · kaveraa/crewcut
v0.17.1: eval tables on six runs
Changed
- Eval tables on at least six runs per case instead of three, since three
did not separate 0.97 from 1.00: Fable 5.2 1.00 against 0.87 without the
plugin, Opus 5.5 0.99 against 0.85, Sonnet 5.5 0.98 against 0.87, Haiku
5.5 0.99 against 0.84. The notebook lists every grader that still fails
with the plugin.
Full Changelog: v0.17.0...v0.17.1
v0.17.0: the cause of a bug comes first in the answer
Changed
- The answer to a bug fix names the cause first. The Output rule now reads
"what changed (a bug: its cause first), where". Fable 5.2 states the
cause onshared-bugin 5 runs out of 5, against 1 out of 3, and Opus
5.5 oncsv-exportin 5 out of 5, against 2 out of 3. - The ruleset ceiling goes from 630 to 640 estimated tokens: the clause
takes it to 634 at its heaviest level. - Eval suite: the
short-answergrader ofdate-pickeraccepts an
answer that says the field was made required and can be made optional. - Eval suite, nine cases: Fable 5.2 1.00 against 0.88 without the plugin,
Opus 5.5 1.00 against 0.84, Sonnet 5.5 0.97 against 0.86, Haiku 5.5 0.97
against 0.84. Sonnet and Haiku move between 0.97 and 1.00 from one suite
to the next under either wording; the notebook has both.
Full Changelog: v0.16.0...v0.17.0
v0.16.0: plain text and comment rules that hold on long answers
Changed
- Two rules reworded after
explain-retryshowed them giving way. Plain
text now says "long answers too" and names the range of numbers, where
Sonnet 5.5 kept an en dash: 5 plain answers out of 5, against 1 out of 3.
Comments now ask for "only a reason the code cannot show", "never a
summary of a function or a line", "in doubt, none": Sonnet 5 files out
of 5 and Haiku 4 out of 5 pass the judge, against 1 out of 3 and 0 out
of 3. - The ruleset ceiling goes from 600 to 630 estimated tokens: the two lines
take it to 628 at its heaviest level. - Eval suite, nine cases, with the reworded rules: Fable 5.2 0.99 against
0.88 without the plugin, Opus 5.5 1.00 against 0.85, Sonnet 5.5 1.00
against 0.86, Haiku 5.5 0.98 against 0.84.
Full Changelog: v0.15.8...v0.16.0
v0.15.8: an eval case for long text and comments
Added
- A ninth eval case,
explain-retry: a long explanation and comments to
write, where a model reaches for long dashes, arrows and comment blocks.
Without the plugin no model writes a plain answer or one-line comments,
0 run out of 12 each. With it, 10 and 11 out of 12, and the explanation
stays complete in every run. Two weak spots: Sonnet 5.5 keeps a long
dash in 2 answers out of 3, and one comment per file often names the
mechanism instead of the reason.
Changed
- The eval tables and the eval card count nine cases: Fable 5.2 0.98
against 0.88 without the plugin, Opus 5.5 0.99 against 0.85, Sonnet 5.5
0.98 against 0.86, Haiku 5.5 0.97 against 0.84.
Full Changelog: v0.15.7...v0.15.8
v0.15.7: eval tables on eight cases
Changed
- The eval tables and the eval card count eight cases: the seven measured
with Opus as judge andzod-email, same day, same plugin code. Fable 5.2
0.98 against 0.89 without the plugin, Opus 5.5 0.99 against 0.86, Sonnet
5.5 0.99 against 0.87, Haiku 5.5 0.98 against 0.86.
Full Changelog: v0.15.6...v0.15.7
v0.15.6: an eval case for current practice
Added
- An eighth eval case,
zod-email, for the rule on current practice: the
manifest declares zod 4, wherez.email()replaces the chained
z.string().email(). Without the plugin Sonnet 5.5 and Haiku 5.5 write
the deprecated form in 3 runs out of 3; with it, the current one in 3 out
of 3. Fable 5.2 and Opus 5.5 write the current form either way. With the
versions line taken out, the plugin gets 2 out of 3 on both: the rule
does most of it, the line closes the gap.
Full Changelog: v0.15.5...v0.15.6
v0.15.5: examples recorded again
Changed
- The five examples are recorded again with crewcut 0.15.4 through
ponytail's harness, unchanged: nocrewcut:comment is left in them.
The date picker run is now 399 lines against 10; the before and after
card and its figures follow.
Full Changelog: v0.15.4...v0.15.5
v0.15.4: graders for the owner rules, Opus as judge
Added
- Eval graders for the rules that had none: plain text in the answer and in
the edited file, no "crewcut" in the code, comments on one line, comments
that give a reason. Twenty-four graders over the seven cases, twenty of
them regular expressions.
Changed
- Opus 5.5 is the judge of the eval suite. Measured on four models, seven
cases, three runs: Fable 5.2 0.98 against 0.87 without the plugin, Opus
5.5 1.00 against 0.80, Sonnet 5.5 0.99 against 0.88, Haiku 5.5 0.98
against 0.87. Without the plugin Opus writes comments that restate the
code in 3 runs out of 6, and three models put a banned character in an
answer. - The eval tables give the cost of the session alone. The figure published
until now included the judge, which was most of it on Haiku.
Full Changelog: v0.15.3...v0.15.4
v0.15.3: new banner, terminal theme, four more cards
Changed
- New banner and one theme for every README image: near-black, a
fixed-width font throughout, red as the only bright colour, and a glowing
cut line in place of the barber pole. The banner is now a PNG rendered
fromassets/banner.html, like the cards. - README: the install, the settings, the FAQ and the limitations are cards
too. The two install commands and the settings table stay as text.
Full Changelog: v0.15.2...v0.15.3
v0.15.2: descriptions aligned with the benchmark
Changed
- The plugin, marketplace and package descriptions give the figure of
ponytail 5's benchmark, 58 % less code on Opus, in place of the best cell
of the October harness, and say "every security run passed", which is
what was measured. No change to the plugin.
Full Changelog: v0.15.1...v0.15.2