Skip to content

Deeds 0.5.0: judged on Clef, and measurably better on code it has never seen

Latest

Choose a tag to compare

@danielmiessler danielmiessler released this 13 Oct 21:41
· 1 commit to main since this release

A better judge, measured. Deeds now judges commits with Cloudflare's Clef model on Workers AI, in your own Cloudflare account. The question system behind it was rebuilt through an evaluation climb against labelled commits, and checked once at the end on 256 commits from eight public repositories that no part of the climb ever touched.

On 256 unseen commits 0.4.0 0.5.0
Overall accuracy 0.843 0.871
Caps 0.777 0.819
Fixes 0.878 0.901
Tends 0.873 0.894
False caps 8 12

Accuracy is balanced accuracy against labels two independent reviewers agreed on. 0.5.0 is better on all three deeds. It credits a few more caps where the reviewers saw none (12 of 256 commits, against 8), which is the trade we are working on next.

How it works. Code reads exact facts from the diff, Clef answers up to 44 narrow questions in 13 stages, each stage asked only when an earlier answer calls for it, and 112 written rules decide the cap, the fix and the tend. The model never makes the final call. The README map shows every stage and question.

Security. A commit's diff is redacted before it leaves your machine, and 0.5.0 also blanks long mixed-case strings shaped like keys. On the security test corpus, 0.5.0 recognises every security fix as a fix (28 of 28), including all four commits from issue #7. It also calls 5 of 38 upkeep commits fixes where the reviewers saw none, against 3 for 0.4.0.

Thanks to everyone whose pull requests landed in this release: #3, #4 and #6.

Upgrade: curl -fsSL https://raw.githubusercontent.com/danielmiessler/deeds/main/install.sh | sh, or update the deeds plugin in Claude Code. Deeds now needs a Cloudflare account ID and API token (CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_API_TOKEN); the README's Install section shows where to find them. Results cached by earlier versions are not reused; the first run on a repo judges it again.