Skip to content

Calibration

Jump to bottom
Cindy Zhang edited this page Aug 24, 2026 · 2 revisions

Calibrating the loop to your taste

Critic-Rules is calibrated to one maintainer: their verdict policy, their ~50-word review as the length anchor, their rulings on component-vs-hook and consolidation. Some of that is Astryx-wide; some is personal.

Edit the kit, not the wiki. ~/astryx/review-loop-kit/review-critic.md is the file that runs; Critic Rules mirrors it.

Do not hand-write your own rules. Nobody has reliable access to their own taste in prose form. Mine it from what you already wrote.

Step 1 — have your agent mine your review history

Give it this prompt verbatim.

You are calibrating a review agent to my taste. My GitHub handle is <HANDLE>. Work read-only; post nothing.

Collect. Pull every review and review comment I have written on facebook/astryx:

gh api repos/facebook/astryx/pulls/<N>/reviews  --jq '.[]|select(.user.login=="<HANDLE>")|{state,body,submitted_at}'
gh api repos/facebook/astryx/pulls/<N>/comments --jq '.[]|select(.user.login=="<HANDLE>")|{path,line,body,created_at}'

Take the most recent 40–60 reviews; older ones reflect a system that has changed. Note each PR's author, size and area.

Measure, do not impressionize. Report actual numbers:

  • median and max WORD count of my summaries by verdict, and of my inlines
  • approve vs request-changes rate, split by whether the author is in .github/ENGOWNERS / .github/DESIGNOWNERS or an outside contributor
  • distinct findings per review (median, max)
  • how often I use a code block, a file:line pointer, a closing question
  • my opening move: thank, name the hard part, or state the verdict first?
  • hedged vs flat phrasing — count "ideally", "probably", "might", "hmm" against imperatives

Then find the rules, each backed by evidence. Quote 2–3 of my real comments per rule:

  • What do I consistently flag that a generic reviewer would not?
  • What do I consistently ignore that a generic reviewer would flag? This is the most valuable one and the hardest — look for findings plainly visible in the diff that I said nothing about.
  • What makes me block versus merge-and-follow-up?
  • When I disagree with a design, do I rule, prefer, or ask?
  • Where do I defer to someone else instead of deciding?

Sanity-check against the diffs. For five reviews, read the PR diff and ask what I could have said and didn't. Silence is data.

Output a numbered rule list in the style of Critic-Rules, each rule ≤4 lines, each quoting me. Mark anything inferred from fewer than three examples [thin evidence]. Do not invent rules to fill gaps.

Step 2 — replace only the personal half

Keep (Astryx-wide, not personal): R1b order of consequence · R5 state the problem at the level it exists · R6 never let one instance justify a system change · R13 never charge a contributor for inherited debt · R14 confidence gate · R15 layout needs a matrix.

Replace with yours: R2/R2b length and register — your measured word counts, your own review as the anchor · R1 verdict policy · R7/R8/R9 how hard you push consolidation · R10 strictness about half-finished APIs · R12 voice.

Step 3 — verify, do not trust

The step people skip, and the only one that proves anything.

Pick 3 PRs I reviewed that are not in the set you mined. Run the loop on each without looking at my review. Then fetch mine and report, per PR: did you match my verdict · which of my findings did you get · which did you miss · what did you say that I did not · word count yours vs mine. Then name the one rule that would close the biggest gap — and change only that one.

Repeat until the misses stop being about taste and start being about the code. Three rounds took one calibration from a 185-word blocking review to a 47-word approve that matched the maintainer's.

Step 4 — keep it fed

Every correction becomes a numbered rule in the kit's review-critic.md, quoting what was actually said — the wiki's Critic Rules is a mirror of that file, so a correction written here is lost at the next sync. Rules accumulate in the critic; the brief stays a description of the job, not a changelog.

Step 3 is the same discipline the loop now runs on itself: the audited numbers for a real night — 17 posted reviews, 1 false block — are in the 2026-08-23 retrospective.

If several people share the loop, keep one critic file each and point your spawn prompt at yours. A merged taste file is worse than either one alone.

Clone this wiki locally