Code style review agent

[raineorshine]

First-pass code reviews

Investigate the feasibility of an AI-powered first-pass code review agent for em.

Not all code reviews are equal. Sometimes, they surface open questions and detailed discussions about how something should be implemented. Those open valuable discussions, and they demand deep knowledge and the ability to make good judgment calls.

Other code reviews are more mechanical – they enforce good standards and code style. My observation has been that this feedback tends to be pretty repetitive in PRs. The time it takes to find and review these kinds of changes adds up – especially over 9 years.

Though we could possibly automate the first kind of code review (and there is interesting work on this being done at the frontier of AI/LLM development), I think we're going to have much more initial success if we try to automate the second kind – especially as we're targeting 90%+ accuracy from the reviewer agent.

That would free you to focus on the important judgment-call feedback that needs you the most.

Originally posted by @fbmcipher in #3974 (comment)

[fbmcipher]

Experiment

• We have 9 years of PR review data.
• Let's see if it's possible to use that data to produce a rulebook codifying em's engineering principles.
• Then, see if it's viable for an AI agent to use that rulebook, along with an incoming PR diff, to produce a first-pass code review that surfaces these kinds of mechanical/repetitive issues.

Approach

Here's the general process I followed:

1. Scrape: Pull all 4,074 comments you've left on em PRs via the GitHub CLI.
2. Seed: Take a randomized sample of ~200 comments, along with the code hunk from the PR they're associated with. For each, extract the underlying coding principle being enforced (e.g. HYGIENE-001: remove debug logs before merging)
   • Prefix each rule with an ID and a domain-specific prefixes (like REDUX-, PANDACSS-, TEST-, REACT-, ARCH-, etc.)
   • This is useful for storage, and allows humans and AIs to quickly refer to a specific rule.
3. Classify: After a human review phase to ensure the initial results from the seed phase look good, run all 4,074 comments against the codebook. For each comment, classify it as: matching an existing rule, proposing a new rule, too specific to generalize, or not a rule (i.e. excerpts of conversation). New rules discovered during classification get added to the codebook.
4. Consolidate: Deduplication phase. Ensure that each rule in the codebook is a single, atomic, distinct rule.
5. Verify: In a separate thread, run the codebook against the full corpus of 4,074 comments to work out how many of them are represented in the final codebook.

From that JSON, we can deterministically produce a .md file to pass as custom instructions to an agent, along with a simple prompt:

You are a code review agent for a project called em. Your task is to catch mechanical, rule-based issues so the lead reviewer can focus on higher-level judgment call feedback.

Read the attached codebook, which contains coding derived from 9 years of review history on this project. Each rule has an ID (e.g. HYGIENE-001), a label, and a description of the violation pattern.

Then, review the attached PR diff. For each violation you find:
1. Cite the rule ID and label.
2. Quote the specific line(s) in the diff.
3. Suggest a fix – a code snippet if straightforward, or a high-level description if the change is more involved.

**IMPORTANT:** Only flag violations of rules explicitly listed in the codebook. Do not invent rules or identify issues that aren't listed in the codebook.

Result

• This process yielded a codebook with 210 distinct rules.
• Of the 4074 comments, 67.4% match to rules in the codebook.
  • 25.7% of the comments were conversational – praise, questions, design discussions, replies or mentoring.
  • The remaining 6.8% of the comments were one-off corrections that the AI couldn't generalize into a rule.
  • This indicates a high level of repetition within the corpus and validates the initial premise. In other words, nearly 7 out of 10 review comments are repetitive or codifiable enough to be covered by the codebook.
• The process identified these high-level categories for rules:
  • Architecture
  • Testing
  • App-specific (em domain rules)
  • Code style
  • Performance
  • React
  • Code strictness
  • Redux
  • TypeScript
  • Documentation
  • PandaCSS
  • Hygiene
  • Naming conventions

The next step is to validate the codebook against some PRs. Initial testing against my PR #3936 flagged 9 legitimate issues, 4 of which you hadn't actually caught – good start! – but I want to do some more thorough testing over the following week.

For the most accurate testing, we'd probably want to run against the next PR that comes through – something that's completely unrepresented in our dataset.

I've attached a few files to this comment in case you want to review:

• codebook-v2-final.json: raw JSON rules
• CODEBOOK.md: a Markdown version of the codebook we can pass to an AI agent as custom instructions
• initial-result.md: an initial code review produced by Claude Code given the prompt and codebook against my PR [Liminal UI] Sidebar header with animations #3936

Originally posted by @fbmcipher in #3974 (reply in thread)

[fbmcipher]

Notes following the discussion on Thursday:

• In April @fbmcipher mined Raine's prior comment history in an attempt to develop an authoritative code style "codebook" derived from Raine's years of code review on the project
• The codebook itself came out well, but actually applying this to a code review proved more difficult
• Early experiments with sub-agents were either too expensive per token or produced too many false positives to be useful
• At this point we put the code review project on pause to focus on other areas with easier gains
• @trevinhofmann's early suggestion:
  • Express the style guide as a numbered list, say ~100 items.
  • Each item: one plain sentence, plus an expandable paragraph beneath with good-code / bad-code examples.
  • ~10 rules each per sub-agent against a small diff (~400 lines)
  • New: A critique agent that re-examines each flag: "did it actually break the rule, and if so why?" – eliminates false positives
• This could work
• One concern: cost to model performance ratio.
  • We tried a similar segmentation (albeit without the critique agent) to this before (though instead of splitting into groups of ~10 rules we split the sub-agents to focus on different high-level categories of style guide issues, e.g. REDUX-, CSS-, REACT-, PERF-)
  • I found that running this with Opus leader and Sonnet sub-agents I accidentally spent ~$80! Luckily they were free tokens from a promotion, but need to watch out for costs spiraling as we introduce sub-agents into the mix
  • Tried with Haiku sub-agents but it's just not smart enough
  • Could potentially use ACP and the Claude Agent SDK to use subsidized subscription allowance instead of per-token billing but this might not be the case for very long
  • Maybe the new models will yield better results as a lot has changed in 4 months