Live dashboard → https://soubedev.github.io/posthog-impact/
An analysis of every pull request merged into PostHog/posthog from Jun 14 to Sep 12, 2026 (91 days, 15,087 PRs). It picks out the five most impactful engineers and explains why, with every score linked back to GitHub evidence.
| # | Engineer | Impact | Quality · Delivery · Leverage | Why | Growth opportunity |
|---|---|---|---|---|---|
| 1 | Tom Owers (@Gilbert09) | 88 | 70 · 100 · 99 | 2,498 PRs; 445 substantive reviews for 60 teammates; rubric 55/70; 0 reverts. Strength: Enumerates tables through the SQL editor's resolve_visible_table_names path, so access-restricted tables never appear in the catalog (#65503) | Safety: Add an optional per-deployment allowlist of servable key names on the mount so a compromised caller key cannot read every platform credential. |
| 2 | Andrew Maguire (@andrewm4894) | 87 | 70 · 100 · 94 | 431 PRs; 405 substantive reviews for 50 teammates; rubric 55/70; 0 reverts. Strength: One shared helper fixes retrieve, signals and state consistently, with a regression test for each endpoint (#62811) | Effectiveness: Add unit tests for scoutDisplayName slugs (bare, underscores, empty) and for _emit_bound_signal merging skill_name into extra. |
| 3 | Daniel RC (@danielcarletti) | 76 | 71 · 63 · 100 | 171 PRs; 545 substantive reviews for 18 teammates; rubric 56/70; 0 reverts. Strength: Postgres rendering uses sql.Identifier and sql.Literal with pre-canonicalized operators, keeping user values out of SQL text (#62773) | Safety: Catch WorkflowAlreadyStartedError in _start_non_billable_resume_workflow so an activity retry after a successful start completes instead of failing repeatedly. |
| 4 | Paul D'Ambra (@pauldambra) | 75 | 70 · 89 · 61 | 484 PRs; 166 substantive reviews for 41 teammates; rubric 55/70; 0 reverts. Strength: API contract documents every metric's semantics, null behaviour and proxy limitations, and routes through the authorized-source facade for access control (#89744) | Integration: Wait for green CI before merging: 85% of checked PRs merged with all checks green. |
| 5 | Julian Bez (@webjunkie) | 75 | 72 · 81 · 70 | 296 PRs; 229 substantive reviews for 37 teammates; rubric 56/70; 2 reverts. Strength: Descends only into database_operations and exempts concurrent index and VALIDATE CONSTRAINT work, keeping false positives low (#62857) | Integration: Wait for green CI before merging: 88% of checked PRs merged with all checks green. |
- Robustness (share of 120 weightings that keep them top 5): Gilbert09 100%, andrewm4894 100%, danielcarletti 69%, pauldambra 42%, webjunkie 61%.
- Coverage check: best engineer outside the shortlist (arthurdedeus) could reach at most 71.8 vs #5's 74.8 → passes.
- Scorer calibration: one shared PR was scored blind in all 6 batches → A 14–15/20, C 10, F 8 every time; 0 of 60 AI records broke the rubric's rules.
- Hand check: three AI findings were verified against the diffs. Two are real bugs:
#comment stripping would erase Postgres#>operators in #62857, andWorkflowAlreadyStartedErrorgoes unhandled on retry in #74967. The third (#79462, no per-deployment key allowlist) is real but a documented design tradeoff in that PR.
How to read it:
- Clear top 2: Tom Owers and Andrew Maguire are in the top 5 under every weighting tested. Both combine top-1% delivery with heavy reviewing.
- #3 fairly solid: Daniel RC is carried by the most review leverage of anyone.
- #4–#6 within 0.4 points: Paul D'Ambra, Julian Bez and Alessandro Pogliaghi (#6, highest Quality of all finalists) are effectively a tie. The weights decide the order.
Impact = shipping meaningful changes that work, safely and at scale, plus making teammates' work better.
| Component | Default weight | What it measures | Coverage |
|---|---|---|---|
| Quality | 40% | A per-PR rubric: A Effectiveness (20), B Practicality, C Safety, D Reliability, E Integration, F Scalability (10 each) | AI code reading on 5 PRs per finalist, plus GitHub data on all PRs |
| Delivery | 35% | Meaningful change shipped: merged PRs weighted by type and √size; reverted work counts 0 | All merged PRs |
| Team leverage | 25% | Substantive human reviews of other people's PRs: approvals, change requests, inline comments | All reviews on merged PRs |
Why these weights?
- Agents make volume cheap. 83% of credited PRs in the window declare agent involvement, and one engineer merged 2,498 PRs, so raw output can't be the main signal. Quality weighs most.
- Reviews multiply other people's work, but approvals are cheap. Leverage weighs least.
- You can disagree. The dashboard has weight sliders and shows how robust each top-5 spot is across 120 weightings.
- Gather. Run one GitHub GraphQL search per UTC day, so the 1,000-result cap never bites, and fetch all 15,087 merged PRs.
- Each PR comes with its reviews, assignees and merge commit.
- The fetched count matches GitHub's search total exactly.
- Credit people, not bots.
- PRs from human accounts go to the author.
- PRs opened by PostHog's coding agent (the
posthogapp) go to the human assignee, who is the person responsible for the PR. - Unassigned agent PRs are dropped. PostHog's own PR template says "fully autonomous → leave unassigned".
- Bot PRs and bot reviews (stamphog auto-approvals, Greptile, Copilot, Codex…) are excluded.
- Score every PR on GitHub data. Practicality (B), Reliability (D) and Integration (E) come from metadata, along with each PR's significance for Delivery and each review's leverage units.
- Shortlist 12 finalists.
- The top 10 by best-case Impact, i.e. Delivery and Leverage with perfect Quality assumed.
- Plus the 2 strongest reviewers not already in.
- Read code with AI. Five PRs per finalist (the 3 most significant from distinct areas, plus 2 seeded-random) go into anonymized packets containing the description, commits, file list and a prioritized diff.
- Six independent Claude subagents scored them against
data/stage2/rubric.md, each on a mixed batch. - Every batch also scored one shared calibration PR to measure how consistent the scorers are.
- Six independent Claude subagents scored them against
- Combine and check. Compute Impact, then:
- validate every AI record against the rubric's rules;
- check the shortlist didn't miss anyone who could have made the top 5;
- measure robustness across weightings.
| Dim | Score | How | Why this way |
|---|---|---|---|
| A Effectiveness | 0–20 | AI: do the description's claims match the diff (0–10), plus how much capability changes before → after (0–10) | Needs code reading |
| B Practicality | 0–10 | 10 × share of PRs in the same size decile that took longer from open to merge | Speed relative to complexity, compared with peers |
| C Safety | 0–10 | AI: start at 10 and deduct for each cited risk (injection, missing team scoping, auth bypass, secrets, risky migrations…) | Needs code reading; every deduction must name a file |
| D Reliability | 0–10 | 0 if reverted; 7 if a fix PR referencing it merged within 14 days; otherwise 10 | Adjusted. Deploys batch many commits, so a deploy outcome can't be pinned on one PR. Reverts are the objective change-failure signal |
| E Integration | 0–10 | CI state on the PR's last commit at merge: success 10, pending 7, failing 3 | Adjusted. CI on master after merge was spot-checked and showed pending checks and flaky E2E failures unrelated to the PR. Merged-with-failing-checks does vary (8% of finalists' PRs) |
| F Scalability | 0–10 | AI: efficient at PostHog scale, reuses existing patterns, proportionate tests | Needs code reading |
- PR significance =
type_weight × √(min(lines, 2000) / 100).- Type weights: feat, fix and perf 1.0; refactor 0.7; test, chore, ci and build 0.5; docs 0.4; style and deps 0.3.
- A reverted PR scores 0.
- Delivery =
100 × √(Σ significance ÷ the 99th-percentile engineer's), capped at 100.- Anchoring to the 99th percentile instead of the maximum stops one extreme outlier from compressing everyone else.
- Leverage: for each PR a person reviewed (not their own, and not one credited to them), award
1 + [requested changes] + min(1, inline comments ÷ 5), halved for PRs under 10 lines.- A lone comment-only review with no inline comments earns nothing.
Leverage = 100 × √(units ÷ 99th-percentile engineer's), capped at 100.
- Quality =
100 − 2 × (70 − rubric points), where rubric points = mean A + mean B + … + mean F. - Impact =
0.40 · Quality + 0.35 · Delivery + 0.25 · Leverage.
- Volume is cheap. 83% of credited PRs declare agent involvement in the PR template's "Autonomy" line. The busiest engineer merged 2,498 PRs, 18% of everything credited. Raw PR and line counts were rejected as the headline metric.
- About half of all review objects are bots (48%): auto-approvals and AI reviewers. Only human reviews count.
- Reverts are rare: 24 effective reverts in 91 days. That makes them an objective reliability flag, not a ranking driver.
- CI on master after merge was spot-checked and rejected: about a quarter of merge commits showed pending checks and ~10% failures, and the failures examined were flaky E2E runs unrelated to the PR.
- Deploy success was rejected: PostHog's deploys batch many PRs, so outcomes can't be attributed to one.
- Every number on the dashboard has a tooltip with its formula and inputs.
- Each finalist has GitHub search links for their authored PRs, the reviews they gave and agent PRs assigned to them.
- What the AI saw:
data/stage2/packets/(one Markdown file per PR). - What it concluded:
data/stage2/batch-*.json: scores, claims checked against the diff, cited risks, rationale, confidence. - Why each PR was sampled:
data/stage2/sample.json. - Raw per-PR and per-engineer metrics:
data/prs.csvanddata/engineers.csv. The engineers file has raw activity plus Delivery and Leverage only; Quality exists only for code-reviewed finalists.
- GitHub only. Incidents, customer outcomes, design work, mentoring and on-call aren't visible.
- Partial review coverage. Reviews count only on merged PRs, and PR conversation comments aren't attributed (inline review comments are).
- Small AI samples. Each finalist's AI scores rest on 5 PRs, and large diffs are truncated. The dashboard shows confidence and standard error, and the shared calibration PR measures scorer spread.
- Cycle time isn't pure author speed. It includes review wait and the merge queue.
- Heuristic follow-up detection. A follow-up fix is a fix PR that references a PR merged within 14 days.
- Not a performance review. The dashboard deliberately names only finalists and frames feedback constructively.
Requires Python 3.9+ (standard library only) and an authenticated gh CLI.
python3 scripts/fetch.py prs # ~15 min, ~650 GraphQL points: all merged PRs → raw/days/*.json
python3 scripts/analyze.py # stage 1: credit, B/D, delivery, leverage, shortlist → data/prs.csv, engineers.csv, stage1.json
python3 scripts/fetch.py ci # CI state on the last commit of each finalist PR → raw/ci.json
python3 scripts/stage2_prepare.py # sample + anonymized packets + batches → data/stage2/
# AI scoring: 6 Claude Code subagents, each given rubric.md plus its batch's packets, wrote data/stage2/batch-0N.json
python3 scripts/build_dashboard.py # combine, validate, calibrate, robustness/coverage → data/dashboard.json
python3 -m http.server # open http://localhost:8000- Score more PRs per finalist, and every eligible engineer, with a cheaper model to shrink sampling noise. Add a second scorer per PR to measure agreement.
- Pull in production signals: PostHog error tracking, incidents linked to PRs, feature-flag rollouts and product usage of shipped features.
- Credit PR conversation comments and reviews on unmerged PRs.
- Get feedback from the engineering leader on the weights, then re-run quarterly to track trends instead of a single snapshot.