Skip to content

Add metric design walkthrough page - #25

Merged
MaxGhenis merged 1 commit into
mainfrom
metric-options-medicaid
May 17, 2026
Merged

Add metric design walkthrough page#25
MaxGhenis merged 1 commit into
mainfrom
metric-options-medicaid

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Summary

Adds a standalone HTML thinking-doc at `/metric-options.html` that walks through scoring options for the multi-output benchmark, step by step on a stylized example (3 households × 5 outputs).

The walkthrough builds up:

  • A continuous row score with no $1,000 floor: `max(0, 1 - |pred-ref| / |ref|)` and `ref=0 → 1 iff pred=0`. The `max(0, …)` already caps the score at 0 when the error exceeds the reference, so the floor wasn't doing extra work. Booleans (`ref ∈ {0, 1}`) drop out at 0 or 1 naturally — no special branch needed.
  • Equal weights, per-household impact, 30% floor (current), global variable weights (recommended), two-score breakdown (amount + participation), five ways to combine A and P (arithmetic, geometric, harmonic / F1, min, weighted), budget-weighted, net-income reconstruction, and a brief "other variants we considered" section.
  • Booleans like medicaid_eligible stay in the same single-bucket scheme: the LLM is graded only on the eligibility flag (binary accuracy), and PE's paired `medicaid_value` supplies `|ref|` for the weighting schemes without the LLM ever having to predict the per-capita value.

Numbers are computed inline from one JS source of truth, so all 19 tables stay consistent.

Recommendation surfaced in the closing card: ship global variable weights as the headline, amount + participation as the diagnostic companion, harmonic mean (F1) of A and P only if a single combined column is non-negotiable.

This is a thinking doc, not a published metric. Follow-up PRs will implement the recommended metric in `analysis.py` and update the paper / app prose to match.

Test plan

  • Page loads at `/metric-options.html` with no console errors.
  • All 11 leaderboard rows render with consistent numbers across worked-example tables (e.g. amount accuracy: Tax 73.3%, Benefit 64.8%; participation: Tax 66.7%, Benefit 93.3%; global variable weights: Tax 65.3%, Benefit 82.1%).
  • Scroll behavior is 1:1 (no header jank from the prior layout).
  • Mobile layout collapses cleanly.

Standalone HTML exploration at /metric-options.html. Twelve-step
walkthrough of scoring options for the multi-output benchmark,
illustrated on a stylized 3-household × 5-output example:

- Row score (continuous, no $1,000 floor; ref=0 special-cased to binary)
- Equal weights, per-household impact, 30% floor, global variable
  weights (recommended), two-score breakdown (amount + participation),
  five combine functions, budget-weighted, net-income reconstruction,
  and other variants we considered.
- Booleans like medicaid_eligible are scored on accuracy of the flag
  itself; PE's paired medicaid_value supplies |ref| for the weighting
  schemes without the LLM having to predict it.

Side-by-side leaderboard for the two stylized models under every
metric. All numbers computed inline from one JS source of truth.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented May 17, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
policybench-site Ready Ready Preview, Comment May 17, 2026 2:15pm

Request Review

@MaxGhenis
MaxGhenis merged commit 3104430 into main May 17, 2026
4 checks passed
@MaxGhenis
MaxGhenis deleted the metric-options-medicaid branch May 17, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant