Add metric design walkthrough page - #25
Merged
Merged
Conversation
Standalone HTML exploration at /metric-options.html. Twelve-step walkthrough of scoring options for the multi-output benchmark, illustrated on a stylized 3-household × 5-output example: - Row score (continuous, no $1,000 floor; ref=0 special-cased to binary) - Equal weights, per-household impact, 30% floor, global variable weights (recommended), two-score breakdown (amount + participation), five combine functions, budget-weighted, net-income reconstruction, and other variants we considered. - Booleans like medicaid_eligible are scored on accuracy of the flag itself; PE's paired medicaid_value supplies |ref| for the weighting schemes without the LLM having to predict it. Side-by-side leaderboard for the two stylized models under every metric. All numbers computed inline from one JS source of truth. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a standalone HTML thinking-doc at `/metric-options.html` that walks through scoring options for the multi-output benchmark, step by step on a stylized example (3 households × 5 outputs).
The walkthrough builds up:
Numbers are computed inline from one JS source of truth, so all 19 tables stay consistent.
Recommendation surfaced in the closing card: ship global variable weights as the headline, amount + participation as the diagnostic companion, harmonic mean (F1) of A and P only if a single combined column is non-negotiable.
This is a thinking doc, not a published metric. Follow-up PRs will implement the recommended metric in `analysis.py` and update the paper / app prose to match.
Test plan