Add a generated Pareto social card and switch to large share previews - #216
Conversation
Every share of a SenseBench link on X, Slack, LinkedIn or Discord rendered as a grey text row: the base template hardcoded twitter:card to summary and no page carried an og:image at all. That silently throttled the one distribution channel a leaderboard depends on. The card is the cost/accuracy Pareto chart, which is the most distinctive thing the project has and the only axis no other LLM leaderboard publishes. The publication figure in the paper cannot be used directly: share previews render around 500px wide, where its per-point callouts and eighteen-family legend are illegible, and a static figure would drift from a leaderboard that moves. So tools/make_og_card.py redraws the same chart for the format, labelling only the frontier knee, the best overall run, and the best plain open-weights run, with the family legend collapsed to open weights versus proprietary. Costs come from the aggregated leaderboard JSON rather than from run artifacts. That matters: self-hosted runs are priced at reference GPU rates on the leaderboard but at the rate actually paid in run.json, so reading artifacts directly would have produced a card that disagreed with the chart it advertises. Reading the aggregate also keeps generation instant, where re-verifying 198 runs takes twenty minutes. The card carries run and model counts, so it goes stale as runs merge. --check compares the committed state against the leaderboard and CI warns rather than fails, since run submissions land often and a stale count is not worth blocking a deploy over. Nothing in CI inspects rendered head markup, so the new test asserts the large-card meta tags on both the index and a run page and that the image is copied into the site output. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe change replaces leaderboard-driven social card generation with compositing from a committed Pareto figure, adds Matplotlib to development dependencies, and documents the regeneration command. Generated pages now include Open Graph image metadata and use large Twitter cards. Site tests verify the metadata and generated image asset. Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 3c6b4872-cf3b-4ab3-b2ec-989f0da07d9a
⛔ Files ignored due to path filters (2)
src/sensebench/site/static/og-card.pngis excluded by!**/*.pnguv.lockis excluded by!**/*.lock
📒 Files selected for processing (8)
.github/workflows/pages.ymlREADME.mdpyproject.tomlsrc/sensebench/site/build.pysrc/sensebench/site/templates/base.html.j2tests/test_site.pytools/make_og_card.pytools/og_card_state.json
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: build
- GitHub Check: package
🧰 Additional context used
🪛 ast-grep (0.44.1)
tools/make_og_card.py
[info] 329-337: use jsonify instead of json.dumps for JSON output
Context: json.dumps(
{
"run_count": state.run_count,
"model_count": state.model_count,
"top_accuracy_pct": state.top_accuracy_pct,
"plotted_point_count": state.plotted_point_count,
},
indent=2,
)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
🔇 Additional comments (10)
tools/make_og_card.py (3)
1-89: LGTM!
91-315: LGTM!
328-408: LGTM! The static-analysis "use jsonify" hint on thejson.dumpscall is a false positive here — this is a local script writing a state file, not an HTTP response.tools/og_card_state.json (1)
1-7: LGTM!pyproject.toml (1)
82-82: LGTM!src/sensebench/site/build.py (1)
136-143: LGTM!Also applies to: 1119-1126
src/sensebench/site/templates/base.html.j2 (1)
13-17: LGTM!tests/test_site.py (1)
81-81: LGTM!Also applies to: 170-172, 739-767
.github/workflows/pages.yml (1)
27-31: LGTM!README.md (1)
277-287: LGTM!
| def _card_state(*, entries: list[LeaderboardEntry], points: list[CardPoint]) -> CardState: | ||
| accuracies = [entry.accuracy for entry in entries if entry.accuracy is not None] | ||
| top_accuracy_pct = max(accuracies) * 100.0 if len(accuracies) > 0 else 0.0 | ||
| return CardState( | ||
| run_count=len(entries), | ||
| model_count=len({entry.model for entry in entries}), | ||
| # Rounded here so the recomputed state compares equal to the stored one. | ||
| top_accuracy_pct=round(top_accuracy_pct, STATE_ACCURACY_DECIMALS), | ||
| plotted_point_count=len(points), | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
top_accuracy_pct should be scoped to the same filter as the chart, not all entries.
The header renders "best {top_accuracy_pct}% on lexEN v1" (line 222), but top_accuracy_pct here is computed from entries — unfiltered by HEADLINE_PROMPT_ID/DEFAULT_LEXEN_RELEASE_ID. _comparable_points already applies that exact filter to produce points, whose accuracy_pct field is directly usable. If any entry with a different prompt or dataset version scores higher, the card will advertise an accuracy that doesn't correspond to anything plotted on the "lexEN v1" chart.
🐛 Proposed fix
def _card_state(*, entries: list[LeaderboardEntry], points: list[CardPoint]) -> CardState:
- accuracies = [entry.accuracy for entry in entries if entry.accuracy is not None]
- top_accuracy_pct = max(accuracies) * 100.0 if len(accuracies) > 0 else 0.0
+ top_accuracy_pct = max((point.accuracy_pct for point in points), default=0.0)
return CardState(
run_count=len(entries),
model_count=len({entry.model for entry in entries}),
# Rounded here so the recomputed state compares equal to the stored one.
top_accuracy_pct=round(top_accuracy_pct, STATE_ACCURACY_DECIMALS),
plotted_point_count=len(points),
)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def _card_state(*, entries: list[LeaderboardEntry], points: list[CardPoint]) -> CardState: | |
| accuracies = [entry.accuracy for entry in entries if entry.accuracy is not None] | |
| top_accuracy_pct = max(accuracies) * 100.0 if len(accuracies) > 0 else 0.0 | |
| return CardState( | |
| run_count=len(entries), | |
| model_count=len({entry.model for entry in entries}), | |
| # Rounded here so the recomputed state compares equal to the stored one. | |
| top_accuracy_pct=round(top_accuracy_pct, STATE_ACCURACY_DECIMALS), | |
| plotted_point_count=len(points), | |
| ) | |
| def _card_state(*, entries: list[LeaderboardEntry], points: list[CardPoint]) -> CardState: | |
| top_accuracy_pct = max((point.accuracy_pct for point in points), default=0.0) | |
| return CardState( | |
| run_count=len(entries), | |
| model_count=len({entry.model for entry in entries}), | |
| # Rounded here so the recomputed state compares equal to the stored one. | |
| top_accuracy_pct=round(top_accuracy_pct, STATE_ACCURACY_DECIMALS), | |
| plotted_point_count=len(points), | |
| ) |
The first version redrew the chart from leaderboard data for small-format legibility. It read as a generic scatter plot and lost what makes the paper figure worth sharing: family colours, square and circle markers for open weights versus proprietary, leader-line callouts on the frontier points, and the model-family legend. The card is now that published figure with a header band above it. The figure is hand-curated in the paper repository -- per-run label offsets and frontier display names are tuned by hand there -- so it is committed as tools/assets/pareto-figure.png rather than regenerated from run data, and the tool only composites the header. Two consequences worth stating. The header carries no run or model counts, so the card no longer goes stale as runs merge; the freshness check and its CI warning are gone with it. And the figure's own annotations are set for print, so at the width a share preview actually renders they read as texture rather than text -- the wordmark and the shape of the frontier are what carry it there, with the labels legible once expanded. Refreshing the card now means re-rendering the figure in the paper repository, trimming its margins, and overwriting the asset. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 61a35dfc-c1e3-4e20-9947-7989372d2da0
⛔ Files ignored due to path filters (2)
src/sensebench/site/static/og-card.pngis excluded by!**/*.pngtools/assets/pareto-figure.pngis excluded by!**/*.png
📒 Files selected for processing (2)
README.mdtools/make_og_card.py
📜 Review details
⏰ Context from checks skipped due to timeout. (3)
- GitHub Check: build
- GitHub Check: checks
- GitHub Check: package
🔇 Additional comments (1)
tools/make_og_card.py (1)
1-12: LGTM!Also applies to: 17-113
| The header deliberately carries no run or model counts, so the card does not go | ||
| stale as runs are merged. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Clarify that the plotted figure can still become stale.
Removing run/model counts only prevents stale header text; newly verified runs can still make the committed chart—and the published “every verified run” alt text—outdated.
tools/make_og_card.py#L13-L14: describe the card as a snapshot that must be regenerated when plotted data should be current.README.md#L286: replace the claim that the card does not go stale with the same snapshot/refresh guidance.
📍 Affects 2 files
tools/make_og_card.py#L13-L14(this comment)README.md#L286-L286
1 participant
What
Every share of a SenseBench link rendered as a grey text row —
base.html.j2hardcodedtwitter:cardtosummaryand no page carried anog:image. This adds a generated card and switches tosummary_large_image.Why generated, not the paper figure
The paper's
f3_paretofigure is the right chart but the wrong artifact for this slot:tools/make_og_card.pyredraws the same chart for the format: only the frontier knee, the best overall run and the best plain open-weights run are labelled, and the legend collapses to open weights vs proprietary.Why it reads the aggregate, not run artifacts
Self-hosted runs are priced at reference GPU rates on the leaderboard but at the rate actually paid in
run.json. Reading artifacts directly would have produced a card whose costs disagreed with the chart it advertises. Readingleaderboard.jsonalso makes generation instant — re-verifying all 198 runs takes ~20 minutes.Freshness
The card carries run and model counts, so it goes stale as runs merge.
--checkcompares committed state against the leaderboard, and CI warns rather than fails — run submissions land often and a stale count is not worth blocking a deploy over.Tests
Nothing in CI inspects rendered head markup (
--strictis a data-eligibility gate and raises before any HTML renders). The new test asserts the large-card meta tags on both the index and a run page, and that the image lands in the site output.matplotlib is added to the dev group only, so the installed package stays lean.