Refreeze manuscript on the 33-model board

app/src/app/paper/page.tsx

@@ -4,10 +4,10 @@ import Link from "next/link";
 
 import SiteHeader from "../../components/SiteHeader";
 
-const SNAPSHOT_DATE_LABEL = "Snapshot 2026-08-17";
+const SNAPSHOT_DATE_LABEL = "Snapshot 2026-09-01";
 
 const PAPER_DESCRIPTION =
-  "PolicyBench paper — the 2026-08-17 household-equal scored manuscript snapshot, with June 12–August 17 model responses and PolicyEngine reference outputs on the populace US microdata.";
+  "PolicyBench paper — the 2026-09-01 household-equal scored manuscript snapshot, with June 12–September 1 model responses and PolicyEngine reference outputs on the populace US microdata.";
 
 export const metadata: Metadata = {
   title: "Paper",
@@ -40,7 +40,7 @@ export const metadata: Metadata = {
 
 const manuscriptPaths = {
   pdf: "/paper/policybench.pdf",
-  web: "/paper/web/index.html?v=20260817-refreeze",
+  web: "/paper/web/index.html?v=20260901-refreeze",
 };
 const ssrnUrl = process.env.NEXT_PUBLIC_POLICYBENCH_SSRN_URL;
 
@@ -49,7 +49,7 @@ export default function PaperPage() {
     <>
       <p className="max-w-2xl text-sm leading-relaxed text-text-secondary sm:text-base">
         Benchmarking no-tool tax-and-benefit estimation in frontier language
-        models. This page embeds the 2026-08-17 scored manuscript snapshot: a
+        models. This page embeds the 2026-09-01 scored manuscript snapshot: a
         100-household public preview using household-equal impact scores against
         PolicyEngine reference outputs.
       </p>

paper/index.qmd

@@ -1080,7 +1080,7 @@ The benchmark requires each model response to include numeric answers and one ex
 
 Because explanations are required, the canonical task measures policy estimation under a public-facing structured-response contract, not isolated arithmetic accuracy. Prompt fairness is part of the benchmark contract. The current release uses one prompt template per country, with no model-specific tuning. Models receive the same household facts and requested outputs. They receive no web or external tool access. The prompt sets unlisted numeric inputs to `0`, unlisted boolean or status facts to false, and household characteristics as constant over the tax-benefit year. Provider-specific differences are limited to structured-output transport, request shape, and the reasoning setup in @tbl-model-runs. The harness sends no reasoning-control or sampling parameters. It leaves reasoning effort, temperature, and related decoding controls at provider defaults. The frozen GPT-5.5 wave uses provider-default reasoning effort.
 
-The forced `tool_choice` suppresses Claude's extended thinking in this snapshot ([`sensitivity/claude-thinking-2026-08.md`](https://github.com/PolicyEngine/policybench/blob/main/sensitivity/claude-thinking-2026-08.md)). A labeled sensitivity with `tool_choice: "auto"` is published beside the board. The next board version moves every model to `tool_choice: "auto"` so each provider's default reasoning posture can engage.
+The forced `tool_choice` suppresses Claude's extended thinking in this snapshot ([`sensitivity/claude-thinking-2026-08.md`](https://github.com/PolicyEngine/policybench/blob/main/sensitivity/claude-thinking-2026-08.md)). A labeled sensitivity with `tool_choice: "auto"` is published beside the board. Claude Fable 5.1, released September 1, 2026, closes the interaction from the API side: it rejects forced tool use with an error, so its row answers through the JSON transport and reasons at the provider default; a sensitivity run with the tool declared under `tool_choice: "auto"` scores 1.2 points above that row. The next board version moves every model to `tool_choice: "auto"` so each provider's default reasoning posture can engage.
 
 Request shape is the one dimension on which requests were not identical across models. `{python} n_chunked_models` of the `{python} r.n_models_fmt` models could not reliably complete the whole-scenario request — their serving stacks rejected the structured-output call, exhausted completion budgets mid-response, or timed out — and instead answered the same prompt template over subsets of the requested outputs, one to three outputs per request (@tbl-model-runs). Subsetting gives each response fewer requested outputs and a fresh completion budget over the same household facts, so the accommodation, if it moves scores at all, should favor the chunked models; their scores are not strictly comparable to whole-scenario scores. This accommodation is closed going forward: a model added after this snapshot either answers the canonical whole-scenario request or is listed as not scorable rather than accommodated.
 
@@ -1243,14 +1243,19 @@ def _lag(frame):
     ]
     anchor = min(closed_above["model"], key=_rel)
     days = (_rel(best_open.model) - _rel(anchor)).days
-    return best_open, anchor, days
+    return best_open, anchor, days, len(closed_above)
+
+
+def _closed_models_phrase(n: int) -> str:
+    word = {1: "one", 2: "two", 3: "three", 4: "four"}.get(n, str(n))
+    return f"{word} closed model{'' if n == 1 else 's'}"
 
 
 _pre_july = us_model[
     us_model["model"].map(_rel) < _date(2026, 7, 1)
 ]
-june_open, june_anchor, june_days = _lag(_pre_july)
-full_open, full_anchor, full_days = _lag(us_model)
+june_open, june_anchor, june_days, _ = _lag(_pre_july)
+full_open, full_anchor, full_days, full_n_closed_above = _lag(us_model)
 june_months = f"{june_days / 30.44:.1f}"
 full_weights_days = (_date(2026, 7, 27) - _rel(full_anchor)).days
 
@@ -1260,7 +1265,7 @@ def _prose_date(model: str) -> str:
     return f"{d.strftime('%B')} {d.day}, {d.year}"
 ```
 
-The leaderboard also measures time. Following the lag construction the UK AI Security Institute applied to cyber capabilities [@aisi2026openweight], the open-weight lag is the time between the closed frontier first reaching a score and the leading open-weight model matching it. Two cuts of the frozen roster give the trajectory. Among models released before July 2026, the leading open-weight model is `{python} model_display_name(june_open.model)` at `{python} f"{june_open.exact:.1f}"`% exact, whose weights shipped on `{python} _prose_date(june_open.model)` [@deepseek2026v4]; the earliest-released closed model above that score is `{python} model_display_name(june_anchor)`, released `{python} _prose_date(june_anchor)` [@techcrunch2025gemini3flash] — a lag of `{python} str(june_days)` days, about `{python} june_months` months. On the full roster, `{python} model_display_name(full_open.model)` at `{python} f"{full_open.exact:.1f}"`% exact trails only `{python} model_display_name(full_anchor)`, generally available `{python} _prose_date(full_anchor)` [@techcrunch2026gpt56], putting the lag at `{python} str(full_days)` days by API availability, or `{python} str(full_weights_days)` days dated to Kimi K3's July 27 weight release on Hugging Face [@techtimes2026kimik3weights].
+The leaderboard also measures time. Following the lag construction the UK AI Security Institute applied to cyber capabilities [@aisi2026openweight], the open-weight lag is the time between the closed frontier first reaching a score and the leading open-weight model matching it. Two cuts of the frozen roster give the trajectory. Among models released before July 2026, the leading open-weight model is `{python} model_display_name(june_open.model)` at `{python} f"{june_open.exact:.1f}"`% exact, whose weights shipped on `{python} _prose_date(june_open.model)` [@deepseek2026v4]; the earliest-released closed model above that score is `{python} model_display_name(june_anchor)`, released `{python} _prose_date(june_anchor)` [@techcrunch2025gemini3flash] — a lag of `{python} str(june_days)` days, about `{python} june_months` months. On the full roster, `{python} model_display_name(full_open.model)` at `{python} f"{full_open.exact:.1f}"`% exact trails `{python} _closed_models_phrase(full_n_closed_above)`; the earliest released of them, `{python} model_display_name(full_anchor)`, generally available `{python} _prose_date(full_anchor)` [@techcrunch2026gpt56], puts the lag at `{python} str(full_days)` days by API availability, or `{python} str(full_weights_days)` days dated to Kimi K3's July 27 weight release on Hugging Face [@techtimes2026kimik3weights].
 
 Both figures are lower bounds in one direction: a closed model outside the evaluated roster that crossed either score earlier would lengthen the corresponding lag. They also carry the uncertainty of any single-benchmark, single-domain estimate — the AISI analysis reports substantially longer lags on cyber tasks — and a lag measured at a moving frontier compresses mechanically when a strong open-weight release lands days after a frontier release.
 

paper/snapshot/20260501/manifest.json

@@ -9,15 +9,15 @@
     "path": "annotations/us_full_run_20260612_policyengine_4_16_1_populace"
   },
   "committed_snapshot_artifacts": {
-    "model_serving_config.json": "75fba1977954c08f62f112a7453e4950ed3291f94240e174a8d53d86bf1665a2",
-    "us_impact_summary_by_model.csv": "17ce4c8f1b7ec810a4a6561ffe74f30aca7f896418e85019d017391a2327d92b",
+    "model_serving_config.json": "591e5a437f117c1f0340b343c0b0016b1cb07cef17c50e355e2aeb8e0554cc30",
+    "us_impact_summary_by_model.csv": "9cab19bf5486b403bd200aa27d9f19eb40e2f6166d4338dd379bb8f1934284f3",
     "us_reference_outputs.csv": "b9136a15e285f9c02ba78bee854b8a3af180e280485512642c829e8ffd7d2368",
     "us_scenarios.csv": "71b16212f0c0b3e5d13d8694ce57e362c23248665806c4d6dea7b23ef472858a"
   },
   "files": [
     {
       "path": "runs/us_full_run_20260612_policyengine_4_16_1_populace/data.json",
-      "sha256": "1cf8ce1ddbde0d22aae4690d8e99739dc15340b9439a85f2f648648cb7da28f3"
+      "sha256": "07e5de1c27248ef90bed81f9f745be268419c798bb664ff89ef0dd323793adb3"
     }
   ],
   "live_dashboard_artifact": {
@@ -29,7 +29,7 @@
     "url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20260901/dashboard-data.json"
   },
   "live_dashboard_note": "The live dashboard payload is a published release asset; the committed pointer app/src/data.artifact.json must reference the artifact pinned under live_dashboard_artifact. The separate published_dashboard_artifact freezes the combined export of the source run data.json files listed under source_run_artifacts. A later publication may advance the live entry without changing the frozen pin.",
-  "model_response_date": "2026-06-12 to 2026-08-22",
+  "model_response_date": "2026-06-12 to 2026-09-01",
   "policy_period": {
     "us": "tax year 2026"
   },
@@ -40,13 +40,13 @@
   },
   "published_dashboard_artifact": {
     "asset": "dashboard-data.json",
-    "bytes": 85042876,
-    "sha256": "b883ec669d510ea29c9c18f18c30030c5bbd29f770bcd90d257779940929a895",
-    "tag": "dashboard-data-20260822",
-    "url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20260822/dashboard-data.json"
+    "bytes": 87392615,
+    "sha256": "1b2de0966daa97238f34c30e06de52b88ecd5f18ae4d34a3ae2b007bdd5f3dea",
+    "tag": "dashboard-data-20260901",
+    "url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20260901/dashboard-data.json"
   },
   "reference_output_refresh": {
-    "date": "2026-08-22",
+    "date": "2026-09-01",
     "policyengine_us_data_artifact_sha256": "f32c2e5e9098bc6540724fdd5debf963af495da4c29b3a7a63fb53c2a4bb5a34",
     "policyengine_us_data_build_id": "populace-us-2024-5da5a95-20260611",
     "policyengine_us_dataset": "populace_us_2024",
@@ -57,12 +57,12 @@
   "rendered_paper_artifacts": {
     "pdf": {
       "path": "app/public/paper/policybench.pdf",
-      "sha256": "5bce51fc696b5d37d7fbed2f70efb3bed9f4f8fb76ea8a79b9663ef0836a3098"
+      "sha256": "4d950654a57848af893bcb956d4f0b4cb5ff9ea3fdfb62cefd6d1e14ad5d41ac"
     },
     "web": {
       "files": {
-        "figures/positive_zero_scatter.png": "f680c4339ed362eaf0e50490789130274ddfc660f03446adfe366296e767c87d",
-        "index.html": "e2ac9e2b00e6e0c11c4bdb0f54b679c8aa976f9bbc83438f924eff70eebbce21",
+        "figures/positive_zero_scatter.png": "ab45ad994420d5242d6905b83e015b50fcb4aa07babc20e10702cfee5dbda138",
+        "index.html": "868557cf9d4f6aa690d739a46146544d80776e23cfed0a248a4167f6b7dfe0ae",
         "pe-tokens.css": "8f24d8da26f583c8ffddffcdcd172b6d52cbecfec20eda55bd39d7aa829f41d8",
         "policybench-theme.css": "0e12c5fd615558259e5bce0167a38424e54f9ceb280666c4afd660d759cd1cb9",
         "site_libs/clipboard/clipboard.min.js": "e17a1d816e13c0826e0ed7febfabc3277f45571234bde0bf9120829a7169edc9",
@@ -84,11 +84,11 @@
   },
   "reproducibility_notes": [
     "The top-level scenario, reference-output, and impact-summary CSVs are byte-identical to the corresponding compact source-run artifacts copied under paper/snapshot/20260501/runs/.",
-    "Model responses were collected in waves between June 12 and August 22, 2026, as models were added to the board; each model's full 100-household run is a single consistent wave. Reference outputs were generated with policyengine.py 4.16.1 and policyengine-us 1.755.4 against the certified PolicyEngine US populace dataset (populace-us-2024-5da5a95-20260611, populace_us_2024).",
+    "Model responses were collected in waves between June 12 and September 1, 2026, as models were added to the board; each model's full 100-household run is a single consistent wave. Reference outputs were generated with policyengine.py 4.16.1 and policyengine-us 1.755.4 against the certified PolicyEngine US populace dataset (populace-us-2024-5da5a95-20260611, populace_us_2024).",
     "Canonical prediction files include parser recovery. Later waves ran under the resumable supervised runner, which retries failed or timed-out scenarios in bounded rounds; every model's canonical file covers all 100 households.",
     "Raw provider responses are retained in the compressed source-run predictions.csv.gz file. The separate LiteLLM cache remains local-only because it is a generated request cache, not the canonical snapshot artifact.",
     "The frozen scenarios.csv source_dataset column carries a stale enhanced_cps_2024 label from the pre-#77 scenario generator; the run metadata (scenarios.csv.meta.json) records the populace_us_2024 build actually loaded.",
-    "Model APIs and upstream model aliases may change after the recorded 2026-06-12 to 2026-08-22 response window, so exact reruns can diverge even with the committed household inputs, reference outputs, parsed dashboard export, and analysis summaries."
+    "Model APIs and upstream model aliases may change after the recorded 2026-06-12 to 2026-09-01 response window, so exact reruns can diverge even with the committed household inputs, reference outputs, parsed dashboard export, and analysis summaries."
   ],
   "response_retry_artifacts": {
     "files": {},
@@ -105,24 +105,24 @@
     "households": {
       "us": 100
     },
-    "models": 32,
+    "models": 33,
     "output_groups": {
       "us": 18
     }
   },
-  "snapshot_date": "2026-08-22",
+  "snapshot_date": "2026-09-01",
   "source_run_artifacts": {
     "note": "Compact copies of run outputs used to verify this snapshot. The run data.json retains parsed scenario predictions, explanations, summaries, heatmaps, and PolicyEngine runtime metadata used by the dashboard. predictions.csv.gz is a deterministic gzip of the run's raw provider responses.",
     "us_full_run_20260612_policyengine_4_16_1_populace": {
       "files": {
-        "analysis/impact_summary_by_model.csv": "17ce4c8f1b7ec810a4a6561ffe74f30aca7f896418e85019d017391a2327d92b",
-        "analysis/metrics.csv": "c36d43c2b488f089c67cf78299659b2fa53eb6419764f02b320512bf496b2c73",
-        "analysis/report.md": "5a38ba58703b287472c356afe0eeb8897c75c97123a426f99b724afb7f272ab4",
-        "analysis/summary_by_model.csv": "301b9cf9b59380ce82f38aa403fb94d7f7050ccebad156c9065a4053b5bbd2cb",
-        "analysis/summary_by_variable.csv": "883f6fac805fa23ba58e0f43e3e9872d39d8f44f96b01c94a4d2f201915c55d5",
-        "analysis/usage_summary.csv": "33e5d74c7018c5117e353d4dc2a3ebe87ee569915444862b5abd4b19948ddfa5",
-        "data.json": "1cf8ce1ddbde0d22aae4690d8e99739dc15340b9439a85f2f648648cb7da28f3",
-        "predictions.csv.gz": "78cb501fa9f6e38704a79c9a46ed7af05771ab061da17dcb0904dfa12cc5c8ee",
+        "analysis/impact_summary_by_model.csv": "9cab19bf5486b403bd200aa27d9f19eb40e2f6166d4338dd379bb8f1934284f3",
+        "analysis/metrics.csv": "fe9b9d33a8804a2558a4ca3dae91f3a6a1652105acb5c4c3511c73f2d8715547",
+        "analysis/report.md": "e4e02f7e157d7e2b8ba65221b224986fe49418d0e8e93b638003e8c0f695fd10",
+        "analysis/summary_by_model.csv": "cba35cab84442c98af0328db832f1f79bd8f809e8e35e3af282901634511e0bb",
+        "analysis/summary_by_variable.csv": "e2f34896befe83cde9e0e891280ee48fa88549c6c0d9e25122d59e34c23ff254",
+        "analysis/usage_summary.csv": "a0761acf191bb847b571a4a0f42e74c1f6347ae1ab62af820c9d152ca657b30e",
+        "data.json": "07e5de1c27248ef90bed81f9f745be268419c798bb664ff89ef0dd323793adb3",
+        "predictions.csv.gz": "f1f3e5e27dbd77c788644a440c696f80eeb2797c3ece524decbf40677394063f",
         "reference_outputs.csv": "b9136a15e285f9c02ba78bee854b8a3af180e280485512642c829e8ffd7d2368",
         "reference_outputs.csv.meta.json": "4ea7911857fabf72fc0f74feab90566e9f6d98cab4958e0c1401043e58c94210",
         "scenarios.csv": "71b16212f0c0b3e5d13d8694ce57e362c23248665806c4d6dea7b23ef472858a",

paper/snapshot/20260501/model_serving_config.json

@@ -8,6 +8,14 @@
       "shared_completion_budget_tokens": 16384,
       "tool_choice": "forced"
     },
+    "claude-fable-5.1": {
+      "answer_contract": "json",
+      "provider_id": "claude-fable-5-1",
+      "reasoning_setup": "provider default; 16,384-token shared budget",
+      "request_shape": "whole scenario",
+      "shared_completion_budget_tokens": 16384,
+      "tool_choice": null
+    },
     "claude-haiku-4.5": {
       "answer_contract": "tool",
       "provider_id": "claude-haiku-4-5-20251001",

paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace/analysis/impact_summary_by_model.csv

@@ -1,4 +1,5 @@
 model,mean_impact_score,mean_household_score,mean_household_coverage,households,total_variables,parsed_variables,floor_share
+claude-fable-5.1,0.9346419512988297,0.9674406077584561,1.0,100,1984,1984,0.3
 inkling,0.9336310139316637,0.9592442671897276,1.0,100,1984,1984,0.3
 gpt-5.6-sol,0.9333432088386975,0.965508936901421,1.0,100,1984,1984,0.3
 gpt-5.5,0.9151482777047545,0.9545673865419815,1.0,100,1984,1984,0.3