Refreeze the manuscript on the 29-model board

paper/index.qmd

@@ -1149,7 +1149,7 @@ md_table(snapshot_provenance)
 
 ```{python}
 #| label: tbl-model-runs
-#| tbl-cap: "Model run configuration in the frozen manuscript snapshot. Every model uses its provider's structured-output transport with no external tools, provider-default temperature and sampling, no reasoning-control parameters, and the same prompt template over the same household facts; request shape and reasoning setup are the only per-model serving configuration. Rows with an output count answered the template over subsets of the requested outputs, an accommodation that predates the canonical whole-scenario rule described in the text. Released is the first public availability (paid tiers count; trusted-tester previews do not), compiled from vendor announcements and contemporaneous press; grok-build-0.1's date rests on secondary trackers. Leaderboard artifact keys match the final segment of the provider id (qwen-3.7-max adds a hyphen). A dagger marks open-weight models; Kimi K3's weights were announced for July 27, 2026, after this snapshot, so it is open-weight by commitment [@willison2026kimik3]."
+#| tbl-cap: "Model run configuration in the frozen manuscript snapshot. Every model uses its provider's structured-output transport with no external tools, provider-default temperature and sampling, no reasoning-control parameters, and the same prompt template over the same household facts; request shape and reasoning setup are the only per-model serving configuration. Rows with an output count answered the template over subsets of the requested outputs, an accommodation that predates the canonical whole-scenario rule described in the text. Released is the first public availability (paid tiers count; trusted-tester previews do not), compiled from vendor announcements and contemporaneous press; grok-build-0.1's date rests on secondary trackers. Leaderboard artifact keys match the final segment of the provider id (qwen-3.7-max adds a hyphen). A dagger marks open-weight models; Kimi K3's weights, announced at its API launch [@willison2026kimik3], shipped on Hugging Face on July 27, 2026 under a custom license [@techtimes2026kimik3weights]."
 md_table(model_runs)
 ```
 
@@ -1311,7 +1311,7 @@ def _prose_date(model: str) -> str:
     return f"{d.strftime('%B')} {d.day}, {d.year}"
 ```
 
-The leaderboard also measures time. Following the lag construction the UK AI Security Institute applied to cyber capabilities [@aisi2026openweight], the open-weight lag is the time between the closed frontier first reaching a score and the leading open-weight model matching it. Two cuts of the frozen roster give the trajectory. Among models released before July 2026, the leading open-weight model is `{python} model_display_name(june_open.model)` at `{python} f"{june_open.exact:.1f}"`% exact, whose weights shipped on `{python} _prose_date(june_open.model)` [@deepseek2026v4]; the earliest-released closed model above that score is `{python} model_display_name(june_anchor)`, released `{python} _prose_date(june_anchor)` [@techcrunch2025gemini3flash] — a lag of `{python} str(june_days)` days, about `{python} june_months` months. On the full roster, `{python} model_display_name(full_open.model)` at `{python} f"{full_open.exact:.1f}"`% exact trails only `{python} model_display_name(full_anchor)`, generally available `{python} _prose_date(full_anchor)` [@techcrunch2026gpt56], putting the lag at `{python} str(full_days)` days by API availability, or `{python} str(full_weights_days)` days dated to Kimi K3's announced July 27 weight release [@willison2026kimik3].
+The leaderboard also measures time. Following the lag construction the UK AI Security Institute applied to cyber capabilities [@aisi2026openweight], the open-weight lag is the time between the closed frontier first reaching a score and the leading open-weight model matching it. Two cuts of the frozen roster give the trajectory. Among models released before July 2026, the leading open-weight model is `{python} model_display_name(june_open.model)` at `{python} f"{june_open.exact:.1f}"`% exact, whose weights shipped on `{python} _prose_date(june_open.model)` [@deepseek2026v4]; the earliest-released closed model above that score is `{python} model_display_name(june_anchor)`, released `{python} _prose_date(june_anchor)` [@techcrunch2025gemini3flash] — a lag of `{python} str(june_days)` days, about `{python} june_months` months. On the full roster, `{python} model_display_name(full_open.model)` at `{python} f"{full_open.exact:.1f}"`% exact trails only `{python} model_display_name(full_anchor)`, generally available `{python} _prose_date(full_anchor)` [@techcrunch2026gpt56], putting the lag at `{python} str(full_days)` days by API availability, or `{python} str(full_weights_days)` days dated to Kimi K3's July 27 weight release on Hugging Face [@techtimes2026kimik3weights].
 
 Both figures are lower bounds in one direction: a closed model outside the evaluated roster that crossed either score earlier would lengthen the corresponding lag. They also carry the uncertainty of any single-benchmark, single-domain estimate — the AISI analysis reports substantially longer lags on cyber tasks — and a lag measured at a moving frontier compresses mechanically when a strong open-weight release lands days after a frontier release.
 

paper/references.bib

@@ -237,3 +237,12 @@ @misc{willison2026kimik3
   howpublished = {simonwillison.net, July 16, 2026},
   url          = {https://simonwillison.net/2026/Jul/16/kimi-k3/}
 }
+
+@online{techtimes2026kimik3weights,
+  author = {{Tech Times}},
+  title = {Kimi K3 Open Weights Arrive Sunday: Self-Hosting Cuts China Data Risk the API Never Can},
+  year = {2026},
+  month = {7},
+  url = {https://www.techtimes.com/articles/321551/20260725/kimi-k3-open-weights-arrive-sunday-self-hosting-cuts-china-data-risk-api-never-can.htm},
+  urldate = {2026-08-05}
+}

paper/snapshot/20260501/manifest.json

@@ -2,21 +2,21 @@
   "audit_annotation_artifacts": {
     "files": {
       "us_audit_row_annotations.csv": "8ae8dd77089bc84187c6eae6386266bf24a313709e41c2af2aef76504e6bcaa8",
-      "us_case_notes.csv": "b557b7e967263d67b1acc60e434cfba6f5fe9ba5cf6c316cdcd6010f86943f46",
+      "us_case_notes.csv": "7e340f1dc6e22e2bbae18fd92579eab1d886918725377d3bf28be63f08ef5030",
       "us_case_reference_explanations.csv": "a3fc7504dcf3d8abe11e98c2228c786a5171a825640b980835e1e210d3afa198"
     },
     "note": "Model-assisted, developer-adjudicated row and case audit annotations for every wrong prediction row in the frozen snapshot, produced under the decisive-diagnosis contract (per-model diagnoses grounded in engine facts; hedged verdicts mechanically rejected and re-judged). Row-level failure_source values are llm_error for substantive misses and parse_contract_failure for missing or unparseable answers; zero rows are reference-suspect. Case notes are stored as us_case_notes.csv with case_failure_sources / case_failure_subtypes columns. Reference narratives the judge and dashboard display are frozen as us_case_reference_explanations.csv.",
     "path": "annotations/us_full_run_20260612_policyengine_4_16_1_populace"
   },
   "committed_snapshot_artifacts": {
-    "us_impact_summary_by_model.csv": "98bc36df93c87904dfaa3932249b1e2da82f9eecf40b5c7bc45016cc75352964",
+    "us_impact_summary_by_model.csv": "6de83b553e62efe29099db4a54b10b31de617d4fe0b29b97366384d3c9981213",
     "us_reference_outputs.csv": "b9136a15e285f9c02ba78bee854b8a3af180e280485512642c829e8ffd7d2368",
     "us_scenarios.csv": "71b16212f0c0b3e5d13d8694ce57e362c23248665806c4d6dea7b23ef472858a"
   },
   "files": [
     {
       "path": "runs/us_full_run_20260612_policyengine_4_16_1_populace/data.json",
-      "sha256": "1bebdfbf4b5527664be594165a17bf66e0d5b71844c2833ba10233dd05f10760"
+      "sha256": "4396793ec0bb0c0e0652629b5874abac0299a048094b600531fd4e0e1b7d8a6c"
     }
   ],
   "live_dashboard_artifact": {
@@ -28,7 +28,7 @@
     "url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20260805/dashboard-data.json"
   },
   "live_dashboard_note": "The live dashboard payload is a published release asset; the committed pointer app/src/data.artifact.json must reference the artifact pinned under published_dashboard_artifact, which equals the combined export of the source run data.json files listed under source_run_artifacts. Future refreshes publish a new artifact (policybench publish-dashboard) and either update this manifest or use a new snapshot directory.",
-  "model_response_date": "2026-06-12 to 2026-07-24",
+  "model_response_date": "2026-06-12 to 2026-08-05",
   "policy_period": {
     "us": "tax year 2026"
   },
@@ -39,13 +39,13 @@
   },
   "published_dashboard_artifact": {
     "asset": "dashboard-data.json",
-    "bytes": 73010187,
-    "sha256": "38453b0225bee779500551b4aa0699a51c6c6c8ba4372d6c130bbd81afd4813d",
-    "tag": "dashboard-data-20260724",
-    "url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20260724/dashboard-data.json"
+    "bytes": 77819217,
+    "sha256": "5463a5d13281e33246ccaf9270726ab0ca1c228b030c7bdb991a768df974fbc5",
+    "tag": "dashboard-data-20260805",
+    "url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20260805/dashboard-data.json"
   },
   "reference_output_refresh": {
-    "date": "2026-07-24",
+    "date": "2026-08-05",
     "policyengine_us_data_artifact_sha256": "f32c2e5e9098bc6540724fdd5debf963af495da4c29b3a7a63fb53c2a4bb5a34",
     "policyengine_us_data_build_id": "populace-us-2024-5da5a95-20260611",
     "policyengine_us_dataset": "populace_us_2024",
@@ -56,12 +56,12 @@
   "rendered_paper_artifacts": {
     "pdf": {
       "path": "app/public/paper/policybench.pdf",
-      "sha256": "9b1728a8dc037ea392b2278831116469cfa81e998ffb6669edaf804485e9c091"
+      "sha256": "9595e5337ccc73b65465e8d167dfd7f6dc3f1283f1239a979777fb335f54b199"
     },
     "web": {
       "files": {
-        "figures/positive_zero_scatter.png": "70ed97a9aa4ee901e449372010ed7ec1887f441ad60cafe973fafb529731c2e6",
-        "index.html": "c306da4f050c76f86b5ab20f19ebe486e996d143c4f0911aafc3df8dbead9eda",
+        "figures/positive_zero_scatter.png": "84adde00967f3a1740e3af2b16d0b97f6a6c66b4b13fd2fe9d8cd8ea20cb6998",
+        "index.html": "ffb68d7a74e61e319443dada1439ae3df02cfc7dfa0bb9587cef37bf18a4fc62",
         "pe-tokens.css": "8f24d8da26f583c8ffddffcdcd172b6d52cbecfec20eda55bd39d7aa829f41d8",
         "policybench-theme.css": "0e12c5fd615558259e5bce0167a38424e54f9ceb280666c4afd660d759cd1cb9",
         "site_libs/clipboard/clipboard.min.js": "e17a1d816e13c0826e0ed7febfabc3277f45571234bde0bf9120829a7169edc9",
@@ -104,24 +104,24 @@
     "households": {
       "us": 100
     },
-    "models": 27,
+    "models": 29,
     "output_groups": {
       "us": 18
     }
   },
-  "snapshot_date": "2026-07-24",
+  "snapshot_date": "2026-08-05",
   "source_run_artifacts": {
     "note": "Compact copies of run outputs used to verify this snapshot. The run data.json retains parsed scenario predictions, explanations, summaries, heatmaps, and PolicyEngine runtime metadata used by the dashboard. predictions.csv.gz is a deterministic gzip of the run's raw provider responses.",
     "us_full_run_20260612_policyengine_4_16_1_populace": {
       "files": {
-        "analysis/impact_summary_by_model.csv": "98bc36df93c87904dfaa3932249b1e2da82f9eecf40b5c7bc45016cc75352964",
-        "analysis/metrics.csv": "972f6b6269344657f9fb3cf60bb01bbb3928c3f6f9491ff72753da8bbfa9412c",
-        "analysis/report.md": "4e157124fe795be4b121691b9f1f39390d32cd805b293c8b6472d01b09bc1824",
-        "analysis/summary_by_model.csv": "7f816a094b8170f53769dad24139a28524f2567e7d3019b8f5e47e17f2961735",
-        "analysis/summary_by_variable.csv": "eed9d261b9b84bc6f3e681a50566d25dd18e78abd2db40fd978b2acf7f8f4e50",
-        "analysis/usage_summary.csv": "836e099c55bb94495f25511dd3b351ccd235331e87fb141610fc4f86ac468f74",
-        "data.json": "1bebdfbf4b5527664be594165a17bf66e0d5b71844c2833ba10233dd05f10760",
-        "predictions.csv.gz": "5251bfb6dd8c547d695c3022748577b37fc42ca9967468490323ba3d12b119e0",
+        "analysis/impact_summary_by_model.csv": "6de83b553e62efe29099db4a54b10b31de617d4fe0b29b97366384d3c9981213",
+        "analysis/metrics.csv": "07fb00315896f06aeef35cfba2d14dc6d61f6ff1695be4723be2b10813bec715",
+        "analysis/report.md": "8bc23f883039cedc1fc3a2100ce9b2989f8ba14349cd4846a407e44ab1fb68f9",
+        "analysis/summary_by_model.csv": "81c6f73cf4163cf9a46ac13274c9d5fefbbbf35521901330d998c7e962e83adc",
+        "analysis/summary_by_variable.csv": "3c552673ab33f2b980a8aca8bc42d9d5a1c956d8448bbaa63521246ee4356a9a",
+        "analysis/usage_summary.csv": "cf999e6e57b1e55947e18bfb950866e1bf025a2b4d183b68bea4483c30a356a0",
+        "data.json": "4396793ec0bb0c0e0652629b5874abac0299a048094b600531fd4e0e1b7d8a6c",
+        "predictions.csv.gz": "017217ff51c94b4e05812dad4701f44e456827f45c3143d23560c89c7d092d64",
         "reference_outputs.csv": "b9136a15e285f9c02ba78bee854b8a3af180e280485512642c829e8ffd7d2368",
         "reference_outputs.csv.meta.json": "4ea7911857fabf72fc0f74feab90566e9f6d98cab4958e0c1401043e58c94210",
         "scenarios.csv": "71b16212f0c0b3e5d13d8694ce57e362c23248665806c4d6dea7b23ef472858a",

paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace/analysis/impact_summary_by_model.csv

@@ -1,4 +1,5 @@
 model,mean_impact_score,mean_household_score,mean_household_coverage,households,total_variables,parsed_variables,floor_share
+inkling,0.9336310139316637,0.9592442671897276,1.0,100,1984,1984,0.3
 gpt-5.6-sol,0.9333432088386975,0.965508936901421,1.0,100,1984,1984,0.3
 gpt-5.5,0.9151482777047545,0.9545673865419815,1.0,100,1984,1984,0.3
 gpt-5.6-terra,0.9124735835366288,0.9553813177844692,1.0,100,1984,1984,0.3
@@ -21,6 +22,7 @@ claude-sonnet-5,0.8280479999964578,0.9019999168319122,1.0,100,1984,1984,0.3
 qwen-3.7-max,0.8166509151209334,0.912171757625735,1.0,100,1984,1984,0.3
 grok-4.3,0.81488413335592,0.9290008068424241,1.0,100,1984,1984,0.3
 glm-5.2,0.8030086701222728,0.8823022952687445,0.9420989929188458,100,1984,1845,0.3
+qwen3.8-max,0.7829529785198477,0.8919538907121978,1.0,100,1984,1984,0.3
 claude-haiku-4.5,0.7704368219678464,0.8955621663775267,1.0,100,1984,1984,0.3
 gpt-5.4-mini,0.7108093364654092,0.88645911887305,1.0,100,1984,1984,0.3
 minimax-m3,0.702110928790043,0.8957016969192351,1.0,100,1984,1984,0.3

paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace/analysis/metrics.csv

@@ -413,6 +413,24 @@ grok-build-0.1,ssi,100,100,1.0,173.9799853515625,0.3036488630565066,,0.95,0.95,0
 grok-build-0.1,state_income_tax_before_refundable_credits,100,100,1.0,281.92015006542204,0.9365117154521054,,0.47,0.49,0.58,0.63,0.5425,0.7510254304153742
 grok-build-0.1,state_refundable_credits,100,100,1.0,80.5593755197525,1.2140010196895343,,0.78,0.78,0.78,0.78,0.78,0.7976913648125662
 grok-build-0.1,tanf,100,100,1.0,6.00958984375,0.09908706487929027,,0.99,0.99,0.99,1.0,0.9924999999999999,0.9990091293512071
+inkling,federal_income_tax_before_refundable_credits,100,100,1.0,756.3287039550783,0.26411409588921064,,0.6,0.68,0.72,0.76,0.69,0.887557693753797
+inkling,federal_refundable_credits,100,100,1.0,101.08229339599609,0.14714227799575236,,0.87,0.92,0.92,0.94,0.9125,0.9408715038605522
+inkling,free_school_meals_eligible,100,100,1.0,0.01,,0.99,0.99,0.99,0.99,0.99,0.99,0.99
+inkling,local_income_tax,100,100,1.0,39.24,,,0.99,0.99,0.99,0.99,0.99,0.99
+inkling,payroll_tax,100,100,1.0,83.17338249511718,0.020576686257239987,,0.81,0.86,0.9,0.96,0.8825,0.9868309207953665
+inkling,person_chip_eligible,177,177,1.0,0.0,,1.0,1.0,1.0,1.0,1.0,1.0,1.0
+inkling,person_early_head_start_eligible,38,38,1.0,0.0,,1.0,1.0,1.0,1.0,1.0,1.0,1.0
+inkling,person_head_start_eligible,38,38,1.0,0.0,,1.0,1.0,1.0,1.0,1.0,1.0,1.0
+inkling,person_medicaid_eligible,177,177,1.0,0.022598870056497175,,0.9774011299435028,0.9774011299435028,0.9774011299435028,0.9774011299435028,0.9774011299435028,0.9774011299435028,0.9774011299435028
+inkling,person_medicare_eligible,177,177,1.0,0.011299435028248588,,0.9887005649717514,0.9887005649717514,0.9887005649717514,0.9887005649717514,0.9887005649717514,0.9887005649717514,0.9887005649717514
+inkling,person_wic_eligible,177,177,1.0,0.011299435028248588,,0.9887005649717514,0.9887005649717514,0.9887005649717514,0.9887005649717514,0.9887005649717514,0.9887005649717514,0.9887005649717514
+inkling,reduced_price_school_meals_eligible,100,100,1.0,0.02,,0.98,0.98,0.98,0.98,0.98,0.98,0.98
+inkling,self_employment_tax,100,100,1.0,34.43888229980468,0.0001841476840988503,,0.99,0.99,0.99,0.99,0.99,0.9899852681852721
+inkling,snap,100,100,1.0,237.71026611328125,0.7134836131632282,,0.79,0.84,0.86,0.87,0.84,0.88312928296638
+inkling,ssi,100,100,1.0,165.6199853515625,0.3229460227791829,,0.96,0.96,0.96,0.96,0.96,0.9703116193166246
+inkling,state_income_tax_before_refundable_credits,100,100,1.0,236.6320845014572,0.30778275705884695,,0.53,0.56,0.66,0.71,0.615,0.8285650578295062
+inkling,state_refundable_credits,100,100,1.0,69.85360373020173,0.7777649256962836,,0.8,0.8,0.81,0.81,0.805,0.8438426275790394
+inkling,tanf,100,100,1.0,16.44958984375,0.2712234309601872,,0.99,0.99,0.99,0.99,0.99,0.9972877656903981
 kimi-k2.6,federal_income_tax_before_refundable_credits,100,51,0.51,3432.6543930951284,0.20181370937600498,,0.4,0.4,0.44,0.44,0.42,0.4798186290623995
 kimi-k2.6,federal_refundable_credits,100,49,0.49,212.5264614307637,0.431626995340822,,0.45,0.45,0.45,0.45,0.45,0.45841865023295886
 kimi-k2.6,free_school_meals_eligible,100,93,0.93,0.021505376344086023,,0.91,0.91,0.91,0.91,0.91,0.91,0.91
@@ -485,3 +503,21 @@ qwen-3.7-max,ssi,100,100,1.0,168.9699853515625,0.4060533232990699,,0.96,0.96,0.9
 qwen-3.7-max,state_income_tax_before_refundable_credits,100,100,1.0,598.8367811811447,0.524998113269531,,0.46,0.46,0.49,0.51,0.48000000000000004,0.7140151580038158
 qwen-3.7-max,state_refundable_credits,100,100,1.0,102.47415976047516,0.9164896651497292,,0.78,0.78,0.79,0.79,0.785,0.7985066384122771
 qwen-3.7-max,tanf,100,100,1.0,60.64958984375,1.0,,0.99,0.99,0.99,0.99,0.99,0.99
+qwen3.8-max,federal_income_tax_before_refundable_credits,100,100,1.0,2935.3017075683597,1.17134340173351,,0.45,0.45,0.49,0.5,0.47250000000000003,0.6468768341609332
+qwen3.8-max,federal_refundable_credits,100,100,1.0,803.6316957153321,0.6553485788005329,,0.66,0.66,0.67,0.67,0.665,0.7061606467175635
+qwen3.8-max,free_school_meals_eligible,100,100,1.0,0.02,,0.98,0.98,0.98,0.98,0.98,0.98,0.98
+qwen3.8-max,local_income_tax,100,100,1.0,0.0,,,1.0,1.0,1.0,1.0,1.0,1.0
+qwen3.8-max,payroll_tax,100,100,1.0,532.1247353759765,0.11477986648971544,,0.71,0.74,0.8,0.85,0.775,0.9293288908751262
+qwen3.8-max,person_chip_eligible,177,177,1.0,0.07909604519774012,,0.9209039548022598,0.9209039548022598,0.9209039548022598,0.9209039548022598,0.9209039548022598,0.9209039548022598,0.9209039548022598
+qwen3.8-max,person_early_head_start_eligible,38,38,1.0,0.0,,1.0,1.0,1.0,1.0,1.0,1.0,1.0
+qwen3.8-max,person_head_start_eligible,38,38,1.0,0.13157894736842105,,0.868421052631579,0.868421052631579,0.868421052631579,0.868421052631579,0.868421052631579,0.868421052631579,0.868421052631579
+qwen3.8-max,person_medicaid_eligible,177,177,1.0,0.1751412429378531,,0.8248587570621468,0.8248587570621468,0.8248587570621468,0.8248587570621468,0.8248587570621468,0.8248587570621468,0.8248587570621468
+qwen3.8-max,person_medicare_eligible,177,177,1.0,0.03389830508474576,,0.9661016949152542,0.9661016949152542,0.9661016949152542,0.9661016949152542,0.9661016949152542,0.9661016949152542,0.9661016949152542
+qwen3.8-max,person_wic_eligible,177,177,1.0,0.10734463276836158,,0.8926553672316384,0.8926553672316384,0.8926553672316384,0.8926553672316384,0.8926553672316384,0.8926553672316384,0.8926553672316384
+qwen3.8-max,reduced_price_school_meals_eligible,100,100,1.0,0.01,,0.99,0.99,0.99,0.99,0.99,0.99,0.99
+qwen3.8-max,self_employment_tax,100,100,1.0,85.60206276855467,0.43511756292674586,,0.92,0.92,0.92,0.94,0.925,0.9551905949658603
+qwen3.8-max,snap,100,100,1.0,815.8907495117187,2.2475093494868057,,0.77,0.77,0.77,0.77,0.77,0.7841655328973512
+qwen3.8-max,ssi,100,100,1.0,348.90601464843746,0.9312523647370412,,0.96,0.96,0.96,0.96,0.96,0.9620624290578889
+qwen3.8-max,state_income_tax_before_refundable_credits,100,100,1.0,560.7825783979415,0.5826562880646626,,0.54,0.56,0.61,0.65,0.59,0.7688227302868575
+qwen3.8-max,state_refundable_credits,100,100,1.0,99.2613774728775,0.9350766735407154,,0.79,0.79,0.79,0.79,0.79,0.8036338985564498
+qwen3.8-max,tanf,100,100,1.0,93.16958984375,1.0,,0.98,0.98,0.98,0.98,0.98,0.98