When Polymarket says 70%, does it happen 70% of the time? Calibration on 19.3M snapshots

[manja316]

One question decides whether a prediction-market price means anything: when the market says 70%, does the event happen ~70% of the time? That's calibration, and it's the single most decision-relevant property of any probabilistic forecaster.

This dataset is built to let you check it honestly. As of 2026-06-16 the archive holds:

• 19,300+ markets
• 19.3M price snapshots (15-min cadence)
• 1.9M order-book snapshots
• 80 days continuous (2026-03-28 → 2026-06-16, ongoing)

Each resolved market gives you a hard 0/1 label — which is exactly what most "is this forecaster calibrated?" questions die without.

Reliability diagram in ~30 lines (free API, no signup)

import requests, pandas as pd

BASE = "https://api.protodex.io"
markets = requests.get(f"{BASE}/markets", params={"resolved": True}).json()

rows = []
for m in markets:
    prices = requests.get(f"{BASE}/prices", params={"market_id": m["id"]}).json()
    label = m["outcome"]                 # 1 if YES resolved else 0
    for p in prices:
        rows.append((p["yes_price"], label))

df = pd.DataFrame(rows, columns=["price", "outcome"])
df["bin"] = (df["price"] * 10).clip(0, 9).astype(int)
print(df.groupby("bin")["outcome"].mean())   # compare each row to its bin midpoint -> the diagonal

Bin midpoint on x, empirical resolution rate on y, draw the 45° diagonal — the gap is the miscalibration, in probability units, no modelling assumptions.

What to look for

The favorite–longshot bias: longshots tend to be overpriced (resolve YES less than their price implies), heavy favorites slightly underpriced. If the curve sags below the diagonal at the low end and rises above it at the high end, that's the fingerprint.

The honest caveats

• Survivorship/selection — resolved markets ≠ all markets; conditioning on resolution biases some analyses. This is the part I'm least sure anyone handles cleanly.
• Liquidity varies — thin markets have stale prices; filter by order-book depth (that's why the orderbook snapshots are in the set).
• Mid ≠ executable — fees/spread are real the moment you make a trading claim.
• 80 days is one window, not multiple regimes.

Free API reproduces everything above. Flat-file archive (parquet/csv) if you'd rather not page market-by-market: Polymarket Full Dataset.

Open question I'd genuinely like input on: how do you handle the selection-bias problem on resolved-only markets? What failure modes have you hit using prediction-market data as a calibration/forecasting benchmark?