Models and Markets
Election prediction helps party officials, campaign operatives, and journalists interpret campaigns in a quantitative manner. Uncertainty is key to a useful election prediction.
The forecast model has become a staple of political punditry. Popularized by the data journalist at FiveThirtyEight, the forecasting model is a statistical tool used to incorporate a number of quantitative inputs and produce a probabilistic view of all possible outcomes.
Prediction markets can be used to generate similarly probabilistic views of election outcomes by utilizing the economic forces of price discovery and risk aversion to overcome the ideological bias of self-interested traders on a binary options exchange.
Can markets predict elections better than the models? If so, under what conditions? I propose a null hypothesis of no difference in the mean Brier score of forecasting models and prediction markets for the 2018 U.S. Congressional midterm elections.
All public input data has been saved on the internet archive and can be accessed through their wayback machine.
Data manipulation is done using the R language and packages from the
The R scripts in the
/code directory can be run in sequential
order to reproduce the results. There are four scripts to perform four
- Read archived data with
- Wrangle and format with
- Evaluate predictions with
- Communicate results with
I will be using the FiveThirtyEight “classic” model to represent the best capabilities of statistical election forecasting. FiveThirtyEight has a track record of accuracy over the last decade.
[The model’s] goal is not to divine some magic formula that miraculously predicts every election. Instead, it’s to make sense of publicly available information in a rigorous and disciplined way.
To achieve this, Silver explains that most forecasting models (1) “take lots of polls, perform various types of adjustments to them, and then blend them with other kinds of empirically useful indicators to forecast each race”. Importantly, they (2) “account for the uncertainty in the forecast and simulate the election thousands of times” to generate a probabilistic forecast.
The model incorporates three types of inputs:
- Polling: District level polling, adjusted by pollster rating.
- CANTOR: polling imputation for districts without any.
- Fundamentals: Historically useful non-polling factors:
- Challenger office
- Incumbent voting
- Previous margin
- Generic ballot
From this data, the model calculates (1) the most likely split of the vote in a race, and (2) the probability distribution around this mean given proven variables of uncertainty.
FiveThirtyEight publishes two files with top-level daily predictions:
Together, there are 110,404 daily “classic” model prediction, from 470 races, with 13 variables:
- Election type
- Candidate name
- Political party
- Model version
- Probability of victory
- Expected share of the vote
- Minimum share
- Maximum share
Prediction markets generate probabilistic forecasts by crowd-sourcing the collection of data from self-interested and risk averse traders. [The efficient market hypothesis][efm] holds that asset prices reflect all available information (including forecasting models).
PredictIt is an exchange run by Victoria University of Wellington, New Zealand. The site offers a continuous double-auction exchange, where traders buy and sell shares of futures contracts tied to election outcomes. As a trader’s perception of probabilities changes, they can sell owned shares. The market equilibrium price then updates to reflect current probability.
PredictIt provided the price history in
Together, there are 44,711 daily market prices, from 118 races, with 11
- Market ID
- Market name
- Market symbol
- Contract name
- Contract symbol
- Prediction date
- Opening contract price
- Low contract price
- High contract price
- Closing contract price
- Volume of shares traded
The FiveThirtyEight model and PredictIt markets data sets were joined using the date and unique election code. The data was then pivoted to a long format, which allows us to compare each method against the ultimate binary results of the race.
Here we can see how each each race was predicted by each method highlighted by the race results.
A probabalistic prediction should find that events with a 60% probability occur 60% of the time. Here we see how many of each method’s predictions occured that frequently. Predictions with a 60% probability that occured 85% of the time are underconfident.
The Brier score allows for probablistic forecasts to be meaningfully tested with mean squared error. Using this test, there is no statistically significant difference in the respective skill scores of each predictive method.
|Test statistic||df||P value||Alternative hypothesis|
|3.14||13749||0.001691 * *||two.sided|
Welch Two Sample t-test:
method (continued below)
|mean in group market||mean in group model|