code/suite_toolkit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tools for SUITE Risk-Limiting Election Audits\n",
    "\n",
    "This Jupyter notebook implements some tools to conduct \"hybrid\" stratified risk-limiting audits as described in Risk-Limiting Audits by Stratified Union-Intersection Tests of Elections (SUITE), by Ottoboni, Stark, Lindeman, and McBurnett.\n",
    "\n",
    "For an implementation of tools for \"comparison\" risk-limiting audits as described in AGI, see http://statistics.berkeley.edu/~stark/Vote/auditTools.htm. For the sister ballot polling tool, see https://www.stat.berkeley.edu/~stark/Vote/ballotPollTools.htm.\n",
    "\n",
    "The tools on this page help perform the following steps:\n",
    "\n",
    "* Choose a number of ballots to audit in each stratum initially, on the assumption that the contest outcome is correct.\n",
    "* Select random samples of ballots in each stratum.\n",
    "* Find those ballots using ballot manifests.\n",
    "* Determine whether the audit can stop, given the votes on the ballots in the sample. \n",
    "* If the audit cannot stop yet, estimate how many additional ballots will need to be audited.\n",
    "\n",
    "This notebook is already filled out with an example election. It can be run from start to finish to demonstrate how the tool works. The numbers in the example can be deleted and replaced with actual data for an audit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Jupyter Notebooks\n",
    "\n",
    "We leave [a comprehensive introduction to the Jupyter notebook](https://jupyter-notebook.readthedocs.io/en/stable/notebook.html) to the experts, but below are a few features you should know to use this tool:\n",
    "\n",
    "* notebooks are comprised of _cells_, blocks of code that can be run together. To the left of a code cell, you will see either [] (indicating that it has not been run yet) or [x] (where x is a number indicating that it was the xth cell to be run). You can the code in a cell by clicking into the cell, indicated by a green box around the cell, and running `Ctrl + Enter`.\n",
    "* code lines that begin with `#` are comments. They're not actually run, but are there to describe what the code is doing.\n",
    "* the text in a notebook is also written in a cell. Instead of a code cell, it's a Markdown cell. Clicking on a text cell will make it editable; running `Ctrl + Enter` will render it back into text.\n",
    "* the order in which cells are executed matters. Code in later cells depends on earlier cells. However, it is _possible_ to run cells out of order or rerun cells that have been run earlier; this can cause problem. In general, it is __best practice__ to rerun the entire notebook after you have filled in the values you want. To do so, click on the `Kernel` menu at the top of the page and select `Restart & Run All`. This will clear the memory and rerun everything in the prescribed order.\n",
    "\n",
    "\n",
    "The following cell imports all the necessary functionality from packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/importlib/_bootstrap.py:321: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
    "import ipywidgets as widgets\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "from collections import OrderedDict\n",
    "from itertools import product\n",
    "import math\n",
    "import json\n",
    "import pprint\n",
    "\n",
    "import numpy as np\n",
    "from ballot_comparison import ballot_comparison_pvalue\n",
    "from fishers_combination import  maximize_fisher_combined_pvalue, create_modulus\n",
    "from sprt import ballot_polling_sprt\n",
    "\n",
    "from cryptorandom.cryptorandom import SHA256\n",
    "from cryptorandom.sample import random_sample\n",
    "\n",
    "from suite_tools import write_audit_parameters, write_audit_results, \\\n",
    "        check_valid_audit_parameters, check_valid_vote_counts, \\\n",
    "        check_overvote_rates, find_winners_losers, print_reported_votes, \\\n",
    "        estimate_n, estimate_escalation_n, \\\n",
    "        sample_from_manifest, write_ballots_to_sample, \\\n",
    "        audit_contest, check_polling_sample_size, plot_nratio_sample_sizes\n",
    "    \n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Input the global audit parameters.\n",
    "\n",
    "For an audit, you should input the following global parameters in the cell below:\n",
    "\n",
    "* contest-specific parameters:\n",
    "    * `risk_limit`: the risk limit for the audit\n",
    "    * `stratum_sizes`: total ballots in the two strata, [CVR total, no-CVR total]\n",
    "    * `num_winners`: number of winners in the contest\n",
    "* software parameters:\n",
    "    * `seed`: the numeric seed for the pseudo-random number generator used to draw samples of ballots. Use, e.g., 20 rolls of a 10-sided die \n",
    "    * `gamma`: the gamma parameter used in the ballot-polling method from Lindeman and Stark (2012). Default value of 1.03905 is generally accepted\n",
    "    * `lambda_step`: the initial step size in the grid search over the way error is allocated across the CVR and no-CVR strata in SUITE. Default 0.05 is acceptable\n",
    "* initial sample size estimate parameters:\n",
    "    * `o1_rate`: expected rate of 1-vote overstatements in the CVR stratum\n",
    "    * `o2_rate`: expected rate of 2-vote overstatements in the CVR stratum\n",
    "    * `u1_rate`: expected rate of 1-vote understatements in the CVR stratum\n",
    "    * `u2_rate`: expected rate of 2-vote understatements in the CVR stratum\n",
    "    * `n_ratio`: what fraction of the sample is taken from the CVR stratum. Default is to allocate sample in proportion to ballots cast in each stratum.\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# contest-specific parameters\n",
    "risk_limit = 0.05    # risk limit\n",
    "stratum_sizes = [100000, 5000]  # total ballots in the two strata, CVR, no-CVR\n",
    "num_winners = 2      # maximum number of winners, per social choice function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# software parameters\n",
    "seed = 12345678901234567890  # use, e.g., 20 rolls of a 10-sided die\n",
    "gamma=1.03905         # gamma from Lindeman and Stark (2012)\n",
    "lambda_step = 0.05    # stepsize for the discrete bounds on Fisher's combining function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# initial sample size parameters\n",
    "o1_rate = 0.002       # expect 2 1-vote overstatements per 1000 ballots in the CVR stratum\n",
    "o2_rate = 0           # expect 0 2-vote overstatements\n",
    "u1_rate = 0           # expect 0 1-vote understatements\n",
    "u2_rate = 0           # expect 0 2-vote understatements\n",
    "n_ratio = stratum_sizes[0]/np.sum(stratum_sizes) \n",
    "                     # allocate sample in proportion to ballots cast in each stratum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "check_valid_audit_parameters(risk_limit, lambda_step, o1_rate, o2_rate, \\\n",
    "                                 u1_rate, u2_rate, stratum_sizes, n_ratio, num_winners)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell saves the input parameters to a JSON file. You may change the file name in quotes but do not change the rest of the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "write_audit_parameters(\"audit_parameters.json\",\\\n",
    "                       risk_limit, stratum_sizes, num_winners, seed, gamma, \\\n",
    "                       lambda_step, o1_rate, o2_rate, \\\n",
    "                       u1_rate, u2_rate, n_ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Enter the reported votes\n",
    "\n",
    "Candidates are stored in a data structure called a dictionary. Enter the candidate name and the votes in each stratum, [votes in CVR stratum, votes in no-CVR stratum], in the cell below. The following cell will calculate the vote totals, margins, winners, and losers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# input number of winners\n",
    "# input names as well as reported votes in each stratum\n",
    "\n",
    "# candidates are a dict with name, [votes in CVR stratum, votes in no-CVR stratum]\n",
    "candidates = { \"candidate 3\": [30000, 500],\n",
    "               \"candidate 2\": [50000, 1000],\n",
    "               \"candidate 1\": [10000, 500],\n",
    "               \"candidate 4\": [500, 10]}\n",
    "\n",
    "# Run validity check on the input vote totals\n",
    "check_valid_vote_counts(candidates, num_winners, stratum_sizes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# compute reported winners, losers, and pairwise margins. Nothing should be printed.\n",
    "(candidates, margins, winners, losers) = find_winners_losers(candidates, num_winners)\n",
    "  \n",
    "# Check that overstatement rates are compatible with the reported results\n",
    "check_overvote_rates(margins=margins, total_votes=sum(stratum_sizes), \n",
    "                     o1_rate=o1_rate, o2_rate=o2_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total reported votes:\n",
      "\t\t\tCVR\tno-CVR\ttotal\t% of all votes\t% of valid votes\n",
      "\t candidate 2 : 50000 \t 1000 \t 51000 \t 48.57% \t 55.13%\n",
      "\t candidate 3 : 30000 \t 500 \t 30500 \t 29.05% \t 32.97%\n",
      "\t candidate 1 : 10000 \t 500 \t 10500 \t 10.00% \t 11.35%\n",
      "\t candidate 4 : 500 \t 10 \t 510 \t 0.49% \t 0.55%\n",
      "\n",
      "\t valid votes:\t 90500 \t 2010 \t 92510 \t 88.10%\n",
      "\n",
      "\t non-votes:\t 9500 \t 2990 \t 12490 \t 11.90%\n",
      "\n",
      "Reported winners:\n",
      "\t candidate 2\n",
      "\t candidate 3\n",
      "\n",
      "Reported losers:\n",
      "\t candidate 1\n",
      "\t candidate 4\n",
      "\n",
      "\n",
      "Reported margins:\n",
      "\t candidate 2 beat candidate 4 by 50,490 votes\n",
      "\t candidate 2 beat candidate 1 by 40,500 votes\n",
      "\t candidate 3 beat candidate 4 by 29,990 votes\n",
      "\t candidate 3 beat candidate 1 by 20,000 votes\n",
      "\n",
      "Smallest reported margin: 20,000 \n",
      "Corresponding reported diluted margin: 19.05%\n"
     ]
    }
   ],
   "source": [
    "# print reported winners, losers, and pairwise margins\n",
    "print_reported_votes(candidates, winners, losers, margins, stratum_sizes,\\\n",
    "                     print_alphabetical=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initial sample size estimates.\n",
    "\n",
    "The initial sample size tool helps you anticipate the number of randomly selected ballots that might need to be inspected to attain a given limit on the risk, under the assumption that the reported percentages for each candidate are correct. \n",
    "\n",
    "It is completely legitimate to sample one at a time and rerun the SUITE calculations, but this form can help auditors anticipate how many ballots the audit is likely to require and to retrieve ballots more efficiently.\n",
    "\n",
    "This code will estimate the sample size needed to attain the desired risk limit in an audit of the contest between each pair of winning and losing candidates. The overall sample size will be allocated to the CVR stratum in `n_ratio` proportion and to the no-CVR stratum in `1-n_ratio` proportion. The sample size estimates for each pair will be printed below. The expected sample size needed for the audit is the _maximum_ of the sample sizes for each winner, loser pair: the sample must be large enough to confirm the closest margin.\n",
    "\n",
    "Taking a larger initial sample can avoid needing to expand the sample later, depending on the rate of ballots for each candidate in the sample. Avoiding \"escalation\" can make the audit less complicated.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Calculate expected sample size across (winner, loser) pairs\n",
    "\n",
    "sample_sizes = {}\n",
    "\n",
    "for k in product(winners, losers):\n",
    "    sample_sizes[k] = estimate_n(N_w1 = candidates[k[0]][0],\\\n",
    "                                 N_w2 = candidates[k[0]][1],\\\n",
    "                                 N_l1 = candidates[k[1]][0],\\\n",
    "                                 N_l2 = candidates[k[1]][1],\\\n",
    "                                 N1 = stratum_sizes[0],\\\n",
    "                                 N2 = stratum_sizes[1],\\\n",
    "                                 o1_rate = o1_rate,\\\n",
    "                                 o2_rate = o2_rate,\\\n",
    "                                 u1_rate = u1_rate,\\\n",
    "                                 u2_rate = u2_rate,\\\n",
    "                                 n_ratio = n_ratio,\\\n",
    "                                 risk_limit = risk_limit,\\\n",
    "                                 gamma = gamma,\\\n",
    "                                 stepsize = lambda_step,\\\n",
    "                                 min_n = 5,\\\n",
    "                                 risk_limit_tol = 0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "estimated sample sizes for each contest, written as (cvr stratum, no-cvr stratum):\n",
      "\n",
      "{('candidate 2', 'candidate 1'): (30, 1),\n",
      " ('candidate 2', 'candidate 4'): (23, 1),\n",
      " ('candidate 3', 'candidate 1'): (67, 3),\n",
      " ('candidate 3', 'candidate 4'): (45, 2)}\n",
      "\n",
      "\n",
      "expected total sample size needed to confirm all pairs: 70\n"
     ]
    }
   ],
   "source": [
    "sample_size = np.amax([v[0]+v[1] for v in sample_sizes.values()])\n",
    "\n",
    "print(\"estimated sample sizes for each contest, written as (cvr stratum, no-cvr stratum):\\n\")\n",
    "pprint.pprint(sample_sizes)\n",
    "print('\\n\\nexpected total sample size needed to confirm all pairs:', sample_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "expected minimum sample size that would be needed using ballot polling ONLY, for all ballots: 69\n"
     ]
    }
   ],
   "source": [
    "check_polling_sample_size(candidates, winners, losers, stratum_sizes, risk_limit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Run this cell to plot the total size as a function of n_ratio\n",
    "\n",
    "#plot_nratio_sample_sizes(candidates, winners, losers, stratum_sizes, n_ratio_step=0.05, o1_rate=o1_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next tool helps generate pseudo-random samples of ballots in each stratum. Further below, there is a form to help find the individual, randomly selected ballots among the batches in which ballots are stored.\n",
    "\n",
    "The first cell below initializes the SHA-256 cryptographically secure pseudo-random number generator. Details on why you might want to use this pseudo-random number generator instead of the Python default can be found in [Stark and Ottoboni (2018)](https://arxiv.org/abs/1810.10985). \n",
    "\n",
    "Input your desired sample sizes in the second cell below. Input the number of ballots you want in the sample. The default values that are pre-filled are taken from the initial sample size estimates above. \n",
    "\n",
    "The third cell should not be modified. It draws the samples from each stratum, using sampling _with_ replacement for the CVR stratum and sampling _without_ replacement for the no-CVR stratum. This means that some ballots in the CVR stratum could be sampled more than once.\n",
    "\n",
    "\n",
    "**NOTE:**\n",
    "If this section is giving errors, you probably need to update your version of `cryptorandom`.\n",
    "\n",
    "```\n",
    "pip install [--update] cryptorandom\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# initialize the PRNG\n",
    "prng = SHA256(seed)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Input the sample sizes for each stratum. \n",
    "# Defaults to those found using the initial sample size tool above.\n",
    "n1 = math.ceil(sample_size*n_ratio)    \n",
    "n2 = sample_size-n1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# CVR stratum initial sample size, sampled with replacement\n",
    "sample1 = prng.randint(1, stratum_sizes[0]+1, size=n1)\n",
    "\n",
    "# No-CVR ballots are sampled without replacement\n",
    "sample2 = random_sample(stratum_sizes[1], size=n2, replace=False, prng=prng)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CVR stratum sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CVR stratum sample:\n",
      " [76116 45424 33501 45326  2081 56264 25122 16602 79743 61814 57922 41676\n",
      " 95332 38891 17757 64352 84257 47365 10908 97791 77941 73573 51855 88527\n",
      " 35549 20934 61419 70683 70220 45067 67903 94304 20823 50570 88735  9973\n",
      " 44578 34320  8262 32532 85102 87511 63375 96612 52917 91127 84152 74227\n",
      " 76674 76640 62444 83868  3974 81503 82205 41161 28136 12244 97608  9057\n",
      " 43082  6522 39347 45600 57836 10233 75516]\n"
     ]
    }
   ],
   "source": [
    "print(\"CVR stratum sample:\\n\", sample1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CVR stratum repeated ballots:\n",
      " []\n"
     ]
    }
   ],
   "source": [
    "m = np.zeros_like(sample1, dtype=bool)\n",
    "m[np.unique(sample1, return_index=True)[1]] = True\n",
    "print(\"CVR stratum repeated ballots:\\n\", sample1[~m])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### No-CVR sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No-CVR stratum sample:\n",
      " [1782 4274 4914]\n"
     ]
    }
   ],
   "source": [
    "print(\"No-CVR stratum sample:\\n\", sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Find ballots using ballot manifest\n",
    "\n",
    "Generally, ballots will be stored in batches, for instance, separated by precinct and mode of voting. To make it easier to find individual ballots, it helps to have a ballot manifest that describes how the ballots are stored. \n",
    "\n",
    "\n",
    "Batch label\t| ballots\n",
    "--- |  ---\n",
    "Polling place precinct 1  |\t130\n",
    "Vote by mail precinct 1\t  | 172\n",
    "Polling place precinct 2  | 112\n",
    "Vote by mail precinct 2\t  | 201\n",
    "Polling place precinct 3  | 197\n",
    "Vote by mail precinct 3   | 188\n",
    "\n",
    "If ballot 500 is selected for audit, which ballot is that? If we take the listing of batches in the order given by the manifest, and we require that within each batch, the ballots are in an order that does not change during the audit, then the 500th ballot is the 86th ballot among the vote by mail ballots for precinct 2: The first three batches have a total of 130+172+112 = 414 ballots. The first ballot in the fourth batch is ballot 415. Ballot 500 is the 86th ballot in the fourth batch. The ballot look-up tool transforms a list of ballot numbers and a ballot manifest into a list of ballots in each batch.\n",
    "\n",
    "There must be separate ballot manifests for ballots in the CVR stratum and for ballots in the no-CVR stratum. The manifests should be input as a CSV file with three columns: Batch ID, Scanner ID, and number of ballots.\n",
    "\n",
    "The total number of ballots in the manifest must equal the number cast in the contest that is to be audited using the sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cvr_sample =  sample_from_manifest(filename=\"sample-cvr-ballot-manifest.csv\", \\\n",
    "                                   sample=sample1, \\\n",
    "                                   stratum_size=stratum_sizes[0])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "write_ballots_to_sample(\"sampled_cvr_ballots.csv\", cvr_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CVR sample\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table><tr><td>sampled ballot</td><td>batch label</td><td>which ballot in batch</td><td># times sampled</td></tr><tr><td>2081</td><td>Pct 6 - Batch 1</td><td>2081</td><td>1</td></tr><tr><td>3974</td><td>Pct 6 - Batch 1</td><td>3974</td><td>1</td></tr><tr><td>6522</td><td>Pct 6 - Batch 1</td><td>6522</td><td>1</td></tr><tr><td>8262</td><td>Pct 6 - Batch 1</td><td>8262</td><td>1</td></tr><tr><td>9057</td><td>Pct 6 - Batch 1</td><td>9057</td><td>1</td></tr><tr><td>9973</td><td>Pct 6 - Batch 1</td><td>9973</td><td>1</td></tr><tr><td>10233</td><td>Pct 6 - Batch 3</td><td>231</td><td>1</td></tr><tr><td>10908</td><td>Pct 6 - Batch 3</td><td>906</td><td>1</td></tr><tr><td>12244</td><td>Pct 6 - Batch 3</td><td>2242</td><td>1</td></tr><tr><td>16602</td><td>Pct 6 - Batch 3</td><td>6600</td><td>1</td></tr><tr><td>17757</td><td>Pct 6 - Batch 3</td><td>7755</td><td>1</td></tr><tr><td>20823</td><td>Pct 6 - Batch 3</td><td>10821</td><td>1</td></tr><tr><td>20934</td><td>Pct 6 - Batch 3</td><td>10932</td><td>1</td></tr><tr><td>25122</td><td>Pct 6 - Batch 3</td><td>15120</td><td>1</td></tr><tr><td>28136</td><td>Pct 6 - Batch 3</td><td>18134</td><td>1</td></tr><tr><td>32532</td><td>Pct 6 - Batch 3</td><td>22530</td><td>1</td></tr><tr><td>33501</td><td>Pct 6 - Batch 3</td><td>23499</td><td>1</td></tr><tr><td>34320</td><td>Pct 6 - Batch 3</td><td>24318</td><td>1</td></tr><tr><td>35549</td><td>Pct 6 - Batch 3</td><td>25547</td><td>1</td></tr><tr><td>38891</td><td>Pct 6 - Batch 3</td><td>28889</td><td>1</td></tr><tr><td>39347</td><td>Pct 6 - Batch 3</td><td>29345</td><td>1</td></tr><tr><td>41161</td><td>Pct 6 - Batch 3</td><td>31159</td><td>1</td></tr><tr><td>41676</td><td>Pct 6 - Batch 3</td><td>31674</td><td>1</td></tr><tr><td>43082</td><td>Pct 6 - Batch 3</td><td>33080</td><td>1</td></tr><tr><td>44578</td><td>Pct 6 - Batch 3</td><td>34576</td><td>1</td></tr><tr><td>45067</td><td>Pct 6 - Batch 3</td><td>35065</td><td>1</td></tr><tr><td>45326</td><td>Pct 6 - Batch 3</td><td>35324</td><td>1</td></tr><tr><td>45424</td><td>Pct 6 - Batch 3</td><td>35422</td><td>1</td></tr><tr><td>45600</td><td>Pct 6 - Batch 3</td><td>35598</td><td>1</td></tr><tr><td>47365</td><td>Pct 6 - Batch 3</td><td>37363</td><td>1</td></tr><tr><td>50570</td><td>Pct 6 - Batch 3</td><td>40568</td><td>1</td></tr><tr><td>51855</td><td>Pct 6 - Batch 3</td><td>41853</td><td>1</td></tr><tr><td>52917</td><td>Pct 6 - Batch 3</td><td>42915</td><td>1</td></tr><tr><td>56264</td><td>Pct 6 - Batch 3</td><td>46262</td><td>1</td></tr><tr><td>57836</td><td>Pct 6 - Batch 3</td><td>47834</td><td>1</td></tr><tr><td>57922</td><td>Pct 6 - Batch 3</td><td>47920</td><td>1</td></tr><tr><td>61419</td><td>Pct 6 - Batch 3</td><td>51417</td><td>1</td></tr><tr><td>61814</td><td>Pct 6 - Batch 3</td><td>51812</td><td>1</td></tr><tr><td>62444</td><td>Pct 6 - Batch 3</td><td>52442</td><td>1</td></tr><tr><td>63375</td><td>Pct 6 - Batch 3</td><td>53373</td><td>1</td></tr><tr><td>64352</td><td>Pct 6 - Batch 3</td><td>54350</td><td>1</td></tr><tr><td>67903</td><td>Pct 6 - Batch 3</td><td>57901</td><td>1</td></tr><tr><td>70220</td><td>Pct 6 - Batch 3</td><td>60218</td><td>1</td></tr><tr><td>70683</td><td>Pct 6 - Batch 3</td><td>60681</td><td>1</td></tr><tr><td>73573</td><td>Pct 6 - Batch 3</td><td>63571</td><td>1</td></tr><tr><td>74227</td><td>Pct 6 - Batch 3</td><td>64225</td><td>1</td></tr><tr><td>75516</td><td>Pct 6 - Batch 3</td><td>65514</td><td>1</td></tr><tr><td>76116</td><td>Pct 6 - Batch 3</td><td>66114</td><td>1</td></tr><tr><td>76640</td><td>Pct 6 - Batch 3</td><td>66638</td><td>1</td></tr><tr><td>76674</td><td>Pct 6 - Batch 3</td><td>66672</td><td>1</td></tr><tr><td>77941</td><td>Pct 6 - Batch 3</td><td>67939</td><td>1</td></tr><tr><td>79743</td><td>Pct 6 - Batch 3</td><td>69741</td><td>1</td></tr><tr><td>81503</td><td>Pct 6 - Batch 3</td><td>71501</td><td>1</td></tr><tr><td>82205</td><td>Pct 6 - Batch 3</td><td>72203</td><td>1</td></tr><tr><td>83868</td><td>Pct 6 - Batch 3</td><td>73866</td><td>1</td></tr><tr><td>84152</td><td>Pct 6 - Batch 3</td><td>74150</td><td>1</td></tr><tr><td>84257</td><td>Pct 6 - Batch 3</td><td>74255</td><td>1</td></tr><tr><td>85102</td><td>Pct 6 - Batch 3</td><td>75100</td><td>1</td></tr><tr><td>87511</td><td>Pct 6 - Batch 3</td><td>77509</td><td>1</td></tr><tr><td>88527</td><td>Pct 6 - Batch 3</td><td>78525</td><td>1</td></tr><tr><td>88735</td><td>Pct 6 - Batch 3</td><td>78733</td><td>1</td></tr><tr><td>91127</td><td>Pct 6 - Batch 3</td><td>81125</td><td>1</td></tr><tr><td>94304</td><td>Pct 6 - Batch 3</td><td>84302</td><td>1</td></tr><tr><td>95332</td><td>Pct 6 - Batch 3</td><td>85330</td><td>1</td></tr><tr><td>96612</td><td>Pct 6 - Batch 3</td><td>86610</td><td>1</td></tr><tr><td>97608</td><td>Pct 6 - Batch 3</td><td>87606</td><td>1</td></tr><tr><td>97791</td><td>Pct 6 - Batch 3</td><td>87789</td><td>1</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"CVR sample\")\n",
    "display(HTML(\n",
    "    '<table><tr>{}</tr></table>'.format(\n",
    "        '</tr><tr>'.join(\n",
    "            '<td>{}</td>'.format('</td><td>'.join(str(_) for _ in row)) for row in cvr_sample)\n",
    "        )\n",
    " ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nocvr_sample =  sample_from_manifest(filename=\"sample-nocvr-ballot-manifest.csv\", \\\n",
    "                                   sample=sample2, \\\n",
    "                                   stratum_size=stratum_sizes[1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "write_ballots_to_sample(\"sampled_nocvr_ballots.csv\", nocvr_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No CVR sample\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table><tr><td>sampled ballot</td><td>batch label</td><td>which ballot in batch</td><td># times sampled</td></tr><tr><td>1782</td><td>Pct 6 - Batch 2</td><td>782</td><td>1</td></tr><tr><td>4274</td><td>Pct 6 - Batch 2</td><td>3274</td><td>1</td></tr><tr><td>4914</td><td>Pct 6 - Batch 2</td><td>3914</td><td>1</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"No CVR sample\")\n",
    "display(HTML(\n",
    "    '<table><tr>{}</tr></table>'.format(\n",
    "        '</tr><tr>'.join(\n",
    "            '<td>{}</td>'.format('</td><td>'.join(str(_) for _ in row)) for row in nocvr_sample)\n",
    "        )\n",
    " ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Enter the sample data\n",
    "\n",
    "The audit cannot stop until **all** the sampled ballots have been examined."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample statistics for the CVR stratum (stratum 1)\n",
    "\n",
    "Enter the number of 1-vote and 2-vote over-/understatements that were observed in the sample using the sliders below, then run the cell beneath the sliders to store the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sample size in the CVR stratum was 67\n"
     ]
    }
   ],
   "source": [
    "print(\"The sample size in the CVR stratum was\", n1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 0, 0, 0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def cvr_audit_inputs(o1, o2, u1, u2):\n",
    "    return (o1, o2, u1, u2)\n",
    "\n",
    "cvr_stats = interactive(cvr_audit_inputs,\n",
    "                             o1 = widgets.IntSlider(min=0,max=n1,value=0),\n",
    "                             u1 = widgets.IntSlider(min=0,max=n1,value=0),\n",
    "                             o2 = widgets.IntSlider(min=0,max=n1,value=0),\n",
    "                             u2 = widgets.IntSlider(min=0,max=n1,value=0))\n",
    "display(cvr_stats)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "(o1, o2, u1, u2) = [cvr_stats.children[i].value for i in range(4)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample statistics for the no-CVR stratum (stratum 2)\n",
    "\n",
    "Enter the number of ballots for each candidate that were observed in the sample using the sliders below, then run the cell beneath the sliders to store the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sample size in the no-CVR stratum was 3\n"
     ]
    }
   ],
   "source": [
    "print(\"The sample size in the no-CVR stratum was\", n2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nocvr_widgets=[]\n",
    "\n",
    "# create the widgets\n",
    "for name in candidates.keys():\n",
    "    nocvr_widgets.append(widgets.IntSlider(value=0,min=0,max=n2,description=name))\n",
    "\n",
    "# group the widgets into a FlexBox\n",
    "nocvr_audit_inputs = widgets.VBox(children=nocvr_widgets)\n",
    "\n",
    "# display the widgets\n",
    "display(nocvr_audit_inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'candidate 1': 0, 'candidate 2': 0, 'candidate 3': 0, 'candidate 4': 0}\n"
     ]
    }
   ],
   "source": [
    "# no-CVR sample is stored in a dict with name, votes in the sample\n",
    "\n",
    "observed_poll = {}\n",
    "for widget in nocvr_widgets:\n",
    "    observed_poll[widget.description] = widget.value\n",
    "\n",
    "assert np.sum(list(observed_poll.values())) <= n2, \"Too many ballots input\"\n",
    "pprint.pprint(observed_poll)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What's the risk for this sample?\n",
    "\n",
    "The audit looks at every (winner, loser) pair in each contest. Auditing continues until there is strong evidence that every winner in a contest got more votes than every loser in the contest. It does this by considering (winner, loser) pairs. The SUITE risk for every pair will appear beneath the cell below after it is run. The audit continues until all the numbers are not larger than the risk limit. E.g., if the risk limit is 10%, the audit stops when the numbers in the table are all less than 0.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{('candidate 2', 'candidate 1'): 1.60720828293881e-05,\n",
      " ('candidate 2', 'candidate 4'): 3.4873757048803355e-07,\n",
      " ('candidate 3', 'candidate 1'): 0.01204121065144581,\n",
      " ('candidate 3', 'candidate 4'): 0.0004700828165858928}\n"
     ]
    }
   ],
   "source": [
    "# Find audit p-values across (winner, loser) pairs\n",
    "\n",
    "audit_pvalues = audit_contest(candidates, winners, losers, stratum_sizes, \\\n",
    "                  n1, n2, o1, o2, u1, u2, observed_poll, \\\n",
    "                  risk_limit=risk_limit, gamma=gamma, stepsize=lambda_step)\n",
    "pprint.pprint(audit_pvalues)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pairs not yet confirmed:\n",
      " []\n"
     ]
    }
   ],
   "source": [
    "# Track contests not yet confirmed\n",
    "\n",
    "contests_not_yet_confirmed = [i[0] for i in audit_pvalues.items() \\\n",
    "                              if i[1]>risk_limit]\n",
    "print(\"Pairs not yet confirmed:\\n\", contests_not_yet_confirmed)\n",
    "\n",
    "winners_not_yet_confirmed = list(set(list(map(lambda x: x[0], contests_not_yet_confirmed))))\n",
    "losers_not_yet_confirmed = list(set(list(map(lambda x: x[1], contests_not_yet_confirmed))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Save everything to file, you may change the file name in quotes\n",
    "\n",
    "\n",
    "write_audit_results(\"audit_results.json\", \\\n",
    "                        n1, n2, sample1, sample2, \\\n",
    "                        o1, o2, u1, u2, observed_poll, \\\n",
    "                        audit_pvalues, prng.getstate())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Escalation guidance: how many more ballots should be drawn?\n",
    "\n",
    "This tool estimates how many more ballots should be examined to confirm any remaining contests. The enlarged sample size is based on the following:\n",
    "* ballots that have already been sampled\n",
    "* assumption that we will continue to see overstatements and understatements at the same rate that they've been observed in the sample so far\n",
    "* assumption that vote proportions in the ballot-polling stratum will reflect the reported proportions\n",
    "\n",
    "Given these additional numbers, return to the sampling tool and draw additional ballots, find them with the ballot manifest tool, update the observed sample values, and rerun the SUITE risk calculations. Additional code cells to do this are included below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sample_sizes_new = {}\n",
    "\n",
    "# Add a reminder note about the candidate dict structure.\n",
    "\n",
    "for k in contests_not_yet_confirmed:\n",
    "    sample_sizes_new[k] = estimate_escalation_n(\\\n",
    "                                 N_w1 = candidates[k[0]][0],\\\n",
    "                                 N_w2 = candidates[k[0]][1],\\\n",
    "                                 N_l1 = candidates[k[1]][0],\\\n",
    "                                 N_l2 = candidates[k[1]][1],\\\n",
    "                                 N1 = stratum_sizes[0],\\\n",
    "                                 N2 = stratum_sizes[1],\\\n",
    "                                 n1 = n1,\\\n",
    "                                 n2 = n2,\\\n",
    "                                 o1_obs = o1,\\\n",
    "                                 o2_obs = o2,\\\n",
    "                                 u1_obs = u1,\\\n",
    "                                 u2_obs = u2,\\\n",
    "                                 n2l_obs = observed_poll[k[1]],\\\n",
    "                                 n2w_obs = observed_poll[k[0]],\\\n",
    "                                 n_ratio = n_ratio,\\\n",
    "                                 risk_limit = risk_limit,\\\n",
    "                                 gamma = gamma,\\\n",
    "                                 stepsize = lambda_step,\n",
    "                                 risk_limit_tol = 0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sample_size_new = np.amax([v[0]+v[1] for v in sample_sizes_new.values()])\n",
    "n1_new = np.amax([v[0] for v in sample_sizes_new.values()])\n",
    "n2_new = np.amax([v[1] for v in sample_sizes_new.values()])\n",
    "\n",
    "\n",
    "print(\"estimated sample sizes for each contest, written as (cvr stratum, no-cvr stratum):\\n\")\n",
    "pprint.pprint(sample_sizes_new)\n",
    "print('\\n\\nexpected total sample size needed to confirm remaining pairs:', sample_size_new)\n",
    "print(\"\\nDraw this many additional ballots in the CVR stratum:\", n1_new - n1)\n",
    "print(\"Draw this many additional ballots in the no-CVR stratum:\", n2_new - n2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Draw additional ballots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# print the current state of the PRNG after drawing the initial samples\n",
    "print(prng) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# CVR stratum sample size, sampled with replacement\n",
    "sample1 = prng.randint(1, stratum_sizes[0]+1, size=n1_new - n1)\n",
    "\n",
    "# No-CVR ballots are sampled without replacement\n",
    "remaining_ballots = [i for i in range(stratum_sizes[1]) if i not in sample2]\n",
    "sample2 = random_sample(remaining_ballots, size=n2_new - n2, replace=False, prng=prng)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CVR stratum sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"CVR stratum sample:\\n\", sample1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "m = np.zeros_like(sample1, dtype=bool)\n",
    "m[np.unique(sample1, return_index=True)[1]] = True\n",
    "print(\"CVR stratum repeated ballots:\\n\", sample1[~m])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### No-CVR sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"No-CVR stratum sample:\\n\", sample2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Find ballots using ballot manifest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cvr_sample = sample_from_manifest(filename=\"sample-cvr-ballot-manifest.csv\", \\\n",
    "                                   sample=sample1, \\\n",
    "                                   stratum_size=stratum_sizes[0])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "write_ballots_to_sample(\"sampled_cvr_ballots_2.csv\", cvr_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"CVR sample\")\n",
    "display(HTML(\n",
    "    '<table><tr>{}</tr></table>'.format(\n",
    "        '</tr><tr>'.join(\n",
    "            '<td>{}</td>'.format('</td><td>'.join(str(_) for _ in row)) for row in cvr_sample)\n",
    "        )\n",
    " ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nocvr_sample =  sample_from_manifest(filename=\"sample-nocvr-ballot-manifest.csv\", \\\n",
    "                                   sample=sample2, \\\n",
    "                                   stratum_size=stratum_sizes[1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "write_ballots_to_sample(\"sampled_nocvr_ballots_2.csv\", nocvr_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(\"No CVR sample\")\n",
    "display(HTML(\n",
    "    '<table><tr>{}</tr></table>'.format(\n",
    "        '</tr><tr>'.join(\n",
    "            '<td>{}</td>'.format('</td><td>'.join(str(_) for _ in row)) for row in nocvr_sample)\n",
    "        )\n",
    " ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Enter the data from the *combined* sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample statistics for the CVR stratum (stratum 1).\n",
    "Update the numbers below to include what was seen in the initial sample PLUS what was seen in the new sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"The initial sample size in the CVR stratum was\", n1, \\\n",
    "      \"and the new sample size was\", n1_new)\n",
    "print(\"The observed overstatements and understatements from the original sample were\")\n",
    "pprint.pprint({\"o1\" : o1, \"o2\" : o2, \"u1\" : u1, \"u2\" : u2})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Number of observed...\n",
    "\n",
    "def cvr_audit_inputs(o1, o2, u1, u2):\n",
    "    return (o1, o2, u1, u2)\n",
    "\n",
    "cvr_stats = interactive(cvr_audit_inputs,\n",
    "                             o1 = widgets.IntSlider(min=0,max=n1_new,value=0),\n",
    "                             u1 = widgets.IntSlider(min=0,max=n1_new,value=0),\n",
    "                             o2 = widgets.IntSlider(min=0,max=n1_new,value=0),\n",
    "                             u2 = widgets.IntSlider(min=0,max=n1_new,value=0))\n",
    "display(cvr_stats)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "(o1, o2, u1, u2) = [cvr_stats.children[i].value for i in range(4)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample statistics for the no-CVR stratum (stratum 2)\n",
    "Update the numbers below to include what was seen in the initial sample PLUS what was seen in the new sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"The initial sample size in the no-CVR stratum was\", n2, \\\n",
    "      \"and the new sample size was\", n2_new)\n",
    "print(\"The observed tallies from the original sample were\")\n",
    "pprint.pprint(observed_poll)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nocvr_widgets=[]\n",
    "\n",
    "# create the widgets\n",
    "for name in candidates.keys():\n",
    "    nocvr_widgets.append(widgets.IntSlider(value=0,min=0,max=n2_new,description=name))\n",
    "\n",
    "# group the widgets into a FlexBox\n",
    "nocvr_audit_inputs = widgets.VBox(children=nocvr_widgets)\n",
    "\n",
    "# display the widgets\n",
    "display(nocvr_audit_inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# no-CVR sample is stored in a dict with name, votes in the sample\n",
    "\n",
    "observed_poll = {}\n",
    "for widget in nocvr_widgets:\n",
    "    observed_poll[widget.description] = widget.value\n",
    "\n",
    "assert np.sum(list(observed_poll.values())) <= n2_new, \"Too many ballots input\"\n",
    "pprint.pprint(observed_poll)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What's the risk for this sample?\n",
    "\n",
    "The audit looks at every (winner, loser) pair in each contest. Auditing continues until there is strong evidence that every winner in a contest got more votes than every loser in the contest. It does this by considering (winner, loser) pairs. The SUITE risk for every pair will appear beneath the cell below after it is run. The audit continues until all the numbers are not larger than the risk limit. E.g., if the risk limit is 10%, the audit stops when the numbers in the table are all less than 0.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Find audit p-values across (winner, loser) pairs\n",
    "\n",
    "audit_pvalues = audit_contest(candidates, winners_not_yet_confirmed, \\\n",
    "                              losers_not_yet_confirmed, stratum_sizes, \\\n",
    "                              n1_new, n2_new, o1, o2, u1, u2, observed_poll, \\\n",
    "                              risk_limit=risk_limit, gamma=gamma, stepsize=lambda_step)\n",
    "pprint.pprint(audit_pvalues)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Track contests not yet confirmed\n",
    "\n",
    "contests_not_yet_confirmed = [i[0] for i in audit_pvalues.items() \\\n",
    "                              if i[1]>risk_limit]\n",
    "print(\"Pairs not yet confirmed:\\n\", contests_not_yet_confirmed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Save everything to file, you may change the file name in quotes\n",
    "\n",
    "write_audit_results(\"audit_results2.json\", \\\n",
    "                        n1_new, n2_new, sample1, sample2, \\\n",
    "                        o1, o2, u1, u2, observed_poll, \\\n",
    "                        audit_pvalues, prng.getstate())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  },
  "widgets": {
   "state": {
    "64630030d50646c8a7d130d5b18070de": {
     "views": [
      {
       "cell_index": 39
      }
     ]
    },
    "c949056a2e57434e829c135a1cccbbad": {
     "views": [
      {
       "cell_index": 43
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}