notebooks/The NIPS Experiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The NIPS Experiment\n",
    "\n",
    "### Corinna Cortes and Neil D. Lawrence\n",
    "\n",
    "### 28th December 2014\n",
    "\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this notebook we perform some preliminary analysis on 'The NIPS Experiment'. This was an experiment to determine the consistency of the review process. After receiving papers we selected 10% that would be independently rereviewed. The idea was to determine how consistent the decisions between the two sets of independent papers would be. In 2014 NIPS received 1678 submissions and we selected 170 for the experiment. These papers are referred to below as 'duplicated papers'.\n",
    "\n",
    "To run the experiment we created two separate committees within the NIPS program committee. The idea was that the two separate committees would review each duplicated paper independently and results compared.\n",
    "\n",
    "\n",
    "## Results and Analysis\n",
    "\n",
    "The final results of the experiment were as follows. From 170 papers 4 had to be withdrawn or were rejected without completing the review process, for the remainder, the 'confusion matrix' for the two committee's decisions is below.\n",
    "\n",
    "<table>\n",
    "  <tr>\n",
    "  <td colspan=\"2\"></td><td colspan=\"2\">Committee 1</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "  <td colspan=\"2\"></td><td>Accept</td><td>Reject</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td rowspan=\"2\">Committee 2</td><td>Accept</td><td>22</td><td>22</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>Reject</td><td>21</td><td>101</td>\n",
    "  </tr>\n",
    "</table>\n",
    "\n",
    "## Summarizing the Table\n",
    "\n",
    "There are a few ways of summarizing the numbers in this table as percent or probabilities. First of all, the *inconsistency*, the proportion of decisions that were not the same across the two committees. The decisions were inconsistent for 43 out of 166 papers or 0.259 as a proportion.  This number is perhaps a natural way of summarizing the figures if you are submitting your paper and wish to know an estimate of what the probability is that your paper would have different decisons according to the different committes. Secondly, the *accept precision*: if you are attending the conference and looking at any given paper, then you might want to know the probability that the paper would have been rejected in an independent rerunning of the conference.  We can estimate this for Committee 1's conference as 22/(22 + 22) = 0.5 (50%) and for Committee 2's conference as 21/(22+21) = 0.49 (49%). Averaging the two estimates gives us 49.5%. Finally, the *reject precision*: if your paper was rejected from the conference, you might like an estimate of the probability that the same paper would be rejected again if the review process had been independently rerun. That estimate is 101/(22+101) = 0.82 (82%) for Committee 1 and 101/(21+101)=0.83 (83%) for Committee 2, or on average 82.5%. A final quality estimate might be the ratio of consistent accepts to consistent rejects, or the *agreed accept rate*, 22/123 = 0.18 (18%). \n",
    "\n",
    "### Aide Memoire\n",
    "\n",
    "There seems to have been a lot of discussion of the result, both at the conference and on bulletin boards since. Such discussion is to be encouraged, and for ease of memory, it is worth pointing out that the approximate proportions of papers in each category can be nicely divided in to eigths as follows. Accept-Accept 1 in 8 papers, Accept-Reject 3 in 8 papers, Reject-Reject, 5 in 8 papers. This makes the statistics we've computed above: *inconsistency* 1 in 4 (25%) *accept precision* 1 in 2 (50%) *reject precision* 5 in 6 (83%) and *agreed accept rate* of 1 in 6 (20%). This compares with the accept rate of 1 in 4. \n",
    "\n",
    "### Sampling Error and Bias\n",
    "\n",
    "Much of the discussion speculates on the number of consistent accepts in the process (using the main conference accept rate as a proxy). It therefore produces numbers that don't match ours above. This is because the computed accept rate of the individual committees is different from that of the main conference. This could be due to a bias for the duplicated papers, or statistical sampling error. We look at these questions below. First, to get the reader primed for thinking about these numbers we discuss some context for placing these numbers.\n",
    "\n",
    "### Context: A Random Committee\n",
    "\n",
    "The first context we can place around the numbers is what would have happened at the 'Random Conference' where we simply accept a quarter of papers at random. In this NIPS the expected numbers of accepts would then have been:\n",
    "<table>\n",
    "  <tr>\n",
    "  <td colspan=\"2\"></td><td colspan=\"2\">Committee 1</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "  <td colspan=\"2\"></td><td>Accept</td><td>Reject</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td rowspan=\"2\">Committee 2</td><td>Accept</td><td>10.4 (1 in 16)</td><td>31.1 (3 in 16)</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>Reject</td><td>31.1 (3 in 16) </td><td>93.4 (9 in 16)</td>\n",
    "  </tr>\n",
    "</table>\n",
    "And for this set up we would expect *inconsistency* of 3 in 8 (37.5%) *accept precision* of 1 in 4 (25%) and a *reject precision* of 3 in 4 (75%) and a *agreed accept rate* of 1 in 10 (10%). The actual committee made improvements on these numbers, in particular the accept precision was markedly better with 50%: twice as many consistent accept decisions were made than would be expected if the process had been performed at random and only around two thirds as many inconsistent decisions were made as would have been expected if decisions were made at random. However, we should treat all these figures with some skepticism until we've performed some estimate of the uncertainty associated with them. \n",
    "\n",
    "### Uncertainty\n",
    "\n",
    "To get a handle on the uncertainty around these numbers we'll start by making use of the (binomial distribution)[http://en.wikipedia.org/wiki/Binomial_distribution]. First, let's explore the fact that for the overall conference the accept rate was around 23%, but for the duplication committees the accept rate was around 25%. If we assume decisions are made according to a binomial distribution, then is the accept rate for the duplicated papers too high?\n",
    "\n",
    "Note that for all our accept probability statistics we used as a denominator the number of papers that were initially sent for review, rather than the number where a final decision was made by the program committee. These numbers are different because some papers are withdrawn before the program committee makes its decision. Most commonly this occurs after authors have seen their preliminary reviews: for NIPS 2014 we provided preliminary reviews that included paper scores. So for the official accept probability we use the 170 as denominator. The accept probabilities were therefore 43 out of 170 papers (25.3%) for Committee 1 and 44 out of 170 (25.8%) for Committee 2. This compares with the overall conference accept rate for papers outside the duplication process of 349 out of 1508 (23.1%). \n",
    "\n",
    "If the true underlying probability of an accept were actually 0.23 independent of the paper, then the probability of generating accepts for any subset of the papers would be given by a binomial distribution. Combining across the two committees for the duplicated papers, we see that 87 papers in total were recommended for accept out of a total of 340 trials. out of 166 trials would be given by a binomial distribution as depicted below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import binom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfUAAADSCAYAAABTllkrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhiklEQVR4nO3deZxcVZn/8c+XsEMgLMGBrCARiPwEQwy4ISMgCSJRfjIGhQAuMSPIMriAOooLouOoIyMS2UGBiMiSgSgwKCJihLAYCCHaQiQhAQJCCKBA5Jk/zmlyU9zqrk6qu6pvvu/Xq15d99ztqbs995y7tCICMzMz6//WaXUAZmZm1hxO6mZmZhXhpG5mZlYRTupmZmYV4aRuZmZWEU7qZmZmFdGUpC5pmqR/b9K0hkt6VtKA3H2zpI82Y9p5ej+XdGSzpteD+X5N0hOSHu3reZeRFJJ2bNG8d5J0t6Tlko5rRQx9SdJRkm5tdRx9SdLnJJ3b6jjWlKQLJX1tDcZfIGm/Bob7kKQbVnc+1jeUXCDpKUm3tzqeMt0m9bxR/i0fgJ+WdJukqZJeGTcipkbEVxucVpcbeEQ8HBGbRsQ/GvsJXc7vVEk/rpn+hIi4aE2n3cM4hgEnAaMj4p9K+u+Tk+yZNeW3Sjqqj8LsS58Bbo6IgRFxRr2B8gF1haTt+jC2sjiaemJZM+2Red0/mz8LJJ3cG/NqJkn7S/pVPi48KekeSZ+VtCFARHw9InplmXUR0z6SFpWU99r6a5aIuCQi3tWb88jL4SlJG/TmfBqIo8s8kNfjy3l/WC5pvqSjezD93lzfbwP2B4ZGxLg68/+gpL9Iek7S1ZK2rDPcNpIuk7RY0jJJv5W0Z6H/P0u6N+fdJyVdJWlIdwE2WlN/T0QMBEYA3wA+C5zX4LgNk7Rus6fZJkYAT0bE410M8xwwWdLIvgmpOVZznY0A5nYz3U2A/w8sAz60GvPobwZFxKbAYcAXJY3vqxn3dB1KOhS4ArgUGBERWwEfAIYCw5ofYf/TbseyfFx5OxDAwa2NpiGL8/6wGXAicI6knVocE6Rj14KIeK6sp6TXAz8EjgBeAzwP/KDOtDYF7gD2ALYELgKuk7Rp7n8/cEBEDAK2A/4EnNVthBHR5QdYAOxXUzYOeBnYNXdfCHwtf98auBZ4Gvgr8BvSycOP8jh/A54l1dZGkjayjwAPA7cUytbN07sZOB24nXSAvwbYMvfbB1hUFi8wHngReCnP7w+F6X00f18H+ALwF+Bx4GJg89yvM44jc2xPAJ/vYjltnsdfmqf3hTz9/fJvfjnHcWHJuPsAi4D/Bi4olN8KHJW/nwr8uNCvbDl9Dbgtz+d/gK2AS4BnSBvPyML4ARwHPJh/27eAdQr9PwzMA54CricdvIvjHkPayB6qszwOJiXup3Nsu+TyXwL/AP6e43xdnfEnAwuB44H7avptCVwALM7xXV3oNxG4J//mPwPjC+vnPGAJ8EheVgNyv6OA3+blvwx4ANg39zutJt7v5/KdgRtJ2/h84F8KMWwFzMgx3A58Fbi1zu9cZT3msjuAT5H2s9/lZbgE+D6wfm+sQ0DAd0n7wTJgDnn/rolXeb2c1M1x41Ty9gr8Aji2pv8fgEMaWJYXAmcC1wHLgd8Dr60zz32oOR6U7POnApeT9tXlpG10bGHYNwJ35X4/AaaTj225/0Gk7etp0r72hppjz2fzsnsBWJeVx6N/Ih3gtyoMvwfpeLEeaRu8tdAvgKl5/TyVl4FyvwHAt/M6fwg4tnYbKlkGXyRt498Brq3pNwy4MsfyJHkbz/0+lreh5aQkMyaXbwf8LI/zEHBczbq/Ii+/5Xl57pb7vSoPNLIeSdvlofn7FqQcszQvm2tJNWdYjf21ZP7bkfbfvwIdwMdy+UfydP+Rp/3lknG/Dlxa6H4tKQ8N7Gp/KQz/DLBHSfkGpDx4f7fTaGAmC6hJ6rn8YeBfCzteZ1I/HZiWN9T1SGeHKpsWKw9oFwObABtRnqweAXbNw/yMlQeLspX/yjyoSYQlO/iH80rbgXTWdCXwo5rYzslx7UbaUXeps5wuJp1wDMzj/hH4SFcHm9qNmLTjPwPslMt7mtQ78ka0OWkH/CPpgLJuju+CmoPGr0gJcngetnO5vDdPa5c87heA22rGvTGPu1HJ73kdqeVh/7wNfCZPb/3addDFMrkJ+A/S2e4K8sEk97uOdMDYIk//Hbl8HCkh7U86oRoC7Jz7XU06g94E2IaUbD+e+x2V53Fint4H8nS2LIs3T2MhcHRePmNIB9jX5/7TSYljE9J2+wgNJHVSwnwr6eC/L+mgv1fuN5J0cD2hN9YhcABwJzAox7ELsG1JvDvncUd2s/5OZeV+Ohn4baHfaFJS3KCBZXkh6eA6Lve/BJje1X5UUv7K+stx/R04kJQcTwdm5X7rk07IO7eD95MqBZ3HtjGk5LJnHvdI0vFmg8Kx5x5Sktyo5Hg0k3zMzN3fBf67sA3WJvVr8/oYTkpgnSeoU0n791DSPvC/dJ/UO4BPkLapl4DX5PIBpBOs7+Z1sSHwttzvUNK2+6a8TexIqqmuk7eVL+ZltgPpxPKAwjJ+KS+/9UgnqA8B63WVU8rWY57XwaQTgTfmsq1IrXgbk463P2XVE/tX1ncj+2vJ/H9Nql1vCOyel/2+ZeupZNxrgM/WlD1LSaIuGXd30ra5eaFsOGlfeTkv06O6nU4DMypdAcAscs2VVZP6V/IP27G7abHygLZD2UGusIK+UXNAeDFvjK+s/LJ50H1Svwn4RKHfTnnBdR5Eg3wGmPvfDkwq+V0DSAl/dKHs46TrxpTF2cVG/B/AT/L3nib1zxf6fxv4eaH7PcA9NQeN8YXuTwA35e8/J5+QFHas58k1vTzuO7v4Pf8OXF4z/iPAPmU7Xcn4w0kb8e65+3rge/n7trnfFiXj/RD4bkn5a/L62ahQdhjwq8KOuph88llY10eUxUtK+r8pmfeX8rbwEvlkIvf7Ot0n9adJtY55FGo9NcOeAFzVG+sQeCfppGAvCrX9khjelsfdsFA2Pcf/fGGZncrKpD6QdJLXOe/TgPO7W5b5+4XAuYV+BwIPdLcfdbHPnwr8b6HfaOBv+fveJdvBbaw8tp0FfLVm2vNZeVK5APhwTf8FrDwefYB8cpO3k0eBcYVtsDapv63QfTlwcv7+S/IJae7ejy6Sel5nLwFb5+4HgBPz9zeTktarxiXtd8eXlO8JPFxTdgq50pCX8ayabW8J8PbaZdLFenw5b1MvkGrGJ3Qx/O7AU2Xru5FtrKZ8WJ7fwELZ6eQW1tr1VDL+TcDUmrJXjn1djLcZcC9wSp3+W5JagfbqajoRsUZ3vw8hnUHX+hbprPAGSQ82eNPPwh70/wvp7G/rhqLs2nZ5esVpr0tKAp2Kd6s/T6rR19qalWf5xWl1e1NDiW8CB0jabTXGfazw/W8l3bWx1y7XzhvSRgDfyzdoPE1az2LV39PVOltluUbEy3n4RpfHEcC8iLgnd18CfFDSeqSd7q8R8VTJeMNITe61RpC2mSWF3/RDUo290yOR956suDzKprdn57Ty9D5EamkZTNqGapdtd7aOiC0iYpfINw9Kep2kayU9KukZ0slB7XbflHUYEb8kNe+fCTwm6WxJm5XE+WT+u21h3EmRrvvdRUpWq4iI5aTWlUm5aBJpnXbGWW9ZdmpkH4TU2rJeSfl6pKRWb3ob5mvg21G+HXQaAZxUE+swVt1OutovrgFGS9qB1Jq0LCK6uoO63u/ermY+3R0/jwRuiIgncveluQxS/H+JiBUl43W1P21Xsxw+x6rHzeK29TKpJbInN7wuztvUZsAZpJNOACRtLOmH+Wa0Z0iXbQcpPzFVJ97utrFO25GOL8sLZT05lj+bYy7ajHQZopSkjUiXS2dFxOllw0TEX0nX3K/p7n6N1Urqkt5E+pGvekwnIpZHxEkRsQOpdvhvkvbt7F1nkvXKOxVvvhlO2kGfIJ39b1yIawDpoNrodBeTVnhx2itYNRk24okcU+20HunhdIiIJ4H/Il2HLVrlt1K+QfZU7XJdnL8vJNUEBhU+G0XEbcVQu5juKstVkvK8Gl0ek4EdcjJ7lHQdcGtgQo5tS0mDSsZbSLr8UFb+Ailxdv6ezSLi9YVhhuQ4OxWXR+1vXQj8umb5bBoR/0qq9azg1ct2dZxFqlWNiojNSAdO1QzTtHUYEWdExB7A60mXUD5dEtMDpPV4SA9/y2XAYZLeTGru/1UhznrLsqceBrYu3GjUue2NoLETqyWUbwedFgKn1cS6cURcVhim7n4REX8n1bg/RDpx/VEDMdWLc2ihu+7NiTlh/AvwjsL+dCKwW644LASG10kUXe1PD9Ush4ERcWBZTEpPSg2l/v5UV0S8QKqh/j9J783FJ5FaVffM+8XenbOqM/2ebGOLSceXgYWynhzL55Iu1aaA0gncBqRWsFfJTyJcnaf/8W6mvS6pIlJ2sv2KHiV1SZtJOojU3PbjiLi3ZJiDJO2Yd4xnSE0ZnY+nPUa6/tJTh0saLWljUvP+FZEeefsj6Sz73bkW9wXSAuz0GDBShcfvalwGnChp+3wg+Dqp6bvsrLWuHMvlwGmSBkoaAfwb8OOux6zrO8BbSNc1O90D7K30HP/mpOauNfVpSVsoPXJ3POk6NaR7Ik7Jd3IiafN8x3OjLgfeLWnfvF5OIiXV27oeDfJB/7Wka6i758+u5NpFRCwhNS3/IMe+nqTOnfo84Og833UkDZG0cx7nBuDbeRteR9JrJb2jMOttgOPy9A4lLfuZuV/tdnst8DpJR+Th15P0Jkm75G3hSuDUXKMYzcpaUU8NJO1Dz0raGSg7CDVlHeb498zr6zlW3hC0ilyLPQn4kqSP5XlL0ihWranVmklKrl8h7WMv5/K6y7KLaZWKiIdJN9J9U9Km+YD5adJJ1qwGJvG7POxxktaVdAhpO+x0DjA1LydJ2iQfewaWTq3cxaQm3INZ/ePD5cDxefseREp69byXtB5Hs3J/2oV0A/Nk0mWmJcA38u/ZUNJb87jnAp+StEf+vTvmY9vtwDNKjzBuJGmApF1zZa/THpIOyScLJ5D2/8510KM8EBEvki4nfjEXDSS1PD6t9LjYl2pGaXh/LZnXQtJx6vS8LN5AukHuktph67gEeI+ktys9wfMV4Mqamj8AeV+7Iv+WyYV9orP/IUrv9FhH0mBSXrg719rrajSp/4+k5aQzns/nidd7bnAU6caNZ0k7yQ8i4ubc73TgC0pNIJ9qcN6QzmgvJDVHbUi645eIWEa6jngu6UznOVIzT6ef5r9PSrqrZLrn52nfQrqR4+/AJ3sQV9En8/wfJLVgXJqn32MR8Qzp2vqWhbIbSQfsOaSbVK5dzTiLrsnTuofUPHpentdVpMsA05Wat+4j1ZIbjX8+cDjpbvInSC0278k7Z3eOBK6JiHsj4tHOD/A94KC8Ex9Bahl5gHTj0gl5vreTtsvvkm50+zUrWwwmky6R3E+6dn0FhSZkUjIYleM9DXh/bjUhz/v9Ss/4npF30HeRmpEXk7bLb7LyhPJYUlPpo6Tt9oIGfneZTwEfJDXdncPKhF3UrHW4WZ7HU6Ra7ZPAf5YNGBE/IdX+DicdE54gJZqzWbnP1Y7zAulkZz/SvtFZ3t2y7KkPkE7QOkjHhH2BA3MtuUt5+zyElHSfytO6stB/Nulu8O/n/h152IZFxG9J14vviogFPRm34BzSSeoc4G7SCdMKSk7CSPvTBZHe/1Hcn75PajEQaf/ckdTSsYj0u4mIn5L2hUtJ2+DVpJtH/5HH2Z103HyCdAzevDDfa/J0niLtr4dEROclkNXJA+eTWhTeQ2rJ3CjPdxbp6Yqinu6vtQ4j3euyGLiKdO39xkaCjIi5pBsZLyEdmwaSchTwyovapuXOt5CepngX6QSl810Vb8/9h+Tftpx0vf1l4H3dxdB5V7rZWkvpBT8fjYi3tTqWnpAUpKb5jlbHYo2T9EvSY09NeeOepAnAtIgY0e3AfUDSqaQbpQ9vdSxrI7/73cysj+Qm6jGUt7o0Oo2NJB2YLxEMITU/X9WsGK1/c1I3M+sDki4iXZo8oewaa08mBXyZ1LR9N+kxyC92OYatNdz8bmZmVhGuqZuZmVWEk7qZmVlFtNV/ErKe2XrrrWPkyJGtDsNa4c47V+3eY4/WxGGVdOeddz4REYO7H9LajZN6PzZy5Ehmz57d6jCsFVTzYjlvB9ZEkhp5A5+1ITe/m5mZVYSTupmZWUU4qZuZmVWEk7qZmVlFOKmbmZlVhO9+N+vCyJOvW6V7wTfe3aJIzMy655p6k0kaL2m+pA5JJ5f0l6Qzcv85ksYU+i2QdK+keyT5GSUzM+sR19SbSNIA4Exgf9L/Jb5D0oyIuL8w2ATS/+0eBewJnJX/dvrniHiij0K2GsWaeb1aeSPDmJm1gmvqzTUO6IiIByPiRWA6MLFmmInAxZHMAgZJ2ravAzUzs+pxUm+uIcDCQveiXNboMAHcIOlOSVN6LUozM6skN783l0rKav+3bVfDvDUiFkvaBrhR0gMRccsqI6dkPwVg+PDhaxrvWs9N6WZWJU7qzbUIGFboHgosbnSYiOj8+7ikq0jN+ask9Yg4GzgbYOzYsbUnDNYCPjEws3bh5vfmugMYJWl7SesDk4AZNcPMACbnu+D3ApZFxBJJm0gaCCBpE+BdwH19GbyZmfVvrqk3UUSskHQscD0wADg/IuZKmpr7TwNmAgcCHcDzwNF59NcAVyn99611gUsj4hd9/BPMzKwfc1JvsoiYSUrcxbJphe8BHFMy3oPAbr0eoJmZVZab383MzCrCNXVbq/imNjOrMid1sybziYOZtYqb383MzCrCSd3MzKwinNTNzMwqwkndzMysIpzUzczMKsJJ3czMrCL8SJtVVvHRMvDjZWZWfU7qZr3MJxdm1lfc/G5mZlYRTupmZmYV4aRuZmZWEU7qZmZmFeGkbmZmVhFO6mZmZhXhpG5mZlYRTupmZmYV4ZfPWGUUX/LiF7yY2drISd2sj/kNc2bWW9z83mSSxkuaL6lD0skl/SXpjNx/jqQxNf0HSLpb0rV9F7WZmVWBk3oTSRoAnAlMAEYDh0kaXTPYBGBU/kwBzqrpfzwwr5dDNTOzCnJSb65xQEdEPBgRLwLTgYk1w0wELo5kFjBI0rYAkoYC7wbO7cugzcysGpzUm2sIsLDQvSiXNTrMfwGfAV7upfjMzKzCnNSbSyVl0cgwkg4CHo+IO7ucgTRF0mxJs5cuXbq6cZqZWQU5qTfXImBYoXsosLjBYd4KHCxpAanZ/p2Sflw7g4g4OyLGRsTYwYMHNzN2MzPr55zUm+sOYJSk7SWtD0wCZtQMMwOYnO+C3wtYFhFLIuKUiBgaESPzeL+MiMP7NHozM+vX/Jx6E0XECknHAtcDA4DzI2KupKm5/zRgJnAg0AE8DxzdqnjNzKxanNSbLCJmkhJ3sWxa4XsAx3QzjZuBm3shPDMzqzAndeu3/FpYM7NVOambtQGfoJhZM/hGOTMzs4pwUjczM6sIJ3UzM7OKcFI3MzOrCCd1MzOzinBSNzMzqwgndTMzs4pwUjczM6sIJ3UzM7OKcFI3MzOrCL8m1vqNtelVqmvTbzWz5nFN3czMrCKc1M3MzCrCSd3MzKwinNTNzMwqwkndzMysIpzUzczMKsJJ3czMrCKc1M3MzCrCSd3MzKwinNSbTNJ4SfMldUg6uaS/JJ2R+8+RNCaXbyjpdkl/kDRX0pf7PnozM+vPnNSbSNIA4ExgAjAaOEzS6JrBJgCj8mcKcFYufwF4Z0TsBuwOjJe0V1/EbWZm1eB3vzfXOKAjIh4EkDQdmAjcXxhmInBxRAQwS9IgSdtGxBLg2TzMevkTfRe6tbPiu+ABFrQmDDNrc07qzTUEWFjoXgTs2cAwQ4AluaZ/J7AjcGZE/L52BpKmkGr4DB8+vHmRt5lXJTH/UxMzs265+b25VFJWW9uuO0xE/CMidgeGAuMk7fqqASPOjoixETF28ODBaxqvmZlViJN6cy0ChhW6hwKLezpMRDwN3AyMb3qEZmZWWU7qzXUHMErS9pLWByYBM2qGmQFMznfB7wUsi4glkgZLGgQgaSNgP+CBPozdzMz6OV9Tb6KIWCHpWOB6YABwfkTMlTQ1958GzAQOBDqA54Gj8+jbAhfl6+rrAJdHxLV9/RvMzKz/clJvsoiYSUrcxbJphe8BHFMy3hzgjb0eoJmZVZab383MzCrCSd3MzKwinNTNzMwqwkndzMysIpzUzczMKsJJ3czMrCL8SJu1hdp3vZuZWc+5pm5mZlYRTupmZmYV4aRuZmZWEU7qZmZmFeGkbmZmVhFO6mZmZhXhpG5mZlYRTupmZmYV4aRuZmZWEX6jnFlFFN/Kt+Ab725hJGbWKq6pm5mZVYRr6tYSrlWamTWfa+pmZmYV4aRuZmZWEU7qTSZpvKT5kjoknVzSX5LOyP3nSBqTy4dJ+pWkeZLmSjq+76M3M7P+zEm9iSQNAM4EJgCjgcMkja4ZbAIwKn+mAGfl8hXASRGxC7AXcEzJuGZmZnU5qTfXOKAjIh6MiBeB6cDEmmEmAhdHMgsYJGnbiFgSEXcBRMRyYB4wpC+DNzOz/s1JvbmGAAsL3Yt4dWLudhhJI4E3Ar+vnYGkKZJmS5q9dOnSZsRsZmYV4aTeXCopi54MI2lT4GfACRHxzKsGjDg7IsZGxNjBgwevUbBmZlYtTurNtQgYVugeCixudBhJ65ES+iURcWUvxmlmZhXkl8801x3AKEnbA48Ak4AP1gwzAzhW0nRgT2BZRCyRJOA8YF5EfKcvg7Zq8gt+zNY+TupNFBErJB0LXA8MAM6PiLmSpub+04CZwIFAB/A8cHQe/a3AEcC9ku7JZZ+LiJl9+BPMzKwfc1JvspyEZ9aUTSt8D+CYkvFupfx6eyW41mhm1vt8Td3MzKwinNTNzMwqwkndzMysIpzUzczMKsJJ3czMrCKc1M3MzCrCj7SZrSWKjxWCHy00qyLX1M3MzCrCSd3MzKwi3PxuTedmXjOz1nBN3czMrCKc1M3MzCrCSd3MzKwinNTNzMwqwkndzMysInz3u9layk8pmFWPa+pmZmYV4aRuZmZWEW5+tzVW24xrZmat4Zq6mZlZRTipm5mZVYSTepNJGi9pvqQOSSeX9JekM3L/OZLGFPqdL+lxSff1bdRmZlYFTupNJGkAcCYwARgNHCZpdM1gE4BR+TMFOKvQ70JgfO9HamZmVeQb5ZprHNAREQ8CSJoOTATuLwwzEbg4IgKYJWmQpG0jYklE3CJpZJ9HbZYVb3r0c+tm/Y9r6s01BFhY6F6Uy3o6jJmZWY85qTeXSspiNYapPwNpiqTZkmYvXbq0R8GZmVm1ufm9uRYBwwrdQ4HFqzFMXRFxNnA2wNixYxs+GWgmN9GambUn19Sb6w5glKTtJa0PTAJm1AwzA5ic74LfC1gWEUv6OlAzM6seJ/UmiogVwLHA9cA84PKImCtpqqSpebCZwINAB3AO8InO8SVdBvwO2EnSIkkf6dMfYGZm/Zqb35ssImaSEnexbFrhewDH1Bn3sN6NzqxnfKnFrH9xTd3MzKwinNTNzMwqws3v1i03wZqZ9Q+uqZuZmVWEk7qZmVlFuPndzBpWvBQDvhxj1m5cUzczM6sIJ3UzM7OKcPO7rcLNq2Zm/ZeTupmtNp8EmrUXN7+bmZlVhJO6mZlZRbj5fS1X23xqZmb9l5O6mTWVXyts1jpufjczM6sIJ3UzM7OKcPP7WsZNo9bXvM2Z9R3X1M3MzCrCNXUz61N+YY1Z73FN3czMrCJcUzezlnLN3ax5XFNvMknjJc2X1CHp5JL+knRG7j9H0phGxzVbW4w8+bpXPmbWONfUm0jSAOBMYH9gEXCHpBkRcX9hsAnAqPzZEzgL2LPBcc3WSr6D3qwxTurNNQ7oiIgHASRNByYCxcQ8Ebg4IgKYJWmQpG2BkQ2Ma2Y4yZvV46TeXEOAhYXuRaTaeHfDDGlwXDMrUXZdvjbxd9dtVgVKFUZrBkmHAgdExEdz9xHAuIj4ZGGY64DTI+LW3H0T8Blgh+7GzeVTgCm5cydg/mqGuzXwxGqO25ccZ/P1l1j7S5zQf2JtNM4RETG4t4Ox5nNNvbkWAcMK3UOBxQ0Os34D4xIRZwNnr2mgkmZHxNg1nU5vc5zN119i7S9xQv+Jtb/EaavPd7831x3AKEnbS1ofmATMqBlmBjA53wW/F7AsIpY0OK6ZmVldrqk3UUSskHQscD0wADg/IuZKmpr7TwNmAgcCHcDzwNFdjduCn2FmZv2Uk3qTRcRMUuIulk0rfA/gmEbH7UVr3ITfRxxn8/WXWPtLnNB/Yu0vcdpq8o1yZmZmFeFr6mZmZhXhpL4WyC+4uULSA5LmSXqzpC0l3SjpT/nvFi2OcSdJ9xQ+z0g6od3i7CTpRElzJd0n6TJJG7ZjrJKOzzHOlXRCLmuLOCWdL+lxSfcVyurGJumU/Arl+ZIOaHGch+Zl+rKksTXDtyTOLmL9Vt7350i6StKgdojVeoeT+trhe8AvImJnYDdgHnAycFNEjAJuyt0tExHzI2L3iNgd2IN0E+FVtFmcAJKGAMcBYyNiV9KNjZNos1gl7Qp8jPSmw92AgySNon3ivBAYX1NWGpuk0aRl/Po8zg/yq5VbFed9wCHALcXCFscJ5bHeCOwaEW8A/gicAm0Rq/UCJ/WKk7QZsDdwHkBEvBgRT5NeQXtRHuwi4L2tiK+OfYE/R8RfaN841wU2krQusDHpnQLtFusuwKyIeD4iVgC/Bt5Hm8QZEbcAf60prhfbRGB6RLwQEQ+Rnh4Z16o4I2JeRJS9+Kllcea4ymK9Ia9/gFmkd2C0PFbrHU7q1bcDsBS4QNLdks6VtAnwmvx8PPnvNq0MssYk4LL8ve3ijIhHgP8EHgaWkN41cAPtF+t9wN6StpK0MelRymG0X5xF9WKr93rldtPucX4Y+Hn+3u6x2mpwUq++dYExwFkR8UbgOdqgCbue/OKdg4GftjqWevJ13onA9sB2wCaSDm9tVK8WEfOAb5KaX38B/AFY0eVI7UslZe346E7bxinp86T1f0lnUclgbRGrrT4n9epbBCyKiN/n7itISf6x/N/hyH8fb1F8tSYAd0XEY7m7HePcD3goIpZGxEvAlcBbaMNYI+K8iBgTEXuTmmX/RBvGWVAvtkZewdwO2jJOSUcCBwEfipXPMbdlrLZmnNQrLiIeBRZK2ikX7Uv6d64zgCNz2ZHANS0Ir8xhrGx6h/aM82FgL0kbSxJpmc6jDWOVtE3+O5x0Y9dltGGcBfVimwFMkrSBpO2BUcDtLYivO20Xp6TxwGeBgyPi+UKvtovVmiAi/Kn4B9gdmA3MAa4GtgC2It1d/Kf8d8s2iHNj4Elg80JZ28WZ4/oy8ADpuvWPgA3aMVbgN6STuD8A+7bTMiWdYCwBXiLVGj/SVWzA54E/k/4z4YQWx/m+/P0F4DHg+lbH2UWsHaRr5/fkz7R2iNWf3vn4jXJmZmYV4eZ3MzOzinBSNzMzqwgndTMzs4pwUjczM6sIJ3UzM7OKcFI3MzOrCCd1MzOzinBSNzMzq4j/A0agMWWEK0MaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "rv = binom(340, 0.23)\n",
    "x = np.arange(60, 120)\n",
    "fig, ax = plt.subplots(figsize=(5,3))\n",
    "ax.bar(x, rv.pmf(x))\n",
    "ax.set_title('Distribution of Number of Accepted Papers Given Underlying Accept Rate of 0.23')\n",
    "ax.axvline(87,linewidth=4, color='red') \n",
    "plt.show()\n",
    "fig.savefig('accept-distribution.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the plot, we can see that whilst the accept rate was slightly higher for duplicated papers it doesn't seem that we can say that it was statistically significant that it was higher, it falls well within the probability mass of the Binomial. \n",
    "\n",
    "Note that Area Chairs knew which papers were duplicates, whereas reviewers did not. Whilst we stipulated that duplicate papers should not be any given special treatment, we cannot discount the possibility that Area Chairs may have given slightly preferential treatment to duplicate papers. \n",
    "\n",
    "For the accept precision, if we assume that accept decisions were drawn according to a binomial, then the distribution for consistent accepts is also binomial. Our best estimate of its parameter is 22/166 = 0.13 (13%). If we had a binomial distribution with these parameters, then the distribution of consistent accepts would be as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAADSCAYAAABtszZlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAc8klEQVR4nO3de7gcVZ3u8e9LEq4BAyTDhCQkqOESUDAnAxkBhzOiknCJjoowQADFiAcURjwMAiIKc0BFjuIwZFDut8AAjhmJAgfNOOiAhDsxXGK4JCRAAAmXgCHyO3+stUml6e7dO1m702G/n+fpZ3dXrVprVdWq+tVaVd1bEYGZmdnqWmdNV8DMzN4ZHFDMzKwIBxQzMyvCAcXMzIpwQDEzsyIcUMzMrIgiAUXSVElfL5TXVpJekdQvf54p6cgSeef8fi7psFL59aDcMyQ9J+npdpfdSMn9ZqtO0h6SHl7T9Vhdkg6XdNtqLH+JpDNaSLfSOaKvkLStpHskvSzpy2u6PvV0G1AkPS7ptbwSL0r6raSjJL21bEQcFRGnt5jXXs3SRMSTETEwIv7c2io0Le80SVfU5D8hIi5d3bx7WI8RwPHAmIj4ywZpNpH0fUlP5oNlbv48uLfqVXK/tUrSKEkhqX8LaQ/PaQ8oUfaqqteOerj8aEnTJC2W9JKkRyX9UNJwgIj4r4jYtlyNW65XSHpvzbTVWtd2KHmOqJWD2rJ8DL4g6RZJ27W4bMttexWdAMyMiI0j4tw65a8n6aLcxp6W9JUmdR0qabqkhbnOo2rmf0fS/JzXE5JObqWCrfZQ9ouIjYGRwFnAPwIXtrhsy3pxR6xpI4HnI+LZejMlrQvcCuwA7A1sAnwQeB7YpV2V7ECHAS/kv2ulfMK+A1gIfCAiNgF2A/4A7L4m69ZJOqy38Z2IGAgMA56iF851q2gkMLvJ/NOA0Tnd/wROkLR3g7RvAr8APtlg/oXAdrm9fhD4e0l/120NI6LpC3gc2Ktm2i65Qjvmz5cAZ+T3g4GfAS+STgb/RQpcl+dlXgNeIUXbUUAAnwOeBH5dmdY/5zcTOBP4HbAE+CmwWZ63J7CgXn1JJ+ZlwBu5vPsq+R2Z368DnAI8ATwLXAa8K8/rqsdhuW7PASc32U7vyssvzvmdkvPfK6/zm7kel9RZ9kjgGWBgk/y3z3V/kdSo9q/MuwQ4D7gReJl0AntPnifg/+b1WwLcv7r7LacfD/w2p78P2LNSn5nA6cBvcn1uBgbneU/m7fpKfv11g/Udmcv9JLAc2KIyrx9wEumk/DJwFzAiz9sBuCWvwzPASZV9fWJe5nngWla0o659PYV04l8EHJ/nNWpHhwPzcvmPAQc3WI8rgP/o5hjbk9yOcx2vq5n/A+DcSju7MNfxKeAMoF+lTrcBZwN/zPWa0KTcAN5bM+004IpqvUi962dzmUdU0m4OTAdeIh2fpwO3VeZvV9kXDwMH1LTZ84EZwKuk4+QSVrTHB0kXsl3pB5COwZ2pf46o297y/MmkY/J54OvUOafV1OuMyueJwKuVz/sA9+R1ng+cVplXt20DnwXm5H1yEzCyyT7Zn3R8v5jXa/s8/ZfAn4HXc97b1Fn2KeCjlc+nA9O6aXv9c51HNUkzDHiAfOw3za/bBA02ft54X6xzYjoTmJobwABgD0D18qo0jMuAjYANGjSWp4Adc5rrqWnwjepL5eCoOdl1BZTPAnOBdwMDgRuAy2vq9qNcr52AP3Xt4Drb4zJSsNs4L/sI8LlG9axZdhpwaZP5A3I9TwLWBf6WdOBsW9n+L5ACfX/gyq6GBHyMdMIdRAou2wNDV3O/DSMdnBNJJ+qP5M9DKtv4D8A2edvNBM6q2a79u2l3Xwd+l98/AHylMu9/52nb5nXaiXRy25gcDID18+dd8zLHAbcDw4H1gH8Frq6p09WkNvY+0oVB3XaU07xU2f5DgR0arMfTwOHdrOtb7YMUSJcCm+TP/fI6jc+f/z3XfSPgL0gn8i/keYeTAt/n83JfJAVINSi3lYCyHPhWbhMTc902rbTba3NddiQdp7dVttF84AhSmxxLCgg7VNreElJvbZ28vy5hRXs8AbimUq9JwAP12hDN29sY0gl4d9Kxc3beRt0GlLwOl5MvIirb5H25zu8nXbR8vFHbBj5OOna3z9vhFOC3DcrehhRcP5K39wl52XVrz111lt00l1298PpU1zZr0vYaBhTSxc0ref48YHizvCJitW7KLwQ2qzP9DdIBNjIi3og0Phzd5HVaRLwaEa81mH95RDwYEa+STjQHFOoiHwycExHzIuIV4GvAgTVDb9+MiNci4j7SlfhOtZnkunwG+FpEvBwRjwPfAw5tsR6bk04ajYwnBbyzImJZRPyS1Js4qJLmhoj4XUQsJwWUnfP0N0gn1u1IJ5Y5EVGvrJ7st0OAGRExIyLejIhbgFmkE06XiyPikbxPr63Up1WTgavy+6tYedjrSOCUiHg4kvsi4nlgX+DpiPheRLye98UdeZkvkHqYCyLiT6QT56fq7OtXI+IB4GJW3r613gR2lLRBRCyKiEZDEYNJQQUAScfke5GvSPpRbeKIeAK4m3QignTxsDQibpe0BTABOC7X81lS7/PAShZPRMSPIt1fuJS0T7dosh7deQP4Vm4TM0gnmG1zm/8kcGquy4O5vC77Ao9HxMURsTwi7iZdDH6qkuanEfGb3IZeryn3CmCipE3y50NJJ/dGGrW3T5F6iLdFxDLgVNIJspmvSnqRdNG2O5XjOCJmRsQDuc73ky5C/qZJXl8AzszH3XLg/wA7SxpZJ+1ngBsj4paIeIMU/DYgDTl1Z2D+u6QybQnp2F8lEXFWXn4sadsvab7E6j3lNYx0VVzru6SoerOkeZJObCGv+T2Y/wQpepe4Wb1lzq+ad39WPgCrT2UtZcWOqxpMuvqpzWtYi/V4nnTgN6vn/Ih4s0n+deuZg88/k4bEnpF0QeUgrerJfhsJfDqfGF/MB9/uNevQynarS9JuwNakK2BIAeV9knbOn0eQrkhrNZreVeefVOo7hzSEUN3Xte1sy3oZ5QubzwBHAYsk3djkxu1K+zYi/jkiBgHfJ7Xjeq5iRTD7e1YE1pF5mUWV9fhXUk+ly1vbPSKW5reNtv2f69RhACmIvFX/fCLs0rUvh5COldpt1mUksGtNGzkYqD6U0vC4j4iFpCGsT0oaRAqkVzZKT+P2tmW1nLxNnm+SD8DZeR+NIg31vvXAhKRdJf0qP2CxhNQGmp2LRgI/qGyDF0i96nrnhpXOR/l4n98gba1X8t/qsb0JKSiusnzBdg9pO3yzu/SrFFAk/RVpJd/2iGC+Kjw+It4N7Ad8RdKHu2Y3qnc3RY6ovN+K1OCfI3UPN6zUqx+pobea70LSDq/mvZzUje2J53KdavN6qsXl/x/wMUkbNanniOqTdT3JPyLOjYj/Qbq/sA1pyKg2TU/223xSr3FQ5bVRvqLptjotpDmMdNDdq/SYdVcvY3Kl/PfUWa7R9K55E2rqvH5EVLdhbTtb2KjOEXFTRHyEFCweIg2N1nMr0P3NzJX9G7BnfgrsE6wIKPNJw66DK+uwSUTs0MP8uzxJOmlWbc3KgaGRxaRjpXabdZkP/GfN9h4YEV+spOmuLVxK6g1/Gvjvmn3VqkWkYU4AJG1AGhHoVkQ8CRxLCggb5MlXke4bjYiId5GGidW1SJ1s5pOGJKvbYYOI+G2dtCudjySJtH27Xe+I+CNpXasjKDvR/CZ+T/Sn8bH1lh4FlPxo676kK8cr8tBAbZp9Jb03b4yXSFdBXY/3PUO6X9FTh0gaI2lD0njudblL/wiwvqR9JA0gjU+uV1nuGWBUzYm46mrgHyRtLWkgqTt6Tc0VWbdyXa4F/knSxrk7+xVSt70Vl5Ma3vWStpO0jqTNJZ0kaSLphPoq6amNAZL2JJ30pzXMMZP0V/mqakDO43VW7I9qup7styuA/SR9TFI/SetL6joBdmcxabiobjuQtD5wAOkG+c6V15eAg/MQ1Y+B05Uex5Wk90vanDQM+JeSjsuPUG4sadec9VTS/hmZyxkiaVJN8V+XtKGkHUhj/9dU1v+tdiRpC0n75wuAP5GuDhs9wnoasIekcyQNy8sPJo2p1xURi0nj5RcDj0XEnDx9EemG8/fysbiOpPdIajbk0sw1wCmShue89iK1q+u6WzC3+RuA0/I2G8PKw5I/A7aRdGhuswNyW2y43nX8O2m45VjSPcpVcR2prX5Q6WnKb7IiAHQrD+cuJLVHSENAL0TE65J2IfUgu9Rr21OBr+U2haR3Sfp0g+KuBfaR9OF8vB5Pal/1gk89l5H256a5x/x50j2huvKx1nW+XC9/JreFL+R8lNfzaNLFUVOtBpT/kPQy6aR3MnAO6YCrZzTpivsV4L+Bf4mImXnemaQVflHSV1ssG9IJ9xJSt3Z94MsAEbEE+F+kE8xTpBPmgspy/5b/Pi/p7jr5XpTz/jXpiZjXSSeuVfGlXP48Us/tqpx/t/KY/l6kK91bWPHUzGDgjjz2uz+p2/8c8C/A5Ih4qIXsNyFdPf+RFU+6nF0nXcv7LSLmk26SnkQ6iOaTej3dtqc85PBPwG9yfuNrknyc1L2+LCKe7nqRnmzqR3rq6hzSwXczaVtdCGwQES+TbmjuR2orj5Ien4T0pNR00pDey6Qb9F3Bpst/kob9biUNe9ycp9e2o3VIB/tC0hDG35DaYb31fYR0D2w4cF8u+zd52WZfKr2K1Cauqpk+mTS8+nvSPr2O5sOlzXyLdLK6Lef1HdLTag+2uPwxpKGlp0nH58VdM/K++Cjp/s7CnObbrHzB11S+H3I9qdd0Q6vL1eQxm3RsTiNdwb9MemLtTz3I5ruki7n1SPv5W3k/nkpqh11lva1tR8RPSOs9TdJLpKfXJjSo68OkHtkPScf5fqQn3Za1WM9vkIZ8nyC15e9GxC+6Zirdt9ujkr7ryU1I557qPexPsOIpyitynX7YXQW6nuIx67OUvtT1GDCgp71T612STiU9IntIofwGkh7JHR0Rj5XI01bwb3mZWUeStBnpO2oXrGY+++VhuY1IvfMHSI/CW2EOKGbWcSR9njSU+vOI+PVqZjeJNOy2kDS0e2B4aKZXeMjLzMyKcA/FzMyKcEAxM7Mi3qm/7tuSwYMHx6hRo9Z0NczsHeauu+56LiKGdJ/ynaVPB5RRo0Yxa9asNV0NM3uHkdTKrw2843jIy8zMinBAMTOzIhxQzMysCAcUMzMrwgHFzMyK6NNPeZn1tlEn3thSusfP2qeXa2LW+9xDMTOzIhxQzMysCA95mXUYD5PZ2so9FDMzK8IBxczMivCQl1mLWh2KAg9HWd/kHoqZmRXhgGJmZkU4oJiZWREOKGZmVoQDipmZFeGAYmZmRfixYbN3AH+73jqBeyhmZlaEA4qZmRXhgGJmZkV0VECRtLekhyXNlXRinfmSdG6ef7+ksZV5/yBptqQHJV0taf321t7MrG/rmIAiqR9wHjABGAMcJGlMTbIJwOj8mgKcn5cdBnwZGBcROwL9gAPbVHUzM6ODAgqwCzA3IuZFxDJgGjCpJs0k4LJIbgcGSRqa5/UHNpDUH9gQWNiuipuZWWc9NjwMmF/5vADYtYU0wyJilqSzgSeB14CbI+Lm3qysrf38qK1ZWZ3UQ1GdadFKGkmbknovWwNbAhtJOqRuIdIUSbMkzVq8ePFqVdjMzFbopICyABhR+Tyctw9bNUqzF/BYRCyOiDeAG4AP1iskIi6IiHERMW7IkCHFKm9m1td1UkC5ExgtaWtJ65Juqk+vSTMdmJyf9hoPLImIRaShrvGSNpQk4MPAnHZW3sysr+uYeygRsVzSMcBNpKe0LoqI2ZKOyvOnAjOAicBcYClwRJ53h6TrgLuB5cA9wAXtXwszs76rYwIKQETMIAWN6rSplfcBHN1g2W8A3+jVCpqZWUOdNORlZmZrMQcUMzMroqOGvMysffw9HCvNPRQzMyvCAcXMzIpwQDEzsyIcUMzMrAgHFDMzK8IBxczMinBAMTOzIhxQzMysCAcUMzMrwgHFzMyK8E+v2DuCf0bEbM1zD8XMzIpwQDEzsyIcUMzMrAgHFDMzK8IBxczMinBAMTOzIhxQzMysCAcUMzMrwl9sNLOWtPrlUfAXSPsq91DMzKwIBxQzMyuiowKKpL0lPSxprqQT68yXpHPz/Pslja3MGyTpOkkPSZoj6a/bW3szs76tYwKKpH7AecAEYAxwkKQxNckmAKPzawpwfmXeD4BfRMR2wE7AnF6vtJmZvaVjAgqwCzA3IuZFxDJgGjCpJs0k4LJIbgcGSRoqaRPgQ8CFABGxLCJebGPdzcz6vE4KKMOA+ZXPC/K0VtK8G1gMXCzpHkk/lrRRvUIkTZE0S9KsxYsXl6u9mVkf10kBRXWmRYtp+gNjgfMj4gPAq8Db7sEARMQFETEuIsYNGTJkdeprZmYVnRRQFgAjKp+HAwtbTLMAWBARd+Tp15ECjJmZtUknBZQ7gdGStpa0LnAgML0mzXRgcn7aazywJCIWRcTTwHxJ2+Z0HwZ+37aam5lZ53xTPiKWSzoGuAnoB1wUEbMlHZXnTwVmABOBucBS4IhKFl8CrszBaF7NPDMz62UdE1AAImIGKWhUp02tvA/g6AbL3guM6836mZlZY5005GVmZmuxjuqhmEHrP0LoHyA06yzuoZiZWREOKGZmVoQDipmZFeGAYmZmRTigmJlZEQ4oZmZWhB8bNrNe40fA+xb3UMzMrAgHFDMzK8IBxczMinBAMTOzIhxQzMysCAcUMzMrwgHFzMyKcEAxM7MiHFDMzKwIBxQzMyvCAcXMzIpwQDEzsyIcUMzMrAgHFDMzK8IBxczMivD/Q7Fe5f+HYdZ3dFQPRdLekh6WNFfSiXXmS9K5ef79ksbWzO8n6R5JP2tfrc3MDDoooEjqB5wHTADGAAdJGlOTbAIwOr+mAOfXzD8WmNPLVTUzszo6JqAAuwBzI2JeRCwDpgGTatJMAi6L5HZgkKShAJKGA/sAP25npc3MLOmkeyjDgPmVzwuAXVtIMwxYBHwfOAHYuFkhkqaQejdstdVWq1VhMyvP993WXp3UQ1GdadFKGkn7As9GxF3dFRIRF0TEuIgYN2TIkFWpp5mZ1dFJAWUBMKLyeTiwsMU0uwH7S3qcNFT2t5Ku6L2qmplZrU4KKHcCoyVtLWld4EBgek2a6cDk/LTXeGBJRCyKiK9FxPCIGJWX+2VEHNLW2puZ9XEdcw8lIpZLOga4CegHXBQRsyUdledPBWYAE4G5wFLgiDVVXzMzW1nHBBSAiJhBChrVaVMr7wM4ups8ZgIze6F6ZmbWRCcNeZmZ2VrMAcXMzIpwQDEzsyIcUMzMrAgHFDMzK8IBxczMinBAMTOzIhxQzMysCAcUMzMrwgHFzMyKcEAxM7MiOuq3vKyztfqPj8D//Mjay/+UqzO4h2JmZkU4oJiZWREOKGZmVoQDipmZFeGAYmZmRTigmJlZEQ4oZmZWhAOKmZkV4YBiZmZFOKCYmVkRDihmZlaEA4qZmRXRUQFF0t6SHpY0V9KJdeZL0rl5/v2SxubpIyT9StIcSbMlHdv+2puZ9W0dE1Ak9QPOAyYAY4CDJI2pSTYBGJ1fU4Dz8/TlwPERsT0wHji6zrJmZtaLOunn63cB5kbEPABJ04BJwO8raSYBl0VEALdLGiRpaEQsAhYBRMTLkuYAw2qWNTN7i3/yvryO6aGQAsD8yucFeVqP0kgaBXwAuKN8Fc3MrJFO6qGozrToSRpJA4HrgeMi4qW6hUhTSMNlbLXVVqtW03cA/7MsMyutk3ooC4ARlc/DgYWtppE0gBRMroyIGxoVEhEXRMS4iBg3ZMiQIhU3M7POCih3AqMlbS1pXeBAYHpNmunA5Py013hgSUQskiTgQmBORJzT3mqbmRl00JBXRCyXdAxwE9APuCgiZks6Ks+fCswAJgJzgaXAEXnx3YBDgQck3ZunnRQRM9q4CmZmfVrHBBSAHABm1EybWnkfwNF1lruN+vdXzMysTTppyMvMzNZiDihmZlaEA4qZmRXhgGJmZkU4oJiZWREOKGZmVoQDipmZFeGAYmZmRXTUFxtt1fmnuM1sTXMPxczMinBAMTOzIhxQzMysCAcUMzMrwgHFzMyKcEAxM7MiHFDMzKwIBxQzMyvCX2zsQP6SopmtjdxDMTOzIhxQzMysCAcUMzMrwgHFzMyKcEAxM7Mi/JRXL/MTW2bWV7iHYmZmRXRUQJG0t6SHJc2VdGKd+ZJ0bp5/v6SxrS5rZma9q2OGvCT1A84DPgIsAO6UND0ifl9JNgEYnV+7AucDu7a47GprdfgKPIRlZn1PJ/VQdgHmRsS8iFgGTAMm1aSZBFwWye3AIElDW1zWzMx6UScFlGHA/MrnBXlaK2laWdbMzHqRImJN1wEASZ8GPhYRR+bPhwK7RMSXKmluBM6MiNvy51uBE4B3d7dsJY8pwJT8cVvg4dWs+mDgudXMw+W7/LW5Di7/7eWPjIgha6Iya1LH3EMh9SpGVD4PBxa2mGbdFpYFICIuAC5Y3cp2kTQrIsaVys/lu/y1rQ4uf823gU7RSUNedwKjJW0taV3gQGB6TZrpwOT8tNd4YElELGpxWTMz60Ud00OJiOWSjgFuAvoBF0XEbElH5flTgRnARGAusBQ4otmya2A1zMz6rI4JKAARMYMUNKrTplbeB3B0q8u2SbHhM5fv8lfRmq6Dyzegg27Km5nZ2q2T7qGYmdlazAGlAUkXSXpW0oOVaZtJukXSo/nvpg2WXe2fgWlQ/nclPZR/duYnkgY1WPZxSQ9IulfSrILlnybpqZzvvZImNli2t9b/mkrZj0u6t8GyJdZ/hKRfSZojabakY/P0trSBJuW3pQ00Kb8tbaBJ+W1pA5LWl/Q7Sffl8r+Zp7ftHLBWigi/6ryADwFjgQcr074DnJjfnwh8u85y/YA/kL4bsy5wHzCmUPkfBfrn99+uV36e9zgwuBfW/zTgq90s12vrXzP/e8Cpvbj+Q4Gx+f3GwCPAmHa1gSblt6UNNCm/LW2gUfntagOAgIH5/QDgDmB8O88Ba+PLPZQGIuLXwAs1kycBl+b3lwIfr7NokZ+BqVd+RNwcEcvzx9tJ37fpFQ3WvxW9tv5dJAk4ALh6FerXavmLIuLu/P5lYA7p1xfa0gYald+uNtBk/VvRa+vfNb+320Akr+SPA/IraOM5YG3kgNIzW0T63gv571/USdOun4H5LPDzBvMCuFnSXUq/DFDSMXm45aIG3f12rP8ewDMR8WiD+UXXX9Io4AOkq9S2t4Ga8qva0gbqlN/WNtBg/Xu9DUjql4fUngVuiYg1sv/XJg4o5anOtKKP0kk6GVgOXNkgyW4RMZb068xHS/pQoaLPB94D7AwsIg05vK16daaVfpTwIJpfmRZbf0kDgeuB4yLipVYXqzNtlbZBo/Lb1QbqlN/WNtBk+/d6G4iIP0fEzqRe4C6Sdmy12vWy62n5ayMHlJ55RunXjcl/n62TppWfkFllkg4D9gUOjjxgWysiFua/zwI/IXXBV1tEPJMPsjeBHzXIt7fXvz/wd8A1TepZZP0lDSCdzK6MiBvy5La1gQblt60N1Cu/nW2gyfq3rQ3kPF4EZgJ70wHngE7mgNIz04HD8vvDgJ/WSdNrPwMjaW/gH4H9I2JpgzQbSdq46z3pJu6D9dKuQvlDKx8/0SDf3v4ZnL2AhyJiQYM6Fln/PEZ/ITAnIs6pzGpLG2hUfrvaQJPy29IGmmx/aEMbkDRE+Qk6SRt0lckaPgd0vDX9VECnvkjd6UXAG6Qrjs8BmwO3Ao/mv5vltFsCMyrLTiQ9lfIH4OSC5c8ljc3em19Ta8snPVlyX37NLlz+5cADwP2kA2RoO9c/T78EOKombW+s/+6kYYr7K9t7YrvaQJPy29IGmpTfljbQqPx2tQHg/cA9ufwHyU+TtWv/r60vf1PezMyK8JCXmZkV4YBiZmZFOKCYmVkRDihmZlaEA4qZmRXhgGJmZkU4oJiZWREOKGZmVsT/B31yPJGBXDH/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "rv = binom(166, 0.13)\n",
    "x = np.arange(10, 30)\n",
    "fig, ax = plt.subplots(figsize=(5,3))\n",
    "ax.bar(x, rv.pmf(x))\n",
    "ax.set_title('Distribution of Consistent Accepts Given Underlying Rate of 0.13')\n",
    "plt.show()\n",
    "fig.savefig('consistent-accept-distribution.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see immediately that there is a lot of uncertainty around this number, for the scale of the experiment as we have it. This suggests a more complex analysis is required to extract our estimates with uncertainty.\n",
    "\n",
    "## A Bayesian Analysis\n",
    "\n",
    "Before we start the analysis, it's important to make some statements about the aims of our modelling here. We will make some simplifying modelling assumptions for the sake of a model that is understandable. In particular, we are looking to get a handle on the uncertainty associated with some of the probabilities associated with the NIPS experiment. [Some preliminary analyses have already been conducted on blogs](http://inverseprobability.com/2015/01/16/blogs-on-the-nips-experiment/). Those analyses don't have access to information like paper scores etc. For that reason we also leave out such information in this preliminary analysis. We will focus only on the summary results from the experiment: how many papers were consistently accepted, consistently rejected or had inconsistent decisions. For the moment we disregard the information we have about paper scores.\n",
    "\n",
    "In our analysis there are three possible outcomes for each paper: consistent accept, inconsistent decision and consistent reject. So we need to perform the analysis with the [multinomial distribution](http://en.wikipedia.org/wiki/Multinomial_distribution). The multinomial is parameterized by the probabilities of the different outcomes. These are our parameters of interest, we would like to estimate these probabilities alongside their uncertainties. To make a Bayesian analysis we place a prior density over these probabilities, then we update the prior with the observed data, that gives us a posterior density, giving us an uncertainty associated with these probabilities. \n",
    "\n",
    "### Prior Density\n",
    "\n",
    "Choice of prior for the multinomial is typically straightforward, the [Dirichlet density](http://en.wikipedia.org/wiki/Dirichlet_distribution) is [conjugate](http://en.wikipedia.org/wiki/Conjugate_prior) and has the additional advantage that its parameters can be set to ensure it is *uninformative*, i.e. uniform across the domain of the prior. Combination of a multinomial likelihood and a Dirichelt prior is not new, and in this domain if we were to consider the mean the posterior density only, then the approach is known as [Laplace smoothing](http://en.wikipedia.org/wiki/Additive_smoothing). \n",
    "\n",
    "For our model we are assuming for our prior that the probabilities are drawn from a Dirichlet as follows,\n",
    "$$\n",
    "p \\sim \\text{Dir}(\\alpha_1, \\alpha_2, \\alpha_3),\n",
    "$$\n",
    "with $\\alpha_1=\\alpha_2=\\alpha_3=1$. The Dirichlet density is conjugate to the [multinomial distribution](http://en.wikipedia.org/wiki/Multinomial_distribution), and we associate three different outcomes with the multinomial. For each of the 166 papers we expect to have a consistent accept (outcome 1), an inconsistent decision (outcome 2) or a consistent reject (outcome 3). If the counts four outcome 1, 2 and 3 are represented by $k_1$, $k_2$ and $k_3$ and the associated probabilities are given by $p_1$, $p_2$ and $p_3$ then our model is, \n",
    "\\begin{align*}\n",
    "\\mathbf{p}|\\boldsymbol{\\alpha} \\sim \\text{Dir}(\\boldsymbol{\\alpha}) \\\\\n",
    "\\mathbf{k}|\\mathbf{p} \\sim \\text{mult}(\\mathbf{p}).\n",
    "\\end{align*}\n",
    "Due to the conjugacy the posterior is tractable and easily computed as a Dirichlet (see e.g. [Gelman et al](http://www.stat.columbia.edu/~gelman/book/)), where the parameters of the Dirichlet are given by the original vector from the Dirichlet prior plus the counts associated with each outcome. \n",
    "$$\n",
    "\\mathbf{p}|\\mathbf{k}, \\boldsymbol{\\alpha} \\sim \\text{Dir}(\\boldsymbol{\\alpha} + \\mathbf{k})\n",
    "$$\n",
    "The mean probability for each outcome is then given by,\n",
    "$$\n",
    "\\bar{p}_i = \\frac{\\alpha_i+k_i}{\\sum_{j=1}^3(\\alpha_j + k_j)}.\n",
    "$$\n",
    "and the variance is\n",
    "$$\n",
    "\\mathrm{Var}[p_i] = \\frac{(\\alpha_i+k_i) (\\alpha_0-\\alpha_i + n + k_i)}{(\\alpha_0+n)^2 (\\alpha_0+n+1)},\n",
    "$$\n",
    "where $n$ is the number of trials (166 in our case) and $\\alpha_0 = \\sum_{i=1}^3\\alpha_i$. This allows us to compute the expected value of the probabilities and their variances under the posterior as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probability of consistent accept 0.13609467455621302 +/- 0.03000057324297458\n",
      "Probability of inconsistent decision 0.2603550295857988 +/- 0.04372767641598182\n",
      "Probability of consistent reject 0.6035502958579881 +/- 0.07517355012145442\n"
     ]
    }
   ],
   "source": [
    "def posterior_mean_var(k, alpha):\n",
    "    \"\"\"Compute the mean and variance of the Dirichlet posterior.\"\"\"\n",
    "    alpha_0 = alpha.sum()\n",
    "    n = k.sum()\n",
    "    m = (k + alpha)\n",
    "    m /= m.sum()\n",
    "    v = (alpha+k)*(alpha_0 - alpha + n + k)/((alpha_0+n)**2*(alpha_0+n+1))\n",
    "    return m, v\n",
    "k = np.asarray([22, 43, 101])\n",
    "alpha = np.ones((3,))\n",
    "m, v = posterior_mean_var(k, alpha)\n",
    "outcome = ['consistent accept', 'inconsistent decision', 'consistent reject']\n",
    "for i in range(3):\n",
    "    print(\"Probability of\", outcome[i], m[i], \"+/-\", np.sqrt(v[i]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we have a probability of consistent accept as $0.136 \\pm 0.06$, the probability of inconsistent decision as $0.260 \\pm 0.09$ and probability of consistent reject as $0.60 \\pm 0.15$. Recall that if we'd selected papers at random (with accept rate of 1 in 4) then these values would have been 1 in 16 (0.0625), 3 in 8 (0.375) and 9 in 16 (0.5625). \n",
    "\n",
    "The other values we are interested in are the accept precision, reject precision and the agreed accept rate. Computing the probability density for these statistics is complex, because it involves [Ratio Distributions](http://en.wikipedia.org/wiki/Ratio_distribution). However, we can use Monte Carlo to estimate the expected accept precision, reject precision and agreed accept rate as well as their variances. We can use these results to give us error bars and histograms of these statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accept precision 0.5087017939007141 +/- 0.06366918406106015\n",
      "reject precision 0.8220214386714193 +/- 0.026207927373358027\n",
      "agreed accept rate 0.1841919278208472 +/- 0.034623481820352194\n"
     ]
    }
   ],
   "source": [
    "def sample_precisions(k, alpha, num_samps):\n",
    "    \"\"\"Helper function to sample from the posterior distibution of accept, \n",
    "    reject and inconsistent probabilities and compute other statistics of interest \n",
    "    from the samples.\"\"\"\n",
    "\n",
    "    k = np.random.dirichlet(k+alpha, size=num_samps)\n",
    "    # Factors of 2 appear because inconsistent decisions \n",
    "    # are being accounted for across both committees.\n",
    "    ap = 2*k[:, 0]/(2*k[:, 0]+k[:, 1])\n",
    "    rp = 2*k[:, 2]/(k[:, 1]+2*k[:, 2])\n",
    "    aa = k[:, 0]/(k[:, 0]+k[:, 2])\n",
    "    return ap, rp, aa\n",
    "\n",
    "ap, rp, aa = sample_precisions(k, alpha, 10000)\n",
    "print('accept precision', ap.mean(), '+/-', np.sqrt(ap.var()))\n",
    "print('reject precision', rp.mean(), '+/-', np.sqrt(rp.var()))\n",
    "print('agreed accept rate', aa.mean(), '+/-', np.sqrt(aa.var()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Giving an accept precision of $0.51 \\pm 0.13$, a reject precision of $0.82 \\pm 0.05$ and an agreed accept rate of $0.18 \\pm 0.07$. Note that the 'random conference' values of 1 in 4 for accept precision and 3 in 4 for reject decisions are outside the two standard deviation error bars. If it is preferred medians and percentiles could also be computed from the samples above, but as we will see when we histogram the results the densities look broadly symmetric, so this is unlikely to have much effect.\n",
    "\n",
    "#### Histogram of Monte Carlo Results\n",
    "\n",
    "Just to ensure that the error bars are reflective of the underlying densities we histogram the Monte Carlo results for accept precision, reject precision and agreed accept below. Shown on each histogram is a line representing the result we would get for the 'random committee'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAswAAADSCAYAAABekpbYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAnsElEQVR4nO3de7xcZX3v8c+XoJFLMNDsIOTCRohg4IUUQkClPVSkRAETW8F4vARMzbGlij1VIWoLfWlO42mPolW0EWhCvcRUocQLt8aDaCVgQBACxAQSSQiQcBOwHiTxd/54np2sTGbPnj33mf19v17z2jPPWmvmN2uv31rPWvOs51FEYGZmZmZm5e3R7gDMzMzMzDqZK8xmZmZmZhW4wmxmZmZmVoErzGZmZmZmFbjCbGZmZmZWgSvMZmZmZmYVuMJsbSXpOklzqpjveUmvbEVMZo1Q7bbdDSS9U9KNVcz3ZUl/04qYzNpJ0imSNrU7DmsdV5hbTNLNkp6WNLrNcWyQ9MYK00+R9LtcUX1O0hpJ5zU6joh4U0QsqWK+fSPioUZ/vlklOU9+k/PgMUmLJe1bzbLVbttDfP4lkr46jBgfl/Qv1cZYrYj4WkT8cRXzvT8iPtnIzzbrlOPmcCl5SNJ9HRBLSDq8wvRzJW3P+5FnJd0t6cxhvH/FOkUvcIW5hST1A38ABPCW9kZTlc0RsS+wH3Ah8BVJU0tnkrRnyyMza52zch4cC/w+ML+94ZQ1EONxwAnAJ0pncJ5aN2rUcbNN2/8fAuOBV0o6oQ2fP1y35v3IWOAyYKmksW2NqIO4wtxa7wFWAouBXX6qlTRJ0tWStkp6UtIXCtPeJ+n+fKX3PknH5fKDJX07L7Ne0gcLy1wi6VuSvpmXu1PSa/K0fwUmA9/JZ5MfrRR0JP8OPA1MzWei/ynps5KeAi6RNFrSP0p6OF/l+rKkvQrxzJR0Vz5zfVDSjFx+s6Q/y88Pl/RDSb+S9ISkbxaW33F2LOnlkq7K3/uXkj4haY887VxJP86xPJ3Xy5uG+48yKxURjwE3kCrOAEg6SdJPJD2Tr8icUpi2Y9vOr9+b8/hpSTdIOqQw7ShJN0l6KufPx3KOfAx4e87Tu6uI8RHgOuDo/L4h6XxJa4G1uezMnIvP5NiPKcRRdj80kFf5uXLub8m5+nNJA5+3WNKnCu/3Pknr8vdaLungwrSQ9H5Ja/M6+aIkVfffsBGk0nHz9yR9Jx9XfirpUwPbaZ4+3O2/0jF1r7x9P610xbiaCvAc4Frg+2Vi3y3nc/monP8PKh2775A0KU87srDMGknnFN5vsdJx96a83A8H9jGSbsmz3Z33JW+vFHRE/A74V2AfYEp+j8Mk/SDvF56Q9DXlyrQGqVNU2j92pYjwo0UPYB3wF8DxwIvAgbl8FHA38FnSBvoy4OQ87WzgEVJyCjgcOIR0snMH8LfAS4FXAg8Bp+flLsmf8TbgJcCHgfXAS/L0DcAbK8R6CrApP98DeGt+vyOAc4FtwAeAPYG9gEuB5cABwBjgO8Df5+WnA78CTsvvNQE4Mk+7Gfiz/PwbwMfzPDvWQZ4WwOH5+VWkndAYoB/4BTA3Tzs3x/m+vF7/HNgMqN3/fz+671HME2AicA/wufx6AvAk8Oa8zZ6WX/fl6cVte1bO/1fnnPkE8JM8bQzwKPDXebsfA5yYp10CfHUYMU4CVgOfzK8DuCnn5V6kK9BbgBNzfszJy4+m8n7oXODH+fnppH3PWNI+6dXAQXnaYuBT+fkbgCfyZ44G/gm4pRB3AN/N7zMZ2ArMaPf/3I/OejDIcTNPW5ofewNTgY0D22mePpztf6hj6kLgR/m9JgH3ko+Rg8S9N/Asaf/wpzkXXpqnVcr5j5D2M0fk/HoN8Hs5JzcC55H2Icfl9zwqL7cYeI50VXs08Lky6+LwCvEWc3wUcD7wW2B8LjuctI8bDfQBtwCXFpbfQKFOwRD7x258tD2AkfIATs7JPi6/fgD4q/z8tflgsWeZ5W4ALihTfiLwcEnZfOBf8vNLgJWFaXvkBP2D/HqXjbvM+58C/A54BngKuAuYnaedW/zsnNS/Bg4rlL0WWJ+f/zPw2UE+52Z2ViquAhYBE8vMFzlhRwEvAFML0/4HcHMhtnWFaXvnZV/R7m3Aj+575Dx5Ph+IAlgBjM3TLgT+tWT+G4A5+Xlx276OfFKXX+8B/Bfp5PcdwM8G+fxLqK7C/HzO1V+SfkrdK08L4A2Feb9ErkwXytYA/22I/dC57DyYvoF0knoSsEfJfIvZWWG+AvjfhWn7kvaB/YXYiifFy4CL2v0/96NzHlQ+bo7K044ozP8pdq8kVrv9D3VMfYjCCR0wj8oV5ncN5BOpkvkM8NY8rVLOrwFmlil/O/CjkrJ/Bi7OzxcDSwvT9gW2A5MK62KoCvO2HOeLwG+AcyrMP6v4Hdi9wlxx/9iNDzfJaJ05wI0R8UR+/XV2/kQzCfhlRGwrs9wk4MEy5YcAB+efOp6R9Azp59sDC/NsHHgS6SeWTcDBVG9zRIyNiAMi4tiIWFruvUlnm3sDdxRiuT6XV/oOpT5KqnzfLmm1pPeWmWcc6ez/l4WyX5LOZgc8NvAkIv4rP23oTVA2osyKiDGkk8gjSdsgpBw8uyQHTwYOKvMehwCfK8z3FGlbn0D1+TFUjGMj4pCI+IuI+E1hWjFXDwH+uiTmSaT9QqX90A4R8QPgC8AXgcclLZK0X5lZD6aQpxHxPOkKU9lcJZ1AOE+tqNJxs49UGS1u38Xn5coqbf9DHVMPLnmv4jFosNiXRcS2iHgBuJpdj/mD5XylY/6JJfG9E3hFue+a8+0phnfMXxkRY4H9Sb8Y/8HABEnjJS2V9IikZ4GvsnNfWM5w9o9dwTeBtIBSW95zgFGSBg4Qo4GxSu2KNwKTJe1Z5mC1ETiszNtuJF3BnVLhoycVYtiD9JPy5lwUw/8muygu/wTpbPSoSG0oy8Va7jvs+oapjej7ACSdDPyHpFsiYl3JZ71ISsaBO48nk5qtmDVNRPxQ0mLgH0lXVzaSrqC8r4rFNwILIuJrpRNyO8N3DPaxtUU76HsMxLGgTByvZfD90K5vGPF54POSxpOuDH8EKO1ObjMpTwfefx/ST8vOVRtSFcfNe0lXRCeSfvGAwjGvYDjbf6Vj6qPsbPIE6bgzWOwTSb/ETJf0p7l4b+BlksblOAbL+YHj5b1lyn8YEacN9rnseszfl9R8ZPPgs5cXEc9L+gvgQUlXRsTPgL8nrctjIuJJSbNIJ847FisTb7X7x67gK8ytMYv008hU0g1Dx5La/f2IdEPD7aRkXChpH0kvk/T6vOzlwIclHa/k8HyAvR14VtKF+WaEUZKO1q534h4v6U+U7g7+EKkpw8o87XFSG6265avXXwE+mw+gSJog6fQ8yxXAeZJOlbRHnnZk6ftIOjvvaCDdYBik9Vb8rO2kA/QCSWPyuvifpLNds2a7FDhN0rGkbe4sSafn/HuZUneME8ss92VgvqSjYMeNq2fnad8FXiHpQ0o3z46RdGKe9jjQn094G+ErwPslnZj3J/tIOkPSGCrvh3aQdEJe/iWkplj/j5I8zb5OyvtjlboD+1/AbRGxoUHfxXrbLCocN/Ox4GrSTed752PKe4Z4z6G2/0rH1GWkHN4/5/gHKnzOu0mV+CMKsb+K9CvvO6ic85cDn5Q0Jcd4jKTfy8u8StK7Jb0kP06Q9OrC575Z0smSXgp8kpRvA1edh3XMj4gncyx/m4vGkJt+SZpAOkkuKn3/4ewfu4IrzK0xh9QO6uGIeGzgQTo7eyfpp9mzSG10HyYl1dsBIuLfgAWkg89zwL8DB+SdxVmkRFxPuvJ6OfDywudem9/naVIC/0lEvJin/T3wCaWfSj7cgO94IenmjJVKP9f8B2lnQUTcTrpR4bOkm/9+SOHKU8EJwG2Snif9HHRBRKwvM98HSAfqh4Afk9bNlQ34DmYVRcRWUlv7v8kHopmkn223kq6ofIQy+9WIuAb4NKmbpmdJV4/elKc9R7oh5ixSE4W1wB/lRf8t/31S0p0NiH8V6VecL5D2C+tIbRcp7FN22w+V2I9U8Xia9LP0k6Sr7qWftYJ01fnbpIr4YcDser+DjRgVj5v5QtBfko55j5F6dfgG6cJQWVVu/8dS/pj6d6TtfT1wY/68SrFfVow7x/5lUhveSjn/GVLl/EbSTYNXkO5JeA74Y1IObc7LfZp01X3A14GLSU0xjifVLwZcAizJx/xzqM6lpEr4Mfn7H0c6hn+PdLJStEudYjj7x26hiEb84medRtIlpAb+72p3LGYjkVJXTpdHxFXtjsVsJJD0adIN3nOGnLnHKDUX2xQRu/XBbo3RtTV9M7NOJWlv0s+T5X4hMbMGUOqX+JjcdGE6MBe4pt1xWW/yTX9mZg2U2/GvI/VF/uMhZjez2o0hNcM4mNS/8v8hNUU0azg3yTAzMzMzq8BNMszMzMzMKnCF2czMzMysgo5vwzxu3Ljof/LJXQuPP749wZh1gDvuuOOJiOgbes7WGzduXPT397c7jPrdcceur73PsRqNmHx1zlgPqJSvHV9h7u/vZ1VphXnVqvYEY9YBJA01JGvb9Pf3s6oX8lPa9XUvfCdrixGTr84Z6wGV8tVNMszMzNpE0pWStkgqHQoZSR+WFErDKQ+UzZe0TtKawmiqKI0Ge0+e9nmptAZrZvVwhdnMzKx9FgMzSgslTSKNBvdwoWwqaaS3o/Iyl0kalSd/CZgHTMmP3d7TzGrnCrOZmVmbRMQtpKGMS30W+ChQ7Pt1JrA0Il6IiPWk/r6nSzoI2C8ibo3UV+xVwKzmRm42srjCbGZm1kEkvQV4JCLuLpk0AdhYeL0pl03Iz0vLzaxBOv6mPzMzs5EiD6v+ceCPy00uUxYVysu9/zxS0w0mT55cY5RmI48rzEb/Rd8bcp4NC89oQSRm1km8b2iLw4BDgbvzfXsTgTslTSddOZ5UmHcisDmXTyxTvpuIWAQsApg2bZqH+q3A278VuUmGmZlZh4iIeyJifET0R0Q/qTJ8XEQ8BiwHZksaLelQ0s19t0fEo8Bzkk7KvWO8B7i2Xd/BrBe5wmxmZtYmkr4B3AocIWmTpLmDzRsRq4FlwH3A9cD5EbE9T/5z4HLSjYAPAtc1NXCzEcZNMszMzNokIt4xxPT+ktcLgAVl5lsFHN3Q4MxsB19hNjMzMzOrwFeYrSq++cHMzMxGKleYzczMzGrgi0kjx5BNMjzOvVl3cc6amZk1VjVtmBfjce7NuslinLNmZmYNM2SF2ePcm3UX56yZmVlj1dRLhse5N+suzcxZSfMkrZK0auvWrQ2M2szMrDMM+6a/Zo9znz/DY92bNUizc9ZD7ZqZWa+r5QpzcZz7Dewc5/4VNGCce0gH4IiYFhHT+vr6agjRzAqanrNmZma9bNgVZo9zb9ZdnLNmZmb1GbJJRh7n/hRgnKRNwMURcUW5eSNitaSBce63sfs494uBvUhj3Huce7MmcM5aK7kfWjMbCYasMHuce7Pu4py1alRT0TUzs6SmXjLMzMzMzEYKD41tZmZmI4p/YbHhcoW5x3mnYGbWuSRdCZwJbImIo3PZPwBnAb8FHgTOi4hn8rT5wFxgO/DBiLghlx/PznsOvg9ckAcdMrMGcJMMMzOz9lnM7sPO3wQcHRHHAL8A5oOHsjdrJ19htobx3fJmZsMTEbdI6i8pu7HwciXwtvx8x1D2wHpJA0PZbyAPZQ8gaWAoe/dsY9YgvsJsZmbWud7Lzoqvh7I3axNfYTYzM+tAkj5O6h/9awNFZWbzUPYdzr++9gZXmM3MzDqMpDmkmwFPLdy856HszdrETTLMzMw6iKQZwIXAWyLivwqTPJS9WZv4CrOZmVmblBvKntQrxmjgplT/ZWVEvN9D2Zu1jyvMZmZmbTLIUPZXVJjfQ9mbtYGbZJiZmZmZVTBkhVnSlZK2SLq3UPYPkh6Q9HNJ10gaW5g2X9I6SWsknV4oP17SPXna53M7KzNrMOesmZlZY1VzhXkxHoXIrJssxjlrZmbWMENWmCPiFuCpkrIbI2JbfrmSnd3Z7BiFKCLWAwOjEB1EHoUod48zMAqRmTWYc9bMzKyxGtGGuaGjEIFHIjJrMo8cZmZmNgx1VZibMQoRpJGIImJaREzr6+urJ0QzK2jWyGHOVzMz62U1dyvnUYjMuotz1szMrDY1XWH2KERm3cU5a2ZmVrshrzB7FCKz7uKcNTMza6whK8wehcisuzhnzczMGstDY5uZWVP1X/S9IefZsPCMFkRiZlYbD41tZmZmZlaBK8xmZmZmZhW4wmxmZtYmkq6UtEXSvYWyAyTdJGlt/rt/Ydp8SeskrZF0eqH8eEn35Gmfz73bmFmDuMJsZmbWPouBGSVlFwErImIKsCK/RtJUYDZwVF7mMkmj8jJfAuaRuoacUuY9zawOvunPzKzHVHOTnXWGiLhFUn9J8UxS15AAS4CbSf2ozwSWRsQLwHpJ64DpkjYA+0XErQCSrgJm4a4gzRrGV5jNzMw6y4F58CDy3/G5fAKwsTDfplw2IT8vLd+NpHmSVklatXXr1oYHbtarXGE2MzPrDuXaJUeF8t0LIxZFxLSImNbX19fQ4Mx6mZtkmJmZdZbHJR0UEY9KOgjYkss3AZMK800ENufyiWXKrUu4r/LO5yvMZmZmnWU5MCc/nwNcWyifLWm0pENJN/fdnpttPCfppNw7xnsKy5hZA/gKs5mZWZtI+gbpBr9xkjYBFwMLgWWS5gIPA2cDRMRqScuA+4BtwPkRsT2/1Z+TetzYi3Szn2/4M2sgV5jNeoykK4EzgS0RcXQuOwD4JtAPbADOiYin87T5wFxgO/DBiLghlx/PzgPw94ELIqJsu0gzq01EvGOQSacOMv8CYEGZ8lXA0Q0MzcwKhmyS4U7VzbrOYtyvq5mZWcNU04Z5MT74mnWNiLgFeKqkeCapP1fy31mF8qUR8UJErAcG+nU9iNyva76qfFVhGTMzsxFlyCYZ7lTdGsl3ArfNLv26Sir267qyMN9A/60vUmW/rmZmZr2u1l4ymtapOrhjdbMWqrtfV+ermZn1ukZ3K1f3wRfcsbpZEzyem1nQ6H5dna9mZtbrau0lw52qm3WXgX5dF7J7v65fl/QZ4GB29uu6XdJzkk4CbiP16/pPrQ/bzGx4qmn6ZzZctV5hdqfqZh0q9+t6K3CEpE25L9eFwGmS1gKn5ddExGpgoF/X69m9X9fLSTcCPojvOTAzsxFqyCvM7lTdrLu4X1czM7PGqqaXDB98zczMzGzEavRNf2ZmZmZmPcUVZjMzMzOzClxhNjMzMzOroNZu5awDuOscMzMzs+bzFWYzM7MOJOmvJK2WdK+kb0h6maQDJN0kaW3+u39h/vmS1klaI+n0dsZu1mtcYTYzM+swkiYAHwSmRcTRwChgNnARsCIipgAr8mskTc3TjwJmAJdJGtWO2M16kSvMZmZmnWlPYC9JewJ7k0bInQksydOXALPy85nA0oh4ISLWkwYcmt7acM16lyvMZmZmHSYiHgH+kTQ42KPAryLiRuDAPHou+e/4vMgEYGPhLTblMjNrAN/0Z2bWZXzDb+/LbZNnAocCzwD/JuldlRYpUxZl3nceMA9g8uTJ9QdqNkL4CrOZmVnneSOwPiK2RsSLwNXA64DHJR0EkP9uyfNvAiYVlp9IasKxi4hYFBHTImJaX19fU7+AWS9xhdnMzKzzPAycJGlvSQJOBe4HlgNz8jxzgGvz8+XAbEmjJR0KTAFub3HMZj2rrgqzu7wx6y7OWbPuEBG3Ad8C7gTuIR2vFwELgdMkrQVOy6+JiNXAMuA+4Hrg/IjY3obQzXpSzRVmd3lj1l2cs2bdJSIujogjI+LoiHh37gHjyYg4NSKm5L9PFeZfEBGHRcQREXFdO2M36zX1Nslwlzdm3cU5a2ZmNkw1V5jd5Y1Zd3HOmpmZ1aaeJhnFLm8OBvZpRJc3+b3nSVoladXWrVtrDdHMCpqVs85XMzPrdfU0yWhKlzfgbm/MmsTdVJmZmdWgnoFLdnR5A/yG1OXNKuDXpK5uFrJ7lzdfl/QZ0tUtd3lj1lrOWTOzLlXNgEUbFp7RgkhGpporzBFxm6SBLm+2AT8jdXmzL7BM0lzSAfrsPP9qSQNd3mzDXd6YtZRz1szMrDZ1DY0dERcDF5cUv0C6clVu/gXAgno+08xq55y1TuWrZ2bWyTzSn5mZmZlZBa4wm5mZmZlV4AqzmZmZmVkFrjCbmZmZmVXgCrOZmZmZWQWuMJuZmZmZVeAKs5mZmZlZBa4wm5mZdSBJYyV9S9IDku6X9FpJB0i6SdLa/Hf/wvzzJa2TtEbS6e2M3azXuMJsZmbWmT4HXB8RRwKvAe4HLgJWRMQUYEV+jaSpwGzgKGAGcJmkUW2J2qwHucJsZmbWYSTtB/whcAVARPw2Ip4BZgJL8mxLgFn5+UxgaUS8EBHrgXXA9FbGbNbLXGE2MzPrPK8EtgL/Iulnki6XtA9wYEQ8CpD/js/zTwA2FpbflMt2IWmepFWSVm3durW538Csh7jCbGZm1nn2BI4DvhQRvw/8mtz8YhAqUxa7FUQsiohpETGtr6+vMZGajQB71rOwpLHA5cDRpMR8L7AG+CbQD2wAzomIp/P884G5wHbggxFxQz2fb2bD45w16xqbgE0RcVt+/S1ShflxSQdFxKOSDgK2FOafVFh+IrC5ZdG2SP9F32t3CDZC1XuF2TckmHUX56xZF4iIx4CNko7IRacC9wHLgTm5bA5wbX6+HJgtabSkQ4EpwO0tDNmsp9V8hblwQ8K5kG5IAH4raSZwSp5tCXAzcCGFGxKA9ZIGbki4tdYYepnPoq3RnLNmXecDwNckvRR4CDiPdKFrmaS5wMPA2QARsVrSMlKlehtwfkRsb0/YZr2nniYZxRsSXgPcAVxAyQ0Jkoo3JKwsLF/2hgSzak4WNiw8owWR9Jym5KykecA8gMmTJzcverMRJiLuAqaVmXTqIPMvABY0MyazkaqeJhlNuSEBfBevWZP4JiIzM7Ma1FNhLndDwnHkGxIAar0hwQdgs6ZoWs6amZn1sporzL4hway7OGfNzMxqU1e3cviGBLNu45w1MzMbproqzL4hway7OGfNzMyGzyP9mZmZmZlVUG+TDDMzayD3wW5m1nl8hdnMzMzMrAJXmM3MzMzMKnCF2czMzMysAleYzczMzMwqcIXZzMzMzKwC95JhZmZm1gOq6WVnw8IzWhBJ7/EVZjMzsw4laZSkn0n6bn59gKSbJK3Nf/cvzDtf0jpJaySd3r6ozXqPK8xmZmad6wLg/sLri4AVETEFWJFfI2kqMBs4CpgBXCZpVItjNetZrjCbmZl1IEkTgTOAywvFM4El+fkSYFahfGlEvBAR64F1wPQWhWrW81xhNjMz60yXAh8FflcoOzAiHgXIf8fn8gnAxsJ8m3KZmTVA3Tf95Z98VgGPRMSZkg4Avgn0AxuAcyLi6TzvfGAusB34YETcUO/nm1n1nK/WzUbSDU2SzgS2RMQdkk6pZpEyZVHmfecB8wAmT55cT4hmI0ojrjC7fZVZ93C+mnWH1wNvkbQBWAq8QdJXgcclHQSQ/27J828CJhWWnwhsLn3TiFgUEdMiYlpfX18z4zfrKXVVmN2+yqx7OF/NukdEzI+IiRHRTzp5/UFEvAtYDszJs80Brs3PlwOzJY2WdCgwBbi9xWGb9ax6rzBfShPaV0maJ2mVpFVbt26tM0Qzyy7F+WrW7RYCp0laC5yWXxMRq4FlwH3A9cD5EbG9bVGa9ZiaK8zF9lXVLlKmbLf2VeCfjMwazflq1r0i4uaIODM/fzIiTo2IKfnvU4X5FkTEYRFxRERc176IzXpPPTf9DbSvejPwMmC/YvuqiHi0lvZVZtYUzlczM7Ma1XyF2e2rzLqH89XMzKx2dXcrV8ZCYJmkucDDwNmQ2ldJGmhftY0R3L6qmq6RzFrE+WpmZjaEhlSYI+Jm4Ob8/Eng1EHmWwAsaMRnmlltnK9mZmbD04wrzGZmZmbD4l9frZN5aGwzMzMzswpcYTYzMzMzq8BNMqwrVfPT3YaFZ7QgEjMzs+7h42dtfIXZzMzMzKwCV5jNzMzMzCpwkwwzsxZxLwBmZt3JV5jNzMzMzCpwhdnMzMzMrAJXmM3MzMzMKnCF2czMrMNImiTp/0q6X9JqSRfk8gMk3SRpbf67f2GZ+ZLWSVoj6fT2RW/We1xhNjMz6zzbgL+OiFcDJwHnS5oKXASsiIgpwIr8mjxtNnAUMAO4TNKotkRu1oNqrjD77NesuzhnzbpHRDwaEXfm588B9wMTgJnAkjzbEmBWfj4TWBoRL0TEemAdML2lQZv1sHq6lRs4+71T0hjgDkk3AeeSzn4XSrqIdPZ7YcnZ78HAf0h6VURsr+8rmFmVnLNmXUhSP/D7wG3AgRHxKKRKtaTxebYJwMrCYptyWel7zQPmAUyePLmJUVs382iAu6u5wpwTdiBpn5NUPPs9Jc+2BLgZuJDC2S+wXtLA2e+ttcZgZtVzztpI0GsHekn7At8GPhQRz0oadNYyZbFbQcQiYBHAtGnTdptuZuU1ZOCSRp795vfr2jNgD0xg3cBXrMw6n6SXkCrLX4uIq3Px45IOyrl6ELAll28CJhUWnwhsbl20Zr2t7pv+Ss9+K81apqzs2W1ELIqIaRExra+vr94Qzayg0TnrfDVrPKVLyVcA90fEZwqTlgNz8vM5wLWF8tmSRks6FJgC3N6qeM16XV1XmH32a9ZdnLPN41+XrMFeD7wbuEfSXbnsY8BCYJmkucDDwNkAEbFa0jLgPtL9Cuf7fgOzxqm5wlzF2e9Cdj/7/bqkz5BuIPLZrzVVr7VlrJdz1qx7RMSPKf8rD8CpgyyzAFjQtKDMRrB6rjD77Nesuzhnzawt/AuMdbt6esnw2a9ZF3HOmpmZ1cYj/ZmZmZmZVeAKs5mZmZlZBa4wm5mZmZlV0JCBS8zMzMxs5BhpPVH5CrOZmZmZWQW+wmxmNgR3iWVmNrL5CrOZmZmZWQW+wjxMvtJkZmZmNrK4wmwj2ki7acHMzMyGzxVmMzMbUXyi3Fj9F32PDWXKzHqJK8xmZmZm1nBDnTh104lpyyvMkmYAnwNGAZdHxMJWxzAYnxGb7aqT87VRyuX9htaHYdYQIyFnzdqhpRVmSaOALwKnAZuAn0paHhH3tTIOMxua89VGsm5stuGcNWueVl9hng6si4iHACQtBWYCTmbrWN144GyQjs5X/yJktpuG56zzzJqpUcfXVhynW11hngBsLLzeBJxY75s6oa0bdGHF2/lq1l2akrNm7dQpx4xWV5hVpix2m0maB8zLL58XrCmZod44xgFP1PsmDeaYhtZp8UCOSZ9uzJtV+T6HNObThlRbvkprSmbpxP9bRYUvnmL/9Jlti6VGXbfOs66KuyRfB4u9VfkKVeRsFfla7wd3as506rbViXF1YkxQZ1z1Hl9bXWHeBEwqvJ4IbC6dKSIWAYuaFYSkVRExrVnvXwvHNLROiwc6M6YGaki+dvM66tbYHXfrdUjsQ+bsSDy+guMajk6MCdofV6uHxv4pMEXSoZJeCswGlrc4BjOrjvPVrLs4Z82apKVXmCNim6S/BG4gdXlzZUSsbmUMZlYd56tZd3HOmjVPy/thjojvA99v9eeWaNrPUXVwTEPrtHigM2NqmAblazevo26N3XG3XkfE3gHH2I5YD2U4rup1YkzQ5rgUsds9PGZmZmZmlrW6DbOZmZmZWVfp6QqzpBmS1khaJ+miMtNnSvq5pLskrZJ0cjvjKcx3gqTtkt7WzHiqiUnSKZJ+ldfRXZL+tt0xFeK6S9JqST9sd0ySPlJYR/fm/98BzY6r3epZL9XmQwfGvUHSPQP7jQ6L++WSviPp7pwb51W7bLPVGXsnr/P9JV2TjyW3Szq62mW7TRXr4khJt0p6QdKHh7NsG+NqyrZVRUzvzNvMzyX9RNJrql22jXE1LQ+riGvQ+lrL8iwievJBuuHhQeCVwEuBu4GpJfPsy85mKccAD7QznsJ8PyC1QXtbB6yjU4Dvdtj/bSxp5KrJ+fX4dsdUMv9ZwA9atc7a9ahnvQx32U6JO7/eAIzrxPUNfAz4dH7eBzyV523b+q439i5Y5/8AXJyfHwmsqGU76/RHletiPHACsAD48HCWbUdczdq2qozpdcD++fmbgNs6ZF2VjatZ62oYcZWtr7Uyz3r5CvOOIUIj4rfAwBChO0TE85HXOLAPZQZlaGU82QeAbwNbmhjLcGNqpWpi+u/A1RHxMEBENHtdDXc9vQP4RpNj6gT1rJd2bnvd+v+sJu4AxkgS6QDzFLCtymWbqZ7Y26mauKcCKwAi4gGgX9KBVS7bTao5pm6JiJ8CLw532TbF1SzVxPSTiHg6v1xJ6jO7qmXbFFcz1VNfa1me9XKFudwQoRNKZ5L0VkkPAN8D3tvOeCRNAN4KfLmJcQwrpuy1+WfS6yQd1QExvQrYX9LNku6Q9J4OiAkASXsDM0gnPb2unvVS9bJNUO//M4Ab87Y3r9xyTVJN3F8AXk0arOIe4IKI+F2VyzZTPbFDZ6/zu4E/AZA0nTRS2MQql+0m9XyfZq6Let+7GdvWcGOaC1xX47Ktigual4f11Ndalmct71auhaoa1jcirgGukfSHwCeBN7YxnkuBCyNiu+of/rsa1cR0J3BIRDwv6c3AvwNT2hzTnsDxwKnAXsCtklZGxC/aGNOAs4D/jIinmhRLJ6lnvQxn2Uar9//5+ojYLGk8cJOkByLiloZHubtq4j4duAt4A3AYKb4fVblsM9Uce0Q8S2ev84XA5yTdRaro/4x0Zbzd67zR6vk+zVwX9b53M7atqmOS9EekiulAm9yOWFdl4oLm5WE99bWW5VkvX2GualjfAfmffpikcW2MZxqwVNIG4G3AZZJmNSmeqmKKiGcj4vn8/PvAS5q4jqqKKc9zfUT8OiKeAG4BXkPzDGdbmk1n/HzfCvWsl2HlZ4PV9f+MiM357xbgGtJPgq1QTdznkZorRUSsA9aT2tW2c31T5ecPFntHr/O8jzwvIo4F3kNqf72+mmW7TD3fp5nroq73btK2VVVMko4BLgdmRsSTw1m2DXE1Mw/rqa+1Ls/qaQDdyQ/SVciHgEPZ2RD8qJJ5DmdnI/LjgEcGXrcjnpL5F9P8m/6qWUevKKyj6cDDzVpHw4jp1aQ2g3sCewP3Ake3M6Y838tJ7S73aeb/rVMe9ayX4eZDB8W9DzCm8PwnwIxOiRv4EnBJfn5g3qeNa+f6bkDsnb7Ox7Lz5sT3AVe1extv17oozHsJu97017R1UWdcTdm2qtxuJgPrgNfV+n1aHFfT8rDKuMrW11qZZy1JtHY9gDcDvyDdQfnxXPZ+4P35+YXAatLPgLcCJ7cznpJ5F9PkCnOV6+gv8zq6m3QDwOvaHVN+/RFSTxn3Ah/qkJjOBZY2O5ZOetSzXsot2+lxk+7Evjs/Vnda3MDBwI2kpgH3Au/qhPVdT+xdsM5fC6wFHgCuJvcw0AnrvA3r4hWkK37PAs/k5/s1e13UGlczt60qYroceJpU/7gLWNWK7abWuJqdh1XENWh9rVV55pH+zMzMzMwq6OU2zGZmZmZmdXOF2czMzMysAleYzczMzMwqcIXZzMzMzKwCV5jNzMzMzCpwhdnMzMzMrAJXmM3MzMzMKnCF2czMzMysgv8PQcUlSCYDaIYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 3, figsize=(12, 3))\n",
    "_ = ax[0].hist(ap, 20)\n",
    "_ = ax[0].set_title('Accept Precision')\n",
    "ax[0].axvline(0.25, linewidth=4, color='r')\n",
    "_ = ax[1].hist(rp, 20)\n",
    "_ = ax[1].set_title('Reject Precision')\n",
    "ax[1].axvline(0.75, linewidth=4, color='r')\n",
    "_ = ax[2].hist(aa, 20)\n",
    "_ = ax[2].set_title('Agreed Accept Rate')\n",
    "ax[2].axvline(0.10, linewidth=4, color='r')\n",
    "fig.savefig('ratio-distributions.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Choice and Prior Values\n",
    "\n",
    "In the analysis above we've minimized the modeling choices: we made use of a Bayesian analysis to capture the uncertainty in counts that can be arising from statistical sampling error. To this end we chose an uninformative prior over these probabilities. However, one might argue that the prior should reflect something more about the underlying experimental structure: for example we *know* that if the committees made their decisions independently it is unlikely that we'd obtain an inconsistency figure much greater than 37.5% because that would require committees to explicitly collude to make inconsistent decisions: the random conference is the worst case. Due to the accept rate, we also expect a larger number of reject decisions than reject. This also isn't captured in our prior. Such questions actually move us into the realms of modeling the process, rather then performing a sensitivity analysis. However, if we wish to model the decision process as a whole we have a lot more information available, and we should make use of it. The analysis above is intended to exploit our randomized experiment to explore how inconsistent we expect two committees to be. It focusses on that single question, it doesn't attempt to give answers on what the reasons for that inconsistency are and how it may be reduced. The additional maths was needed only to give a sense of the uncertainty in the figures. That uncertainty arises due to the limited number of papers in the experiment.\n",
    "\n",
    "### Initial Conclusions\n",
    "\n",
    "Under the simple model we have outlined, we can be confident that there is inconsistency between two independent committees, but the level of inconsistency is much less than we would find for a random committee. If we accept that the bias introduced by the Area Chairs knowing when they were dealing with duplicates was minimal, then if we were to revisit the NIPS 2014 conference with an independent committee then we would expect between **38% and 64% of the presented papers to be the same**. If the conference was run at random, then we would only expect 25% of the papers to be the same.\n",
    "\n",
    "It's apparent from comments and speculation about what these results mean, that some people might be surprised by the size of this figure. However, it only requires a little thought to see that this figure is likely to be large for any highly selective conference if there is even a small amount of inconsistency in the decision making process. This is because once the conference has chosen to be 'highly selective' then because by definition only a small percentage of papers are to be accepted. Now if we think of a type I error as accepting a paper which should be rejected, such errors are easier to make because by definition many more papers should be rejected. Type II errors (rejecting a paper that should be accepted) are less likely becaue (by setting the accept rate low) there are fewer papers that should be accepted in the first place. When there is a difference of opinion between reviewers, it does seem that many of the arugments can be distilled down to (a subjective opinion) about whether controlling for type I or type II errors is more important. Further, normally when discussing type I and type II errors we believe that the underlying system of study is genuinely binary: e.g. diseased or not diseased. However, for conferences the accept/reject boundary is not a clear separation point, there is a continuum (or spectrum) of paper quality (as there also is for some diseases). And the decision boundary often falls in a region of very high density. To better quantify these ideas we can explore our duplication experiment in more detail, by introducing the paper scores, that's a task we will perform in a fresh notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}