diff --git a/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_1.ipynb b/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_1.ipynb new file mode 100644 index 0000000..65bb3f3 --- /dev/null +++ b/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_1.ipynb @@ -0,0 +1,668 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "4304eb40-80fd-46b5-a6ef-6e7a84d33dbe", + "metadata": {}, + "source": [ + "## Monte Carlo Tests\n", + "To motivate resampling methods, suppose we wish to infer from measurements whether the weights of adult human males in a medical study are normally distributed [[1](https://www.jstor.org/stable/2333709https://www.jstor.org/stable/2333709)]. This may be prerequisite to performing other statistical tests, many of which are developed under the assumption that samples are drawn from a normally distributed population (although some tests are quite robust to deviations from this assumption, as we will see later). The weights are recorded in the array `x` below." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "02824778-802d-4d9d-8cd5-0a69cc028d13", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "x = np.array([148, 154, 158, 160, 161, 162, 166, 170, 182, 195, 236]) # weights (lbs)" + ] + }, + { + "cell_type": "markdown", + "id": "fe14279d-496f-45fe-9ce0-937ff5844a3b", + "metadata": {}, + "source": [ + "One way of testing for departures from normality, chosen based on its simplicity rather than its sensitivity, is the [Jarque-Bera test [2]](https://www.sciencedirect.com/science/article/abs/pii/0165176580900245) implemented in SciPy as `scipy.stats.jarque_bera`. The test, like many other hypothesis tests, computes a *statistic* based on the sample and compares its value to the distribution of the statistic derived under the *null hypothesis* that the sample is normally distributed. If the value of the statistic is extreme compared to this *null distribution* - that is, if there is a low probability of sampling such data from a normally distributed population - then we have evidence to reject the null hypothesis.\n", + "\n", + "The statistic is calculated based on the skewness and kurtosis of the sample as follows." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "8475312e-7810-4f61-8fed-7fe8297afe28", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "6.982848237344646" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from scipy import stats\n", + "\n", + "def statistic(x):\n", + " # Calculates the Jarque-Bera Statistic\n", + " # Compare against `scipy.stats.jarque_bera`:\n", + " # https://github.com/scipy/scipy/blob/4cf21e753cf937d1c6c2d2a0e372fbc1dbbeea81/scipy/stats/_stats_py.py#L1583-L1637\n", + " n = len(x)\n", + " mu = np.mean(x, axis=0)\n", + " x = x - mu # remove the mean from the data\n", + " s = stats.skew(x) # calculate the sample skewness\n", + " k = stats.kurtosis(x) # calculate the sample kurtosis\n", + " statistic = n/6 * (s**2 + k**2/4)\n", + " return statistic\n", + "\n", + "stat1 = statistic(x)\n", + "stat2, _ = stats.jarque_bera(x)\n", + "np.testing.assert_allclose(stat1, stat2, rtol=1e-14)\n", + "stat1" + ] + }, + { + "cell_type": "markdown", + "id": "7a38c293-1414-4d5c-8c6f-84d9d2e77e93", + "metadata": {}, + "source": [ + "Note that the value of the statistic is unaffected by the scale and location of the data." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "138b5da5-d953-4630-97c4-ebd8ad6420c7", + "metadata": {}, + "outputs": [], + "source": [ + "old_location = np.mean(x)\n", + "old_scale = np.std(x)\n", + "x_new = (x - old_location) / old_scale # make location 0 and scale 1\n", + "stat3 = statistic(x_new)\n", + "np.testing.assert_allclose(stat1, stat3, rtol=1e-14)" + ] + }, + { + "cell_type": "markdown", + "id": "69fad794-1d58-49c9-9c75-09f642cdfbf0", + "metadata": {}, + "source": [ + "Consequently, it can be shown that large samples drawn from a normal distribution with any mean and variance will produce statistic values that are distributed according to the chi-squared distribution with two degrees of freedom. We can check this numerically by drawing 1000 samples of size 500 from a standard normal distribution and computing the value of the statistic for each sample." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "9e2c2e75-1f3b-477c-9b6a-20d6bbfd6efd", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA/RElEQVR4nO3deXhTZfbA8e9J0tKWfZdNWWRvaQuF4iCbC5sIo6yOIoiI6CDuA+MGLqOOMC64DoLAb0ZBRUHGDVRAQdlaKJvsilJQtkJZSpc07++PpDEtaZuWpmna83mePEnufe/NyaXk3Pvee88rxhiUUkpVXJZAB6CUUiqwNBEopVQFp4lAKaUqOE0ESilVwWkiUEqpCs4W6ACKqk6dOqZp06aBDkMppYJKYmLicWNMXW/zgi4RNG3alISEhECHoZRSQUVEfslvnnYNKaVUBaeJQCmlKjhNBEopVcEF3TmC4sjKyiI5OZn09PRAh6KUUn4VFhZG48aNCQkJ8XmZCpEIkpOTqVq1Kk2bNkVEAh2OUkr5hTGGEydOkJycTLNmzXxerkJ0DaWnp1O7dm1NAkqpck1EqF27dpF7P/yaCESkn4jsFpF9IjLFy/xeIpIqIkmuxxN+jMVfq1ZKqTKjOL91fusaEhEr8DpwLZAMbBSRpcaYH/M0XW2MGejreu0OLZutlFIlyZ9HBF2AfcaYn4wxmcBCYPDFrnTPkTME6xgKixcvRkTYtWtXQD7/wIEDvPfee4W2S0pK4vPPP3e/X7p0Kc8//7zPn9O0aVOGDBnifr9o0SLGjBlT6HJVqlRxxxkZGXnB/AMHDhAeHk5sbCxt27alS5cuzJ8/3+c4836vvBISEpg0aRIA06ZNY8aMGYXG7Onll18mLS3N/X7AgAGcOnWqSOtQKhD8mQgaAQc93ie7puV1hYhsEZEvRKS9txWJyHgRSRCRhGyHYc+Rs/6I1+8WLFjAlVdeycKFCwPy+cVNBIMGDWLKlAt69gqUkJDAjh07ihxjYVq0aMHmzZvZuXMnCxcu5KWXXmLu3Lk+xVlQIrDb7cTFxTFz5sxix5Y3EXz++efUqFGj2OtTqrT4MxF466jKuyu/CbjMGBMNvAos8bYiY8wsY0ycMSYOYN1PJ0oyzlJx9uxZvv/+e+bMmZMrEfz222/06NGDmJgYIiMjWb16NXPmzOH+++93t3n77bd54IEHOHDgAG3atGHcuHFERkZy88038/XXX9OtWzdatmzJhg0bAOfe7KhRo7jqqqto2bIlb7/9NgBTpkxh9erVxMTE8NJLL5Gens5tt91GVFQUsbGxrFy5kszMTJ544gnef/99YmJieP/995k3bx4TJ04E4MiRI9xwww1ER0cTHR3NDz/84PX7PvTQQzz77LMXTM+7px0ZGcmBAweKtU2bN2/Oiy++6P7x9ozzww8/JDIykujoaHr06OH1e02bNo3x48fTp08fbr31VlatWsXAgX/0Um7ZsuWCbZi3zcSJE5k3bx4zZ87k8OHD9O7dm969ewPOI6Pjx48D8OKLLxIZGUlkZCQvv/wy4EzMbdu25Y477qB9+/b06dOH8+fPF2tbKHUx/Hn5aDLQxON9Y+CwZwNjzGmP15+LyBsiUscYczy/lYZYLaz76QSj/9S0WEE9+b8d/Hj4dOENi6Bdw2pMvd7rwYzbkiVL6NevH61ataJWrVps2rSJjh078t5779G3b18effRRsrOzSUtLo2PHjnTo0IEXXniBkJAQ5s6dy7///W8A9u3bx4cffsisWbPo3Lkz7733HmvWrGHp0qU8++yzLFmyBICtW7eybt06zp07R2xsLNdddx3PP/88M2bM4NNPPwXgX//6FwDbtm1j165d9OnThz179vDUU0+RkJDAa6+9Bjh/YHNMmjSJnj17snjxYrKzszl71vvR2fDhw3njjTfYt2/fxWzaQnXs2NFrV9tTTz3FsmXLaNSoEadOnSI0NPSC7zVt2jQSExNZs2YN4eHhrFq1Ktc6vG3D/EyaNIkXX3yRlStXUqdOnVzzEhMTmTt3LuvXr8cYQ3x8PD179qRmzZrs3buXBQsW8PbbbzN8+HA++ugjbrnllovfMEoVgT+PCDYCLUWkmYiEAiOBpZ4NROQScZ3iFpEurngK3N2vUsnG+p9TcATZSeMFCxYwcuRIAEaOHMmCBQsA6Ny5M3PnzmXatGls27aNqlWrUrlyZa666io+/fRTdu3aRVZWFlFRUQA0a9aMqKgoLBYL7du35+qrr0ZEiIqKyrVnPXjwYMLDw6lTpw69e/d2Hy14WrNmDaNGjQKgTZs2XHbZZezZs6fA77FixQruuusuAKxWK9WrV/fazmq18vDDD/Pcc88VbUMVUX7ni7p168aYMWN4++23yc7Oznf5QYMGER4e7nWeL9vQF2vWrOGGG26gcuXKVKlShRtvvJHVq1cDzn/PmJgYADp16lTsoyOlLobfjgiMMXYRmQgsA6zAO8aYHSIywTX/LWAocJeI2IHzwEhTyJngypWspJzLZO/Rs7S+pGqR4ypsz90fTpw4wYoVK9i+fTsiQnZ2NiLCCy+8QI8ePfjuu+/47LPPGDVqFA8//DC33nor48aN49lnn6VNmzbcdttt7nVVqlTJ/dpisbjfWywW7Ha7e17eS8i8XVLm75Puo0aN4rnnnqN9+z+2uc1mw+FwuN9f7N3emzdvpm3bthdMf+utt1i/fj2fffYZMTExJCUleV2+cuXK+a7b2zYsTvwFbWfPf0+r1apdQyog/HofgTHmc2NMK2NMC2PMP1zT3nIlAYwxrxlj2htjoo0xXY0x3jucPVQOdeautfvz7T0qcxYtWsStt97KL7/8woEDBzh48CDNmjVjzZo1/PLLL9SrV4877riD22+/nU2bNgEQHx/PwYMHee+997jpppuK/JmffPIJ6enpnDhxglWrVtG5c2eqVq3KmTNn3G169OjBu+++C8CePXv49ddfad269QXtPF199dW8+eabAGRnZ3P6dP7dbCEhIdx///3uPnFw9pvnfMdNmzbx888/F/m75Thw4AAPPfQQ99xzzwXz9u/fT3x8PE899RR16tTh4MGDBX4vb7xtw8suu4wff/yRjIwMUlNT+eabb9zt81t/jx49WLJkCWlpaZw7d47FixfTvXv34n1ppfwg6O4sDrVZaFQjnHU/pQQ6FJ8tWLCAG264Ide0IUOG8N5777Fq1SpiYmKIjY3lo48+4t5773W3GT58ON26daNmzZpF/swuXbpw3XXX0bVrVx5//HEaNmxIhw4dsNlsREdH89JLL3H33XeTnZ1NVFQUI0aMYN68eVSqVInevXvz448/uk+qenrllVdYuXIlUVFRdOrUqdArg26//fZcRypDhgwhJSWFmJgY3nzzTVq1alWk77V//3735aPDhw/nnnvuyXXElOPhhx8mKiqKyMhIevToQXR0dIHfyxtv27BJkyYMHz6cDh06cPPNNxMbG+tuP378ePr37+8+WZyjY8eOjBkzhi5duhAfH8+4ceNyLadUoEmwXZMfFxdnek2ewzc7j5D42LVYLIXfRbdz506v3Qdl3cCBA7n//vu5+uqri7TctGnTqFKlCg899JCfIlNKlWXefvNEJDHnysu8grLoXNfmtVmUmMyeo2doc0m1P2ZM837ikhHrSiewEnLq1Cm6dOlCdHR0kZOAUkoVVVAmgvhmtQBYu/9E7kRQTtSoUaPQq3cKMm3atJILRilV7gXdOQKAJrUiaFwzPChvLFNKqbImKBMBwBXNawfl/QRKKVXWBG0i6Nq8NqfSsth9xPfLAZVSSl0oaBNBfPM/zhMopZQqvqBNBI1rRtCkVjHPE0yrXrIPH4iIu5wDOKtd1q1bN1cBs6I4deoUb7zxRpGXO3v2LHfeeSctWrSgffv29OjRg/Xr1xdpHb169SIhIaFI7ePi/rhqLSEhgV69ehW6nGfRtpwS1XlZrVZiYmJo37490dHRvPjii+47fz3LSntTWDXWw4cPM3ToUCB3QTtfzZs3j8OH/yivNW7cOH78Me9wHMWzZMkSnnrqKffn1K1bl5iYGGJiYpg9e7a73fz582nZsiUtW7bMVbL7559/Jj4+npYtWzJixAgyMzN9/uzibAt/WLJkSa7t+cQTT/D111+X+Of069ePQ4cOlfh6C+JLGfTXXnvNXXn3YgVtIoDgOk9QuXJltm/f7i4h8NVXX9Gokbeq3L4pbiIYN24ctWrVYu/evezYsYN58+a5f2x9UVDdnoIcPXqUL774oljLFiQ8PJykpCR27NjBV199xeeff86TTz4JUGhZ6YISgd1up2HDhixatKjYseVNBLNnz6Zdu3bFXp+nF154gbvvvtv9fsSIESQlJZGUlMS4ceMASElJ4cknn2T9+vVs2LCBJ598kpMnTwIwefJk7r//fvbu3UvNmjWZM2dOicRVXJ43HfoqbyJ46qmnuOaaa0oyLM6fP09KSspF/V/1l7Fjx15U2XRPQZ0IujavTer5LHb9HhznCfr3789nn30GOO829iwdkZKSwp///Gc6dOhA165d2bp1K+DcMxg7diy9evWiefPm7n/4KVOmsH//fmJiYnj44YcBmD59Op07d6ZDhw5MnTr1gs/fv38/69ev55lnnsFicf7TN2/e3F1V889//jOdOnWiffv2zJo1y71clSpVeOKJJ4iPj2ft2rW51rlgwQL3HbyTJ0/O97s//PDDPPPMMxdMz7t3OXDgwAuqgPqqXr16zJo1i9deew1jTK6S0d9++617jzk2NpYzZ85cUJZ73rx5DBs2jOuvv54+ffpcMEDOwYMH6devH61bt3Ynm7xtZsyYwbRp01i0aBEJCQncfPPNxMTEcP78+VxHUvlttypVqvDoo48SHR1N165dOXLkyAXfc8+ePVSqVOmCKqd5LVu2jGuvvZZatWpRs2ZNrr32Wr788kuMMaxYscJ9tDN69Gh31VpP3sqUF7Qtzp07x3XXXUd0dDSRkZHuu7cTExPp2bMnnTp1om/fvvz222+A80jxkUceoWfPnvzjH/+gadOm7qO5tLQ0mjRpQlZWFm+//TadO3cmOjqaIUOGkJaWxg8//MDSpUt5+OGHiYmJYf/+/YwZM8aduL/55htiY2OJiopi7NixZGRkAM6jzKlTp9KxY0eioqLclWu9/X2As+x4ztHrlClTaNeuHR06dHDfrPm///2P+Ph4YmNjueaaa9z/XtOmTWP06NH06dOHpk2b8vHHH/O3v/2NqKgo+vXrR1ZWljueyZMn06VLF7p06eK1Wu/+/fvp168fnTp1onv37u6YIyIiaNq0abGLIXoK6kQQ37w2EDzjE4wcOZKFCxeSnp7O1q1biY+Pd8+bOnUqsbGxbN26lWeffZZbb73VPW/Xrl0sW7bMvVeXlZXF888/T4sWLUhKSmL69OksX76cvXv3smHDBpKSkkhMTOS7777L9fk7duwgJiYGq9XqNb533nmHxMREEhISmDlzJidOOLfruXPniIyMZP369Vx55ZXu9ocPH2by5MmsWLGCpKQkNm7c6PUHBeCKK66gUqVKuX5M/KF58+Y4HA6OHj2aa/qMGTN4/fXXSUpKYvXq1YSHh/P888/TvXt3kpKS3OM/rF27lvnz57NixYoL1r1hwwbeffddkpKS+PDDDwvsHhs6dChxcXHu9p4VTgvabufOnaNr165s2bKFHj16uMdB8PT999/TsWPHXNM++ugjOnTowNChQzl40Dke1KFDh2jS5I9K8I0bN+bQoUOcOHGCGjVqYLPZck3P6/XXXwecZcoXLFjA6NGj3UX2vG2LL7/8koYNG7Jlyxa2b9/u/sG75557WLRoEYmJiYwdO5ZHH33U/RmnTp3i22+/ZerUqURHR/Ptt98Czh/Yvn37EhISwo033sjGjRvZsmULbdu2Zc6cOfzpT39i0KBBTJ8+naSkJFq0aOFeZ3p6OmPGjOH9999n27Zt2O12d30sgDp16rBp0ybuuusud/eLt78PgC+++IJ+/fqRkpLC4sWL2bFjB1u3buWxxx4D4Morr2TdunVs3ryZkSNH8sILL7g/Z//+/Xz22Wd88skn3HLLLfTu3Ztt27YRHh7u3iEEqFatGhs2bGDixIncd999F/w7jB8/nldffZXExERmzJiR60gwLi7OXcn2YgR1ImhUI5xLa0WwNkgSQYcOHThw4AALFixgwIABueZ5loS+6qqrOHHiBKmpqQBcd9117j3AevXqed1LXL58OcuXLyc2NtZdo3/v3r1Fim/mzJnuPdGDBw+6l7darbmGnsyxceNGevXqRd26dbHZbNx8880XJB9Pjz32mNejgpLmrWxKt27deOCBB5g5cyanTp1y/wjmlbMHnd+82rVrEx4ezo033siaNWuKFV9B2y00NNR9FJNfWerffvuNunXrut9ff/31HDhwgK1bt3LNNdcwevRowPt2EJF8p+dVUJlyb9siKiqKr7/+msmTJ7N69WqqV6/O7t272b59O9deey0xMTE888wzJCcnuz9jxIgRuV7nHEUsXLjQPW/79u10796dqKgo3n333ULrW+3evZtmzZq561iNHj0619/ljTfeCOTevvn9fXz//fdceeWVVKtWjbCwMMaNG8fHH39MREQEAMnJyfTt25eoqCimT5+eK7b+/fsTEhJCVFQU2dnZ9OvXD+CCkvE5PQM33XTTBUfcZ8+e5YcffmDYsGHExMRw5513uo+owHkU7Nn9WFxBnQjAeZ5gQ5CcJwBn/fuHHnrogoqiBf3nzFuq2Ft/qjGGv//97+5+4n379nH77bfnatO+fXu2bNmSq4xyjlWrVvH111+zdu1atmzZQmxsrHvvLywszOtRRFHrVF111VWkp6ezbt0fJT9Kuiz1Tz/9hNVqpV69ermmT5kyhdmzZ3P+/Hm6du2a77jRgS5LHRIS4v6c/P6tw8PDc31O7dq13X8jd9xxB4mJiYBzTz/n6ACcP1oNGzakTp06nDp1yr3unOlFidPbtmjVqhWJiYlERUXx97//naeeegpjDO3bt3f/XW7bto3ly5e7l/Pc3oMGDeKLL74gJSWFxMRErrrqKgDGjBnDa6+9xrZt25g6dWqh27iwv8ucbeW5fb39ffz00080adKE0NBQbDYbGzZsYMiQIe5BpgDuueceJk6cyLZt2/j3v/+dKzbPEvGe/64FlYzPu10dDgc1atRwb7+kpCR27tzpnp+enp7veBpFEfSJoGuLWqSez2Ln7yU76pi/jB07lieeeMI90EwOz5LQq1atok6dOlSrln/5jLwlj/v27cs777zjHjHs0KFDF3SPtGjRgri4OKZOner+z7J3714++eQTUlNTqVmzJhEREezatSvXj3V+4uPj+fbbbzl+/DjZ2dksWLCAnj17FrjMo48+muvwuWnTpiQlJeFwODh48OBF9XceO3aMCRMmMHHixAv+Q+3fv5+oqCgmT55MXFwcu3btKnJZ6q+++oqUlBTOnz/PkiVL6NatG/Xr1+fo0aOcOHGCjIwM9+hvkH9Z6uJsN09t27bN1ZfsuYe4dOlSd7Gxvn37snz5ck6ePMnJkydZvnw5ffv2RUTo3bu3uz99/vz5DB48+ILPya9MeX7b4vDhw0RERHDLLbfw0EMPsWnTJlq3bs2xY8fce7pZWVn57tFXqVKFLl26cO+99zJw4ED3zseZM2do0KABWVlZ7ngK2r5t2rThwIED7m30n//8p9Dt6+3vI6dbCJx75qmpqQwYMICXX37ZPb5Famqq+0Sy51VZRZFzFPT+++9zxRVX5JpXrVo1mjVrxocffgg4k9yWLVvc8/fs2ZPrHFVxBWWtIU/xzXLOE6Tg85Az01L9Fk9hGjdunKvUdI5p06Zx22230aFDByIiIgr9o6pduzbdunUjMjKS/v37M336dHbu3On+Q6pSpQr//e9/L9gznj17Ng8++CCXX345ERER1K5dm+nTp9OhQwfeeustOnToQOvWrenatWuh36VBgwY899xz9O7dG2MMAwYM8PqD4mnAgAG5ujW6devmHnUtMjLygr7vwpw/f56YmBiysrKw2WyMGjWKBx544IJ2L7/8MitXrsRqtdKuXTv69++PxWJxl+UeM2ZMoeW+r7zySkaNGsW+ffv4y1/+4r4kNudEerNmzWjTpo27/ZgxY5gwYQLh4eG5DvmLs9089ejRgwcffBBjDCLCzJkzWbp0KTabjVq1armHFq1VqxaPP/44nTt3dseZ0+31z3/+k5EjR/LYY48RGxt7wdEjwN13382ECROIiorCZrO5y5Tnty2WLVvGww8/7N4DfvPNNwkNDWXRokVMmjSJ1NRU7HY79913X67BijyNGDGCYcOG5bpg4OmnnyY+Pp7LLruMqKgo94//yJEjueOOO5g5c2auq7vCwsKYO3cuw4YNw26307lzZyZMmFDgNvX29zF06FBeffVVwJmMBg8eTHp6OsYYXnrpJcD5/3bYsGE0atSIrl27Fmt8jYyMDOLj43E4HO6RCz29++673HXXXTzzzDNkZWUxcuRIoqOjAWfXlbcLQ4oqKMtQ5z1J13P6SlrWq8rsn71X6tw5Yl1QlqFWKj/33nsv119/fYlfLqmcMjIy6NatW5HulymOpk2bkpCQUOgVYN5s3ryZF198kf/85z8XzCtqGeqg7xqCnPMEJ8g2hY9NoFR58Mgjj5CWlhboMMqtSpUq+T0JXKzjx4/z9NNPl8i6gq9r6PDmC+7m7ZrdjYVZf2Vn6KVEyi8BCkyp0lO/fn0GDRoU6DDURfJ2VZivrr322hKLo1wcEcRbnGfR1znyv2sz2LrAlFKqOIrzW1cuEkEDSaGp/J5vIggLC+PEiROaDJRS5ZoxhhMnThAWFlak5YKvaygf3SzbWZLdjQxjo5Lkvva6cePGJCcnc+zYsQBFp5RSpSMsLIzGjRsXaZlykwh6W5J4N/saEhyt6WbNfZ1ySEgIzZo1C1BkSilVtpWLriGAP1l2EEoWKx0xgQ5FKaWCSrlJBBGSQbxlpyYCpZQqonKTCMDZPbTfNOJXR73CGyullALKYSIAWOWIDmwgSikVRMpVImhm+Z2m8rt2DymlVBGUq0QA0MuSxA+O9qSbkECHopRSQaHcJYLeliQyCGVtAXcZK6WU+kO5SwTxlp2EkcEq7R5SSimflLtEECZZdLPsYKUjBq0ooZRShfNrIhCRfiKyW0T2iciUAtp1FpFsERlaEp/by5LEr6Y+P5kGJbE6pZQq1/yWCETECrwO9AfaATeJyAUd9652/wSWldRn93JdRqpXDymlVOH8eUTQBdhnjPnJGJMJLAS8jcd3D/ARcNTLvGJpYjlOS0nmW72fQCmlCuXPRNAIOOjxPtk1zU1EGgE3AG8VtCIRGS8iCSKScCzNt47/XpYk1jvacs5UKlrUSilVwfgzEXgbNzLvr/jLwGRjTHZBKzLGzDLGxBlj4upG+DYcZW9LEpmE8IPD5yHtlVKqQvJnGepkoInH+8bA4Txt4oCFIgJQBxggInZjzJKL/fA4y24qc56VjhhKbkA3pZQqf/yZCDYCLUWkGXAIGAn8xbOBMcY9SICIzAM+LYkkABAq2Vxp2caq7BiMMbiSjVJKqTz81jVkjLEDE3FeDbQT+MAYs0NEJojIBH99rqfeliQOU4c9R86WxscppVRQ8usIZcaYz4HP80zzemLYGDOmpD+/l3UL2GHl7qO0vqRqSa9eKaXKhXJ3Z7GnS+QkbeUAK3eV2JWpSilV7pTrRADQ27KFhF9Ocjo9K9ChKKVUmVT+E4E1iWyHYc3e44EORSmlyqRynwhiZS/Vw0P4eueRQIeilFJlkl9PFpcFNnFwdcY3fL2pE5k7riJU8ty7Ni01MIEppVQZUe6PCACus67nNJX53hEV6FCUUqrMKTQRiMgMEQnqOg1XWrZRlXN85ogPdChKKVXm+HJEsAuYJSLrXTeDVfd3UCWtkti51pLI8uw4Mo010OEopVSZUmgiMMbMNsZ0A24FmgJbReQ9Eent7+BKknYPKaWUdz6dI3ANHtPG9TgObAEeEJGFfoytRDm7h9K0e0gppfLw5RzBizi7hwYAzxpjOhlj/mmMuR6I9XeAJUW7h5RSyjtfjgi2A9HGmDuNMRvyzOvih5j85jrrOlf3UGSgQ1FKqTLDl0RwszEmzXOCiHwDYIwJqovw/+ge6hroUJRSqszINxGISJiI1ALqiEhNEanlejQFGpZahCVIu4eUUupCBR0R3Akk4jxBvMn1OhH4BHjd/6H5xwD31UPaPaSUUlBAIjDGvOIaQewhY0wzj0e0Mea1UoyxRHW3bKUqaXyuVw8ppRRQcNfQVa6Xh0TkxryPUoqvxFUSO9do95BSSrkVVHSuJ7ACuN7LPAN87JeISsF11vUsdnTne0ckQXVXnFJK+UG+icAYM9X1fFvphVM6PLuHNBEopSo6X24ou1dEqonTbBHZJCJ9SiM4f/HsHsrKdgQ6HKWUCihf7iMYa4w5DfQB6gG3Ac/7NapSMMC6nlSq8P0+HblMKVWx+ZIIxPU8AJhrjNniMS1odbdsowppfL7tt0CHopRSAeVLIkgUkeU4E8EyEakKBH1/SphkcY1lE8t2HNHuIaVUheZLIrgdmAJ0dpWaCMXZPRT0rrOuI/V8lnYPKaUqNF/GI3AAR4B2ItIDaA/U8HNcpaK7ZRvVwmws2Xwo0KEopVTAFDp4vYj8ExgB/AjkjPxugO/8GFepCJMsBsc04oOEgzx5Povq4SGBDkkppUqdL11DfwZaG2MGGGOudz0G+TmuUjM8rgkZdgf/23I40KEopVRA+JIIfgLK7a5yZKNqtLmkKh8mHAx0KEopFRCFdg0BaUCSawyCjJyJxphJfouqFIkIw+Oa8NSnP7Lr99O0uaRaoENSSqlS5csRwVLgaeAH/ihFnejPoErbn2MbEWIVPkxIDnQoSilV6go9IjDGzBeRcOBSY8zuUoip1NWqHMq17eqzePMhJvdrQ6jNl/yolFLlgy+1hq4HkoAvXe9jRGSpn+MqdcPimpByLpMVu44EOhSllCpVvuz6TsM5SP0pAGNMEtDMl5WLSD8R2S0i+0Rkipf5g0Vkq4gkiUiCiFzpc+QlZVp1mFadHu+14hJO8MF7s93TlFKqIvAlEdi9DFJvCltIRKw4h7TsD7QDbhKRdnmafQNEG2NigLHAbB/i8QurGIZYV7PKEcMRUyNQYSilVKnzJRFsF5G/AFYRaSkir+I8cVyYLsA+Y8xPxphMYCEw2LOBMeasMSYnqVTGhwTjT8Os3+LAwkfZ3QMZhlJKlSpfEsE9OMtKZAALgNPAfT4s1wjwvDg/2TUtFxG5QUR2AZ/hPCoImKaWI3SRnXyY3QsT0JSklFKlx5daQ2nGmEeNMZ2NMXGu1+k+rNtbqeoLfl6NMYuNMW1w3sH8tNcViYx3nUNIOJbm31/o4bZV/GwakGBa+/VzlFKqrCgwEYjIaNeIZOdcjwQRudXHdScDTTzeNwbyreNgjPkOaCEidbzMm+VKQnF1I/w7FMIAywYqc54Psnv69XOUUqqsyDcRuH7w7wMeBBri7Nb5G3Cvj8lgI9BSRJqJSCgwEufNaZ6fcbmIiOt1R5wlrk8U43uUmAjJ4HrrWj7L7srZDHsgQ1FKqVJR0BHB3cANxpiVxphUY8wpY8wKYIhrXoGMMXZgIrAM2Al8YIzZISITRGSCq9kQnCejk3BeYTTC4+RxwAyzfksaYXy+VUcvU0qVf5Lf766I/GiMyXu5Z6Hz/C2uodUkjK/i188wBq7JnE6NSyP56K4/+fWzlFKqNIhIojEmztu8go4IzhdzXtATgeHWb0n85ST7jp4JdDhKKeVXBSWCtq67fvM+tgFtSivAQLnR+h2hVgvzf/gl0KEopZRfFVR0rm2pRVEG1ZXTDIppyKLEZB7s04oaEaGBDkkppfwi30RgjKnwu8Jjt93CoqznWfDsWO6y/e+PGdPyVtxQSqngpfWWC9DO8it/smxnvr0PWcYa6HCUUsovNBEU4nbrF/xObT53dAl0KEop5Re+jEcwUEQqbMLobUmiuRzmHfsArT+klCqXfPmBHwnsFZEXRKTCnUC2iOE265dsMS1INK0CHY5SSpU4X4rO3QLEAvuBuSKy1lUErqrfoysjhlhXU52zzLH3D3QoSilV4godsxjAGHNaRD4CwnHWH7oBeFhEZhpjXvVjfGVChGRwk3UFs7IHctBRlyYFjV6mVxQppYKML+cIBonIYmAFEAJ0Mcb0B6KBh/wcX5lxq205gmFedt9Ah6KUUiXKl3MEQ4GXjDEdjDHTjTFHwTlOAQEeSKY0NZQUBljW8352L86Y8ECHo5RSJcaXRPCba6wANxH5J4Ax5hu/RFVG3W77grNE8KGOVaCUKkd8SQTXeplWIc+axlj200l2Mze7H9nGvwPkKKVUaSloYJq7cgrM5Sk69zOwtfRCLFtut33BQVOPrxydAh2KUkqViIKuGnoP+AJ4DpjiMf2MMSbFr1GVYX0sCTTiGO/Y+9PPmhDocJRS6qIV1DVkjDEHgL8CZzweiEgt/4dWNtnEwW22ZWwwbUl0tAx0OEopddEKSgTvuZ4TgQTXc6LH+wrrJus31CaVl+xDAx2KUkpdtHwTgTFmoOu5mTGmues559G89EIseypLBhNs/2ONI4r1jnI/Ro9Sqpwr6GRxx4IepRlkWXSL9WvqcpKX7EMCHYpSSl2Ugk4W/6uAeQa4qoRjCSrhksndtqU8aR/ND9nt+JP1R+eM/MpPaOkJpVQZVdAIZb1LM5BgdJN1BW/Zr+cl+1CusDyF6K0FSqkglG8iEJGrjDErRORGb/ONMR/7L6zgECZZTLQt4XH7WNY4Iulu3R7okJRSqsgKumoop47C9V4eA/0cV9AYbl1FQ47zon2YDlyjlApKBXUNTXU931Z64QSfSmJnom0Jj9jHscoRTW/rlkCHpJRSReJLGeraIjJTRDaJSKKIvCIitUsjuGAx1PotjeUoL9uH6lGBUiro+FJ0biFwDBiCsyT1MeB9fwYVbEIlm0nWxWwxLVjhiA10OEopVSS+JIJaxpinjTE/ux7PADX8HFfQucG6hsvkd17UowKlVJDxJRGsFJGRImJxPYYDn/k7sGATItlMsi1mh2nGckdcoMNRSimfFXRn8RkROQ3cibPuUKbrsRC4v3TCCy6DLd/TXA7zkn2ojleglAoaBdUaqmqMqeZ6thhjbK6HxRhTrTSDDBY2cfCg7UN2mUtZmK334ymlgkNBJSbcRKQm0BIIy5mWd/hK5TTAsp54+ZEZ9hEMtK6nupwLdEhKKVUgXy4fHQd8BywDnnQ9T/NvWMFLBKaFzCeVylqQTikVFHw5WXwv0Bn4xVV/KBbnJaSFEpF+IrJbRPaJyBQv82/2GALzBxGJLlL0ZVRby0Futn7Nf7KvZZejSaDDUUqpAvmSCNKNMekAIlLJGLMLaF3YQiJiBV7HOdB9O+AmEWmXp9nPQE9jTAfgaWBWUYIvyx6wLaIqaTxpv1UvJ1VKlWm+JIJkEakBLAG+EpFPgMM+LNcF2GeM+ckYk3O10WDPBsaYH4wxJ11v1wGNfQ28rKspZ3nQ9iFrHe350tE50OEopVS+Ck0ExpgbjDGnjDHTgMeBOcCffVh3I+Cgx/tk17T83A584W2GiIwXkQQRSTiWFjy71zdZV9BGfuGZrFs4n5kd6HCUUsorX44IckYrmwR0AJJde/iFLuZlmtdfcRHpjTMRTPY23xgzyxgTZ4yJqxsRPNfn28TBtJD5HKIu//5uf6DDUUopr3y5augJYD5QG6gDzBWRx3xYdzLgeaa0MV66lESkAzAbGGyMOeFL0MGkq2UX11nW8uaq/SSfTAt0OEopdQExhZzJFJGdQKzHCeNwYJMxpm0hy9mAPcDVwCFgI/AXY8wOjzaXAiuAW40xP/gScFxDq0kYX8WXpmXGIVObqzNmcLVlM6+HzrywgQ5jqZTyMxFJNMZ4rX/jS9fQATxuJAMqAYX2cxhj7MBEnPcd7AQ+MMbsEJEJIjLB1ewJnEcab4hIkogk+BBP0GkkJ7jbtpTPHF35ITvvhVNKKRVY+R4RiMirOPv0L8V5H8FXrvfXAmuMMSNLK0hPwXhEAJBuQrgmczphZPJp6KOESdYfM/WIQCnlZwUdERRUYiJn7zwRWOwxfVUJxVWhhEkW/7C9w+isKbxsH8KUkIWBDkkppYCCh6qcn/NaREKBVq63u40xWd6XUgXpad3KTY4VzMoeSB9rAh0t+wIdklJK+XTVUC9gL867hN8A9ohID/+GVX49YnuXBpzgoawJpJuQQIejlFI+nSz+F9DHGNPTGNMD6Au85N+wyq+qcp4XQmbxk2nIDPvwQIejlFI+JYIQY8zunDfGmD2A7spehG7WHYyyLmdOdn82OAot26SUUn7lSyJIFJE5ItLL9Xgb5wlkdRGm2BbQWI7zcNadpGXaAx2OUqoC8yURTAB2AJNwlqT+0TVNXYTKksH0kLf4xVzCC1/uLnwBpZTykwJHKBMRC5BojIkEXiydkCqOrpZd3Gb9grk/9KdP+/r8qUWdQIeklKqACjwiMMY4gC2uUhDKD/5me5+mtSP426KtnM3QLiKlVOnzpWuoAbBDRL4RkaU5D38HVlGESyYzhkVz6NR5nly6o/AFlFKqhPkyeP2Tfo+igotrWouJvS/n1RX7iGtakxGd9QBMKVV68k0EIhKG86Tw5cA2YI6rkJzyg/uuaUXSwVM8/skO2jesTmSj6oEOSSlVQRTUNTQfiMOZBPrjvLFM+YnVIrwyMpY6lUOZ8N9ETqX5MvaPUkpdvIKqj24zxkS5XtuADcaYjqUZnDfBWn3UV5sdLRieOZXulq3MfuYxLJbgGZFNKVV2Fbf6qLuwnDHGLqI/SKUh1rKfJ2z/x+P2sbz+xBjusS3x3lBLVyulSkhBiSBaRE67XgsQ7novgDHGVPN7dBXULdav2eRoyYv2ocTIPrpbtwc6JKVUOZbvOQJjjNUYU831qGqMsXm81iTgRyLwj5B3aCXJTMq6h0OmdqBDUkqVY77cR6ACIEIyeCvkZexYuTvzXjKML1f6KqVU0WkiKMOaWX5neshbbDGX8/esO8jnvL5SSl0UTQRlXD9rAg/aPuBjR3em20cEOhylVDmk/Q1BYKJ1Cb+ZWryRPZj6cpLRtuWBDkkpVY5oIggCIvC0bS7HTA2m2W+lvpykX6CDUkqVG9o1FCSsYpgZ8hqxso9JWX9lw88pgQ5JKVVOaCIIIuGSyZzQGTSW44ybv5G9R84EOiSlVDmgiSDI1JSzzA95nkohVka/s4HfUs8HOiSlVJDTRBCEmliOM++2zpxOtzPmnY2kpmUVvpBSSuVDE0GQat+wOv8e1Ymfj5/jprfXkXJOq5UqpYpHE0EQ63Z5HWbd2on9x87yl7fXcfxsRqBDUkoFIU0EQa5X63q8M6YzB06cY+SsdRw9nR7okJRSQUYTQTnQ7fI6zL+tC7+dOs+IWev0BLJSqkg0EZQT8c1r83+3x3P8TAbD/72WgylpgQ5JKRUkNBGUI50uq8l/x8WTmpbFyFnr+OXEuUCHpJQKAn5NBCLST0R2i8g+EZniZX4bEVkrIhki8pA/Y6koopvUYMH4rqRl2hn+77XsOKwjmSmlCua3RCAiVuB1nAPftwNuEpF2eZqlAJOAGf6KoyJqP+tSFtrvx3L6MMNmfs1Xj/eGadUDHZZSqozy5xFBF2CfMeYnY0wmsBAY7NnAGHPUGLMRj/GRVclobUnmk0qP01IOMT7rAd62D8DogAZKKS/8mQgaAQc93ie7phWZiIwXkQQRSTiWpj9mvqonp1gY+jT9LRv5h/0WHlm8jaxsR6DDUkqVMf4sQy1ephXrV9wYMwuYBRDX0KqZAHzu6gmXTF4LmclL9qG8uuEGfkn4gjdCXqGG5DmRPE3PJShVUfkzESQDTTzeNwYO+/HzVD4sYngw5EOaWX5jStYd3Jj5JHNCZtDM8vsfjfJLLJoglCr3/Nk1tBFoKSLNRCQUGAks9ePnqULcaF3Df0Of5aSpyuDMp/kyu3OgQ1JKlQF+SwTGGDswEVgG7AQ+MMbsEJEJIjIBQEQuEZFk4AHgMRFJFpFq/opJQRfLbpaGPkYz+Z0JWffzWNZtpJuQQIellAogCbYrSeIaWk3C+CqBDiPoZRorM+wjmJU9kDbyK6+FzORyi5eeO+0aUqpcEJFEY0yct3l6Z3EFFSrZPBLyHnND/slRU4PrM5/hA3tPgmy/QClVAjQRVHC9rVv4otIUYi37+Jv9Tu7N+itnTHigw1JKlSJNBIr6cor/hDzHQ7b3+czRlX4Zz7MyOzrQYSmlSokmAgWAVQwTbZ/wQehThEkmt2VN5t7Mv+pgN0pVAJoIVC6dLHv5PPTv3Gv9iM8d8Vzz4rcsSkzW8hRKlWOaCNQFKomd+0M+4rPQv9OibhUe+nALo+Zs0LLWSpVTmghUvlpZDvHhnVfw9OD2JB08Rd+Xv+PVb/ZyPjM70KEppUqQJgJVIItFGHVFU756oAc9W9XlX1/tofeMVSxKTCbbod1FSpUHekOZKpL1jjY8m3UzW0wL2jWoxiMD2nJlyzqBDkspVQi9oUyVmHjLLhaHPsErIa+Sej6LW+asZ8zcDez+/UygQ1NKFZMmAlVkFjEMtq7lmwd78siANiT+cpL+r3zHfQs3s/eIJgSlgo0/y1Crci4sxMr4Hi0Y1qkJb6zax3/X/cqSpMP0a38JE6+6nMhGOjymUsFAzxGo4stTkC7lXCbvrPmZ+T8c4EyGnV6t6zKx9+XENa0VoACVUjkKOkegiUAVXz6VSU+nZ/Gftb8wZ83PpJzLJL5ZLcZe2Yyr29TDZtXeSKUCQROB8o9CSlSnZdpZsOEgs1f/xG+p6TSsHsbNXS9jROcm1KlSqZSCVEqBJgLlLz6OVWDPdvD1zqP8Z90Bvt93glCrhQFRlzDqiqZ0vLQGIt6Gt1ZKlaSCEoGeLFZ+Z7Na6Bd5Cf0iL2Hf0bP8d90vfJSYzJKkw7RrUI0hnRozKLohdavqUYJSgaBHBKr4LmL0snMZdhZvPsTCjb+y/dBprBahO5u5wbqaPpZEwiWzRD5HKeWkRwSqzKlcycYtXS/jlq6XsffIGRZvPsSSVce413EPVUijn3Ujf7Z8T7xlJzqislL+pUcEqvhKeE/dMbUG6x1tWOy4ki+y4zlDBNU5y9WxrenT/hJ6tKpDRKjuuyhVHHpEoIKCRQxXWHdyhXUnT9nm8a0jmmXZcXyzqyYfbz5EWIiF7i3r0qddfa5uW59alUMDHbJS5YIeEaiSV9wjhWne70TOevwkG39OYdmO31n+4xF+S01HBKIaVad7yzp0b1mXjpfWJNSm9ygolR+9fFSVrhJOBJ7rM8aw7VAqq3YfY/XeY2z69RTZDkNEqJUrmteme8s6XNGiDi3rVcFi0ctSlcqhXUOq3BAROjSuQYfGNZh0dUtOp2exbv8JVu89zuq9x/hm11EAakSEEHdZTTo3rUXnZrWIbFhdjxiUyocmAlXy8tuzhxI/wVzt+Tr0Afq43h8Mrcs6R1sS2v2DjQdS+HqnMzGEhViIaVKD2Etr0qFRdTo0qUHD6mF6M5tSaCJQ5UwTyzGaWI4xbGgHAI6dySDhQAobDqSQcOAkb3/3E3bXyGp1qoS6ji6q06Fxddo1qE79apU0OagKRxOBKtfqVq1E/6gG9I9qAEB6VjY7fzvN1uRUtiSfYmtyKit3HyXnVFmNiBDaXFKVNpdUo20D53Or+lUJD7UG8Fso5V+aCFT5lE/3VBgQOy2V2EtruqedSc/ix8On2fX7GXb9fpqdv53h/Y0HOZ+V7W7TqEY4LepVoUXdyrSoW8X5qFeZulX0CEIFP00Equwr6JxDCagaFkJ889rEN6/tnuZwGH5NSWPX72fY/fsZfloxl/2pDdm4pwHnCXO3q0Ialza4hMtqR3BprQgudT1fVqsyDWuEadltFRQ0ESjlhcUiNK1TmaZ1KtMv8hJY8zoADiP8Tk32Oxqy3zTkZ9OAX6vdwu4jZ/hm51Eysx1/rEOgfrUwGtYIdz3CaOx6Xb9aGPWrhVG7cqhe5qoCThOBUkVgEUNDUmhoTaE7250Tb5sJOI8ifj+dzi8n0vg15RyHTp7n0Kl0Dp86z9bkUyzbnp4rUQDYLEK9qpWoVy2M+tUqUX/Xf6gjqdQhlTqSSm05TV3X64gnj5b211UVhN5QplSOgi5tLU73VJ71ORyG4+cyOHTyPEfeHsYRU9P1qMFRarrfp+L97zs8xEqtyqHUrBxCzYhQakaEOt9HOKdVDw+hWrjzuXp4CDVc70O0e0qhN5QpVSZYLEK9qmHUqxoG1o35tss0Vk5QneMm51GN41TnePwjnEzL5OS5TE6mZfFrShop5zI5k24v8HMjQq1UDbNRNSwk13O1MBtVKtmoXMn5HBFqo3Ilq3ta5VAb4aFWItwPm96UV075NRGISD/gFcAKzDbGPJ9nvrjmDwDSgDHGmE3+jEmpUlPMk9yhkk0DUmggKblnDPyv1/ZZ2Q5OpWWRet75OH3+j9c5j7Ppds5kZHEm3U7q+SyST6ZxJt3O2XR7rqujCmOzCOGhVsJDrO7nsBArYSEW97RKNuf7SjYrlWwWKoW4nnNeWy1UCrEQ6n62EmqzEGqzEGIVQq0WQqw5753tbFYhxOqcr1dplTy/JQIRsQKvA9cCycBGEVlqjPnRo1l/oKXrEQ+86XpWqvT5+eqki5ZPfCFAXdejaOtzdl1lOwznMu2kZWRzNsPOude6c86EcY4w0qjEeVOJNNfrtCsfIS0zm/SsbM5n5Tw7SN/7HScI5TyVyCCEdBPqfCaUDEq2SqwNOyHYsZFNCNmuZzu22pdhswg2izNx2KwWbBbBapE8z67pVsEqf8zzfFgkzzQRLDnP4jy6c7YDi/zR3vL5g1hwYMFgxYFgsIrzvQx5293WIs5yKTnLSwHPQu724noteEzD2RY8lvOYXtj1CP48IugC7DPG/IQz4IXAYMAzEQwG/s84T1SsE5EaItLAGPObH+NSSnmwWoRqYSFUC3MNAWT5Kf/G/f7P+/Rp1+a7iJl6igy7gwy7g0y7gwx7Npl2B5mvdiWDEDIJIdPYyMRGJiFkYXU+u6fZsGMjCytZxkYWNuxYXdOt2LGSZazYm0Rjdxjs2Q7s2Qa7w5DtMNgdDrKyHZzPcr7PyjY4XNOzHYZsY8j2aJ9tnM8Oj9fZDoPDp9OpY/OftTDJlxUEhD8TQSPgoMf7ZC7c2/fWphGQKxGIyHhgvOtthjx5envJhlrq6gDHAx3ERQj2+CH4v8PFx/9kMbpYirVMvucVSvjfYHrJrco3wfY3dFl+M/yZCLz9xeTNqb60wRgzC5gFICIJ+Z35DhbB/h2CPX4I/u8Q7PFD8H+HYI/fkz8vAUgGmni8bwwcLkYbpZRSfuTPRLARaCkizUQkFBgJLM3TZilwqzh1BVL1/IBSSpUuv3UNGWPsIjIRWIbz8tF3jDE7RGSCa/5bwOc4Lx3dh/Py0dt8WPUsP4VcmoL9OwR7/BD83yHY44fg/w7BHr9b0N1ZrJRSqmTpbYJKKVXBaSJQSqkKLqgSgYj0E5HdIrJPRKYEOp6iEJEmIrJSRHaKyA4RuTfQMRWHiFhFZLOIfBroWIrDddPiIhHZ5fq3uCLQMRWViNzv+hvaLiILRCSs8KUCR0TeEZGjIrLdY1otEflKRPa6nmsWtI5Ay+c7THf9HW0VkcUiUiOAIV6UoEkEHiUr+gPtgJtEpF1goyoSO/CgMaYt0BX4a5DFn+NeYGegg7gIrwBfGmPaANEE2XcRkUbAJCDOGBOJ80KMkYGNqlDzgH55pk0BvjHGtAS+cb0vy+Zx4Xf4Cog0xnQA9gB/L+2gSkrQJAI8SlYYYzKBnJIVQcEY81tOQT1jzBmcP0CNAhtV0YhIY+A6YHagYykOEakG9ADmABhjMo0xpwIaVPHYgHARsQERlPF7b4wx3wF5KugxGJjvej0f+HNpxlRU3r6DMWa5MSan9Os6nPdBBaVgSgT5laMIOiLSFIgF1gc4lKJ6Gfgb4CikXVnVHDgGzHV1b80WkcqBDqoojDGHgBnArzhLsaQaY5YHNqpiqZ9zz5DruV6A47lYY4EvAh1EcQVTIvCpHEVZJyJVgI+A+4wxpwMdj69EZCBw1BiTGOhYLoIN6Ai8aYyJBc5R9rskcnH1pQ8GmgENgcoicktgo6rYRORRnF2/7wY6luIKpkQQ9OUoRCQEZxJ41xjzcaDjKaJuwCAROYCzW+4qEfFeIL/sSgaSjTE5R2KLcCaGYHIN8LMx5pgxJgv4GPhTgGMqjiMi0gDA9RyU43CKyGhgIHCzCeKbsoIpEfhSsqLMcg3CMwfYaYx5MdDxFJUx5u/GmMbGmKY4t/0KY0xQ7YkaY34HDopIa9ekq8ldFj0Y/Ap0FZEI19/U1QTZCW+XpcBo1+vRwCcBjKVYXANvTQYGGWPSAh3PxQiaROA6KZNTsmIn8IExZkdgoyqSbsAonHvSSa7HgEAHVQHdA7wrIluBGODZwIZTNK6jmUXAJmAbzv/DZbrUgYgsANYCrUUkWURuB54HrhWRvTgHr3q+oHUEWj7f4TWgKvCV6//zWwEN8iJoiQmllKrgguaIQCmllH9oIlBKqQpOE4FSSlVwmgiUUqqC00SglFIVnCYCFRREZJWI9M0z7T4ReaOQZfw6uLir+udWEbnfh7YxvlwynLediAwqqNpuUdsrlZcmAhUsFnBhlc2RrukBISKXAH8yxnQwxrzkwyIxOIdmLVI7Y8xSY0xB19kXtb1SuWgiUMFiETBQRCqBu3BfQ2CNiLwpIgmuGv1PeltYRM56vB4qIvNcr+uKyEcistH16OZl2TARmSsi21zF6nq7Zi0H6rluJuqeZ5lhrvECtojId6674Z8CRrjajxCRLiLyg2udP4hI63zajRGR14qwXs/29V218re4HsFYjkL5md8Gr1eqJBljTojIBpw14T/BeTTwvjHGiMijxpgU15gV34hIB2PMVh9X/QrwkjFmjYhcivPO9bZ52vzVFUOUiLQBlotIK2AQ8KkxJsbLep8A+hpjDolIDWNMpog8gXMcgYnwR1lsY4xdRK4BnjXGDPHSbkwR1+vZfibwrTHmBtf2qeLjdlEViCYCFUxyuodyEsFY1/ThIjIe599zA5wDF/maCK4B2jnL9gBQTUSqusaMyHEl8CqAMWaXiPwCtAIKqh77PTBPRD7AWRjOm+rAfBFpibOSbogP8fqyXk9XAbe6Ys8GUn1YRlUwmghUMFkCvCgiHYFwY8wmEWkGPAR0NsacdHX5eBu60bOWiud8C3CFMeZ8AZ/rrQR6gYwxE0QkHudAPkkiEuOl2dPAStfeelNgVQmtV6ki0XMEKmgYY87i/LF8hz9OElfDOa5AqojUxzmUqTdHRKStiFiAGzymL8dZzBBwXoHjZdnvgJtd81sBlwK7C4pVRFoYY9YbY54AjuMsoX4GZ5GyHNWBQ67XYzym521X1PV6+ga4y7Ws1dUdpVQumghUsFmAc6zhhQDGmC3AZmAHzgTxfT7LTQE+BVbgHNkrxyQgznUJ6I/ABC/LvgFYRWQb8D4wxhiTUUic010nl7fjTCRbgJU4u6GSRGQE8ALwnIh8j3Ps4Rx52xV1vZ7uBXq7Yk8E2hcSt6qAtPqoUkpVcHpEoJRSFZwmAqWUquA0ESilVAWniUAppSo4TQRKKVXBaSJQSqkKThOBUkpVcP8PUob1Gf3FeJgAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "np.random.seed(0)\n", + "n_observations = 500 # number of observations\n", + "n_samples = 1000 # number of samples\n", + "# standard normal distribution can be used, since the\n", + "# statistic is unaffected by location and scale\n", + "norm_dist = stats.norm()\n", + "# Draw 1000 samples, each with 500 observations\n", + "y = norm_dist.rvs(size=(n_observations, n_samples))\n", + "\n", + "# calculate the value of the statistic for each sample\n", + "# we'll call this the \"Monte Carlo null distribution\"\n", + "null_dist_mc = statistic(y)\n", + "\n", + "# the asymptotic null distribution is chi-squared with df=2\n", + "null_dist = stats.chi2(df=2)\n", + "y_grid = np.linspace(0, null_dist.isf(0.001))\n", + "pdf = null_dist.pdf(y_grid)\n", + "\n", + "# compare the two\n", + "import matplotlib.pyplot as plt\n", + "plt.plot(y_grid, pdf)\n", + "plt.hist(null_dist_mc, density=True, bins=100)\n", + "plt.xlim(0, np.max(y_grid))\n", + "plt.xlabel(\"Value of statistic\")\n", + "plt.ylabel(\"Probability Density\")\n", + "plt.legend(['Asymptotic Null Distribution', 'Monte Carlo Null Distribution (500 observations/sample)'])\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "87d87faf-f1e1-462f-a94c-cf09d9445767", + "metadata": {}, + "source": [ + "As we can see, the *Monte Carlo null distribution* [[2]](#cite_note-2) of the test statistic when samples are drawn according to the null hypothesis (from a normal distribution) appears to follow the *asymptotic null distribution* (chi-squared with two degrees of freedom). \n", + "\n", + "[[2]](#cite_note-2) Named after the Monte Carlo Casino in Monaco, apparently [[3]](https://en.wikipedia.org/wiki/Monte_Carlo_method#Historyhttps://en.wikipedia.org/wiki/Monte_Carlo_method#History).\n" + ] + }, + { + "cell_type": "markdown", + "id": "4195c3fd-12fe-46b4-b681-83e0f57550ae", + "metadata": {}, + "source": [ + "Note that the originally observed value of the statistic, 6.98, is located in the right tail of the null distribution. Random samples from a normal distribution usually produce statistic values less than 6.98, and only occasionally produce higher values. Therefore, there is rather low probability of observing such an extreme value of the statistic under the null hypothesis that the sample is drawn from a normal population. This probability is quantified by the *inverse survival function* of the null distribution:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "0f2fad67-7dc3-4bf4-af5f-12bad97321e2", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Under the null hypothesis, the chance of drawing a sample that produces a statistic value greater than \n", + "6.982848237344646\n", + "is\n", + "0.03045746622458189\n" + ] + } + ], + "source": [ + "pvalue = null_dist.sf(stat1)\n", + "message = (\"Under the null hypothesis, the chance of drawing a sample \"\n", + " f\"that produces a statistic value greater than \\n{stat1}\\n\"\n", + " f\"is\\n{pvalue}\")\n", + "print(message)" + ] + }, + { + "cell_type": "markdown", + "id": "54c6b2aa-98ce-4463-be49-78732a50083d", + "metadata": {}, + "source": [ + "This is the `pvalue` returned by `stats.jarque_bera`." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "bead807c-e722-4269-bfbe-3be8fccb2da8", + "metadata": {}, + "outputs": [], + "source": [ + "np.testing.assert_allclose(pvalue, stats.jarque_bera(x).pvalue, rtol=1e-14)" + ] + }, + { + "cell_type": "markdown", + "id": "78720622-ff97-4248-85a1-6e13b786ea06", + "metadata": {}, + "source": [ + "When the $p$-value is small, we take this as evidence against the null hypothesis, since samples drawn under the null hypothesis have a low probability of producing such an extreme value of the statistic. For better or for worse, a common \"confidence level\" used for statistical tests is 0.99, meaning that the threshold for rejection of the null hypothesis is $p \\leq 0.01$. If we adopt this criterion, then the Jarque-Bera test was inconclusive; it gives no evidence that the null hypothesis is false. Although this should *not* be taken as evidence that the null hypothesis is *true*, the lack of evidence against the hypothesis of normality is often considered sufficient to proceed with tests that assume the data is drawn from a normal population." + ] + }, + { + "cell_type": "markdown", + "id": "8feb92d9-99b7-4863-885c-8876a64905f6", + "metadata": {}, + "source": [ + "There are a few shortcomings with the test procedure outlined above. The Monte Carlo null distribution agreed with the asymptotic null distribution when the number of observations per sample was 500, but our original sample `x` had only 11 observations. If we generate a Monte Carlo null distribution of the statistic for sample size of only 11 observations, there is marked disagreement between the Monte Carlo null distribution and the asymptotic null distribution." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "1fe77f87-b4cc-4b0f-baee-cf79ca2088ea", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABAX0lEQVR4nO3dd3wUdf748dc7m0AIJbSAAlJVENJAmqJURZooFiyIoiKH/Syc3Hkqlp/nCWc79TxEgdMDCwrytStFwEOkSAeVpjQFQhdC2vv3x0zWTdhsNmWz2eT9fDzmsTuzU96z2ex7Zz4z74+oKsYYYyqvqHAHYIwxJrwsERhjTCVnicAYYyo5SwTGGFPJWSIwxphKLjrcARRV/fr1tXnz5uEOwxhjIsry5cv3qWqCv9ciLhE0b96cZcuWhTsMY4yJKCLyU0Gv2akhY4yp5CwRGGNMJWeJwBhjKrmIayMoTZmZmezYsYP09PRwh2KMMaUiNjaWJk2aEBMTE/QylToR7Nixg5o1a9K8eXNEJNzhGGNMiagqaWlp7NixgxYtWgS9XKU+NZSenk69evUsCRhjKgQRoV69ekU+y1FoIhCRZSJyu4jUKXZ05ZglAWNMRVKc77RgjgiuBhoBS0XkLRG5SOzb0xhjKoxCE4GqblLVB4EzgWnA68DPIvKoiNQNdYCVwcyZMxERNm7cGJbtb9u2jWnTphU638qVK/n444+947Nnz+app54KejvNmzfn8ssv947PmDGDESNGFLpcjRo1vHEmJiae9Pq2bduoVq0a7du356yzzqJz585MnTo16Djz71d+y5Yt46677gJg3LhxTJgwodCYfT333HMcO3bMOz5gwAAOHjxYpHUYE0pBtRGISDLwD2A88B5wBXAYmBu60EJgXPzvQzkyffp0zjvvPN56662wbL+4iWDw4MGMHTu2SNtatmwZ69atK3KMhWnVqhXfffcdGzZs4K233uLZZ59l8uTJQcUZKBFkZWXRsWNHXnjhhWLHlj8RfPzxx9SuXbvY6zOmtAXTRrAceBZYCiSr6l2qukRV/wFsCXWAFd3Ro0f5+uuvee211/Ikgt27d9O9e3dSU1NJTExk4cKFvPbaa9xzzz3eeV599VXuvfdetm3bRps2bRg5ciSJiYkMGzaML7/8km7dunHGGWfw7bffAs6v2eHDh9O7d2/OOOMMXn31VQDGjh3LwoULSU1N5dlnnyU9PZ0bb7yRpKQk2rdvz7x588jIyODhhx/m7bffJjU1lbfffpspU6Zwxx13APDrr78yZMgQUlJSSElJ4X//+5/f/b3//vt58sknT5qe/5d2YmIi27ZtK9Z72rJlS5555hnvl7dvnO+++y6JiYmkpKTQvXt3v/s1btw4Ro0aRd++fbn++uuZP38+gwYN8q5/1apVJ72H+ee54447mDJlCi+88AK7du2iV69e9OrVC3COjPbt2wfAM888Q2JiIomJiTz33HOAk5jPOussbrnlFtq1a0ffvn05fvx4sd4LY4IRzOWjV6qq3y98Vb2slOMJm0f/bx3rdx0u1XW2bVSLRy5uF3CeWbNm0a9fP84880zq1q3LihUr6NChA9OmTeOiiy7iwQcfJDs7m2PHjtGhQweSk5N5+umniYmJYfLkyfz73/8GYNOmTbz77rtMnDiRTp06MW3aNBYtWsTs2bN58sknmTVrFgCrV6/mm2++4bfffqN9+/YMHDiQp556igkTJvDhhx8C8I9//AOANWvWsHHjRvr27csPP/zAY489xrJly3jxxRcB5ws211133UWPHj2YOXMm2dnZHD161O/+Dh06lJdffplNmzaV5K0tVIcOHfyeanvsscf47LPPaNy4MQcPHqRKlSon7de4ceNYvnw5ixYtolq1asyfPz/POvy9hwW56667eOaZZ5g3bx7169fP89ry5cuZPHkyS5YsQVXp0qULPXr0oE6dOvz4449Mnz6dV199laFDh/Lee+9x3XXXlfyNMcaPYE4NjRSR2rkjIlJHRJ4IXUiVy/Tp07n66qsBuPrqq5k+fToAnTp1YvLkyYwbN441a9ZQs2ZNqlevTu/evfnwww/ZuHEjmZmZJCUlAdCiRQuSkpKIioqiXbt29OnTBxEhKSkpzy/rSy65hGrVqlG/fn169erlPVrwtWjRIoYPHw5AmzZtaNasGT/88EPA/Zg7dy633norAB6Ph/h4/6ffPB4PY8aM4W9/+1vR3qgiKqgv7m7dujFixAheffVVsrOzC1x+8ODBVKtWze9rwbyHwVi0aBFDhgyhevXq1KhRg8suu4yFCxcCzt8zNTUVgLPPPrvYR0fGBCOYI4L+qvqX3BFVPSAiA4C/hi6sslfYL/dQSEtLY+7cuaxduxYRITs7GxHh6aefpnv37ixYsICPPvqI4cOHM2bMGK6//npGjhzJk08+SZs2bbjxxhu966patar3eVRUlHc8KiqKrKws72v5L/jydwFYQV+ipWX48OH87W9/o12739/z6OhocnJyvOMlvdv7u+++46yzzjpp+iuvvMKSJUv46KOPSE1NZeXKlX6Xr169eoHr9vceFif+QO+z79/T4/HYqSETUsEcEXhExPupFJFqQNUA85sgzZgxg+uvv56ffvqJbdu2sX37dlq0aMGiRYv46aefaNCgAbfccgs333wzK1asAKBLly5s376dadOmcc011xR5mx988AHp6emkpaUxf/58OnXqRM2aNTly5Ih3nu7du/Pf//4XgB9++IGff/6Z1q1bnzSfrz59+vCvf/0LgOzsbA4fLvg0W0xMDPfcc4/3nDg4581z93HFihVs3bq1yPuWa9u2bdx///3ceeedJ722efNmunTpwmOPPUb9+vXZvn17wP3yx9972KxZM9avX8+JEyc4dOgQc+bM8c5f0Pq7d+/OrFmzOHbsGL/99hszZ87k/PPPL95OG1MCwSSCN4E5InKziNwEfAFMLWQZE4Tp06czZMiQPNMuv/xypk2bxvz580lNTaV9+/a899573H333d55hg4dSrdu3ahTp+j3+HXu3JmBAwfStWtXHnroIRo1akRycjLR0dGkpKTw7LPPctttt5GdnU1SUhJXXXUVU6ZMoWrVqvTq1Yv169d7G1V9Pf/888ybN4+kpCTOPvvsQq8Muvnmm/McqVx++eXs37+f1NRU/vWvf3HmmWcWab82b97svXx06NCh3HnnnXmOmHKNGTOGpKQkEhMT6d69OykpKQH3yx9/7+Fpp53G0KFDSU5OZtiwYbRv3947/6hRo+jfv7+3sThXhw4dGDFiBJ07d6ZLly6MHDkyz3LGlBUJ5jSAiPQH+gACfK6qn4U6sIJ07NhRi90xje9lo+MOsWHDBr+nD8q7QYMGcc8999CnT58iLTdu3Dhq1KjB/fffH6LIjDHlgb/vNhFZrqod/c0fVNE5Vf0E+KTk4ZmSOHjwIJ07dyYlJaXIScAYYwpSaCIQkcuAvwMNcI4IBFBVrRXi2Ew+tWvXLvTqnUDGjRtXesEYYyqMYI4IngYuVtUNoQ7GGGNM2QumsfhXSwLGGFNxBXNEsExE3gZmASdyJ6rq+6EKyhhjTNkJJhHUAo4BfX2mKWCJwBhjKoBgylDf6Ge4qSyCK3O+1UlLYwiCiHjLOYBT7TIhISFPAbOiOHjwIC+//HKRlzt69Ch/+MMfaNWqFe3ataN79+4sWbKkSOvo2bMnRbm0t2fPnnTs+PvVbMuWLaNnz56FLudbtC23RHV+Ho+H1NRU2rVrR0pKCs8884z3zl/fstL+FFaNddeuXVxxxRVA3oJ2wZoyZQq7du3yjo8cOZL169cXaR0FmTVrFo899hgACxYsoEOHDkRHRzNjxow88/Xr14/atWsX63NW1L9zqOQvXnjuueeW+jZ2795N3759C5+xlPl+xgtywQUXcODAgVLZXjDVR88UkTkistYdTxaRClVeIpyqV6/O2rVrvSUEvvjiCxo3blzs9RU3EYwcOZK6devy448/sm7dOqZMmVLoB9FXoLo9gezZs4dPPin9K5OrVavGypUrWbduHV988QUff/wxjz76KEChZaUDJYKsrCwaNWp00hdrUeRPBJMmTaJt27bFXp+vp59+mttuuw2Apk2bMmXKFK699tqT5hszZgxvvPFGqWyzpFQ1T3mOYOVPBAVVvC2JTz/9lIsuuqjU11sahg8fXqz/dX+CaSx+FfgzkAmgqqtxei2LHOWwDwJf/fv356OPPgKcu419S0fs37+fSy+9lOTkZLp27crq1asB51LQm266iZ49e9KyZUvvF9vYsWPZvHkzqampjBkzBoDx48fTqVMnkpOTeeSRR07a/ubNm1myZAlPPPEEUVHOR6Jly5beqpqXXnopZ599Nu3atWPixIne5WrUqMHDDz9Mly5dWLx4cZ51Tp8+3XsH7wMPPFDgvo8ZM4Ynnji5hmH+X9qDBg06qQposBo0aMDEiRN58cUXUdU8JaO/+uorUlNTvXdxHzly5KSy3FOmTOHKK6/k4osvpm/fvid1kLN9+3b69etH69atvckm/zwTJkxg3LhxzJgxg2XLljFs2DBSU1M5fvx4nl/YBb1vNWrU4MEHHyQlJYWuXbvy66+/nrSfP/zwA1WrVvVWOW3evDnJycnev6mvPn36ULNmzYDv28qVK+natSvJyckMGTIkz6/PN998k3PPPZfExERv0T1/7yX4//zlltq+7bbb6NChA48//jh/+tOfvOufMmWKt0SIv8/f2LFjOX78OKmpqQwbNsz7HoGTWMaMGUNiYiJJSUneu8Xnz59Pz549ueKKK2jTpg3Dhg3z1nsaO3Ysbdu2JTk5Oc8Nl59++in9+/f3WxYe4NZbb6Vjx460a9cuz/9W8+bN+ctf/sI555xDx44dWbFiBRdddBGtWrXilVde8cbTvXt3hgwZQtu2bRk9erTfhPjmm2/SuXNnUlNT+cMf/uD90TV48GBvkcoSU9WAA7DUffzOZ9rKwpYL1XD22WdrkT1S6+RBVdevX1/4fCUZglC9enVdtWqVXn755Xr8+HFNSUnRefPm6cCBA1VV9Y477tBx48apquqcOXM0JSXFCfWRR/Scc87R9PR03bt3r9atW1czMjJ069at2q5dO+/6P/vsM73llls0JydHs7OzdeDAgfrVV1/lieGDDz7QSy+9tMAY09LSVFX12LFj2q5dO923b5+qqgL69ttve+fr0aOHLl26VHfu3KmnnXaa7tmzRzMzM7VXr146c+bMk9abO3+vXr107ty5unTpUu3Ro4eqqk6ePFlvv/1277wDBw7UefPmqapqs2bNdO/evd73r6D3Nb/atWvrL7/8kuf9HTRokC5atEhVVY8cOaKZmZl5Xs+NpXHjxt73wfc9njx5sp5yyim6b98+7/uzdOnSk/4O48eP10ceeSTPfhflfQN09uzZqqo6ZswYffzxx0/av9dff13vvffek6bfcMMN+u677540Pf9+5peUlKTz589XVdWHHnpI7777bm+8I0eOVFXVr776yruf/t7Lgj5/W7duVRHRxYsXq6rqnj17tFWrVt5t9+vXTxcuXKiqBX/+8v+Nc8dnzJihF1xwgWZlZekvv/yip512mu7atUvnzZuntWrV0u3bt2t2drZ27dpVFy5cqGlpaXrmmWdqTk6OqqoeOHBAVVWzsrK8/28TJkzQJ554wjv98OHDeWLLysrSHj166KpVq1TV+Yy+/PLLqqr6xz/+UZOSkvTw4cO6Z88eTUhI8L7/VatW1c2bN2tWVpZecMEF3r9T7md8/fr1OmjQIM3IyFBV1VtvvVWnTp3q3efTTz/d+374Oum7TVWBZVrA92owRwT7RKQVTgMxInIFsLt00lCIlfMjgVzJycls27aN6dOnM2DAgDyv+ZaE7t27N2lpaRw6dAiAgQMHen8BNmjQwO+vxM8//5zPP/+c9u3be2v0//jjj0WK74UXXvD+Et2+fbt3eY/Hk6fryVxLly6lZ8+eJCQkEB0dzbBhw1iwYEGB6//rX//q96igtKmfcirdunXj3nvv5YUXXuDgwYNER/u/fuLCCy+kbl3/PbNeeOGF1KtXj2rVqnHZZZexaNGiYsUX6H2rUqWK9yimoLLUu3fvJiEhoVjbzu/QoUMcPHiQHj16AHDDDTfk+RvmHrV2796dw4cPc/DgQb/vZaDPX7NmzejatSsACQkJtGzZkm+++Ya0tDS+//57unXrBhT8+SvIokWLuOaaa/B4PDRs2JAePXqwdOlSwKkT1aRJE6KiokhNTWXbtm3UqlWL2NhYRo4cyfvvv09cXBwAS5YsoUuXLoD/svAA77zzDh06dKB9+/asW7cuT1vP4MGDAUhKSqJLly7UrFmThIQEYmNjvV2Vdu7cmZYtW+LxeLjmmmtO+uzMmTOH5cuX06lTJ1JTU5kzZw5btvzePUyDBg3ynGYsrmCuGrodmAi0EZGdwFbAesgoZYMHD+b+++9n/vz5pKWleaf7+/LKLYOcv1SxbxE33+X//Oc/84c//KHAbbdr145Vq1aRk5Nz0mmE+fPn8+WXX7J48WLi4uLo2bOnt8RybGwsHo/H7zaLonfv3jz00EN888033mmlXZZ6y5YteDweGjRowIYNv98WM3bsWAYOHMjHH39M165d+fLLL/0uH+6y1DExMd7tFPS3rlatmvdHQqj522d/72VBn79t27ad9J5eddVVvPPOO7Rp04YhQ4YgIgE/fwUJ9D76+5+Jjo7m22+/Zc6cObz11lu8+OKLzJ07l08++YR+/foB+C0Lf/755zNhwgSWLl1KnTp1GDFiRJ7YfEvB5y8Tn/v3K6wsvKpyww03FNh/R3p6eoH9ZhRFMFcNbVHVC4AEoI2qnqeq20q8ZZPHTTfdxMMPP+ztaCaXb0no+fPnU79+fWrVKri6R/6SxxdddBGvv/66t8ewnTt3smfPnjzLtGrVio4dO/LII494/4l+/PFHPvjgAw4dOkSdOnWIi4tj48aNeb6sC9KlSxe++uor9u3bR3Z2NtOnT/f+sizIgw8+yNNPP+0db968OStXriQnJ4ft27cXu/MXgL179zJ69GjuuOOOk/7RNm/eTFJSEg888AAdO3Zk48aNRS5L/cUXX7B//36OHz/OrFmz6NatGw0bNmTPnj2kpaVx4sQJb+9vUHBZ6uK8b77OOuusUuv5LT4+njp16njPhb/xxht5Ysk9775o0SLi4+OJj4/3+14G8/nLddlllzFr1iymT5/OVVddBRDw8xcTE0NmZuZJ6+nevTtvv/022dnZ7N27lwULFtC5c+cC9/Xo0aMcOnSIAQMG8Nxzz3n7qJgzZ463ppe/svCHDx+mevXqxMfH8+uvvxbroodvv/2WrVu3kpOTw9tvv815552X5/U+ffowY8YM73u2f/9+fvrpJ8BJEr/88gvNmzcv8nbzC6bW0MP5xnODeKzEWy9vxpXNryl/mjRpkqfUdK5x48Zx4403kpycTFxcHFOnBq4AXq9ePbp160ZiYiL9+/dn/PjxbNiwgXPOOQdwGtTefPNNGjRokGe5SZMmcd9993H66acTFxdHvXr1GD9+PMnJybzyyiskJyfTunVr76F8IKeeeip/+9vf6NWrF6rKgAEDuOSSSwIuM2DAgDynNbp16+btdS0xMZEOHToUul1fuQ2JmZmZREdHM3z4cO69996T5nvuueeYN28eHo+Htm3b0r9/f6KiorxluUeMGFFoue/zzjuP4cOHs2nTJq699lrvJbG5DektWrSgTZs23vlHjBjB6NGjqVatWp5G9uK8b766d+/Offfdh6oiIixdutTbyPt///d/PPLII97y4Oeffz4bN27k6NGjNGnShNdee+2kq2OmTp3K6NGjOXbsGC1btmTy5Mne1+rUqcO5557L4cOHef311wt8L6tWrer38+fvSLJOnTq0bduW9evXe7+4+/XrV+Dnb9SoUSQnJ9OhQwfvjyWAIUOGsHjxYlJSUrwdPZ1yyil+uy4FOHLkCJdccgnp6emoKs8++yx79+4lNjbW+6Nr/vz5jB8/npiYGGrUqMF//vMfWrRoQfv27WnXrh0tW7b0nsoqinPOOYexY8eyZs0ab8Oxr7Zt2/LEE0/Qt29fcnJyiImJ4aWXXqJZs2YsX76crl27Fng6sygKLUMtIvf5jMYCg4ANGsS9BCLSD3ge8ACTVPWpfK/H4/R30BQnKU1Q1cknrchHkcpQB2ofiOAy1MYU5O677+biiy/mggsuCHcoEe3NN99kx44djB07NmTbmD9/fp6+wovq7rvvZvDgwX4rEZd6GWpV/Ue+lU0AZhe2nIh4gJeAC4EdwFIRma2qvnfO3A6sV9WLRSQB+F5E/quqGYWt3xhzsr/85S9FvhHQnOy668p/M2hiYmKplaMP5qqh/OKAlkHM1xnY5LYxZABvAfmPcxWoKc75phrAfuDkVjBjTFAaNmzovVrFlG89e/Ys9tEAwC233FJqsQTTRrAG99JRnFM8CUAw7QONge0+4zuALvnmeRHn6GIXUBO4SlVPuqNCREYBo8C5W7I05Z5PNcaYiqCoV+1BcJeP+hYjycIpSx3Mr3Z/3675I7wIWAn0BloBX4jIQlXN0/O5qk7EuYSVjh07Fn0vCxAbG0taWhr16tWzZGCMiXiqSlpaGrGxsUVaLphEkP86t1q+X5qqur+A5XYAp/mMN8H55e/rRuAp9663TSKyFWgDFP9aQQj6JrImTZqwY8cO9u7dW6LNGWNMeREbG0uTJk2KtEwwiWAFzhf6AZxf+bWBn93XlILbC5YCZ4hIC2AnTn2i/NWvfgb6AAtFpCHQGthCGYmJiaFFixZltTljjCmXgmks/hSnq8r6qloP51TR+6raQlULbDR2Tx/dAXwGbADeUdV1IjJaREa7sz0OnOu2Q8wBHlDV4EtelkSElJ8wxphQC+aIoJOq5n5xo6qfiMjjwaxcVT8GPs437RWf57vI2+GNMcaYMhZMItjn9j/wJs6poOuAtMCLGGOMiRTBnBq6BueS0ZnukOBOM8YYUwEEc2fxfuBuEamhqkfLICZjjDFlKJiuKs8VkfXAenc8RURKp380Y4wxYRfMqaFncW78SgNQ1VVA91AGVabs6iFjTCUXVK0hVd2eb1Lxeio3xhhT7gRz1dB2ETkXUBGpAtyFc1+AMcaYCiCYI4LROOWiG+OUjUh1x40xxlQAAY8I3D4FnlPVYWUUjzHGmDIW8IhAVbOBBPeUkDHGmAoomDaCbcDXIjIb+C13oqo+E6qgjDHGlJ1gEsEud4jC6TzGGGNMBVJgIhCRKao6QlUfFZEbVHVqWQZmjDGmbARqI0jxeX53qAMxxhgTHoESQal1CWmMMab8CtRG0EREXsDplSz3uZeq3hXSyIwxxpSJQIlgjM/zZaEOxBhjTHgUmAiscdgYYyqHoIrOGWOMqbgsERhjTCVnicAYYyq5QDeU/ZMAl5DaVUPGGFMxBDoiWAYsB2KBDsCP7pCKdUxjjDEVRqFXDYnICKCXqma6468An5dJdMYYY0IumDaCRuQtNlfDnWaMMaYCCKb66FPAdyIyzx3vAYwLWUQlYZ3QG2NMkRWaCFR1soh8AnRxJ41V1V9CG5YxxpiyUuipIRER4AIgRVU/AKqISOeQR1bWxsXbEYUxplIKpo3gZeAc4Bp3/AjwUsgiMsYYU6aCaSPooqodROQ7AFU9YH0YG2NMxRHMEUGmiHhwby4TkQQgJ6RRGWOMKTPBJIIXgJlAAxH5f8Ai4MmQRmWMMabMBHPV0H9FZDnQB6eTmktVdUPIIzPGGFMmCk0EIvI88Laqlt8GYrvaxxhjii2YU0MrgL+KyCYRGS8iHUMdlDHGmLJTaCJQ1amqOgDoDPwA/F1Efgx5ZMYYY8pEUfojOB1oAzQHNgazgIj0E5Hv3aOJsQXM01NEVorIOhH5qgjxGGOMKQXBtBH8HbgM2Ay8DTyuqgeDWM6Dc+PZhcAOYKmIzFbV9T7z1Ma5Ya2fqv4sIg2KsxPGGGOKL2AicMtLHAXOUdV9RVx3Z2CTqm5x1/UWcAmw3meea4H3VfVnAFXdU8RtGGOMKaGAp4ZUVXEuFy1qEgBoDGz3Gd/hTvN1JlBHROaLyHIRub4Y2zHGGFMCwZSY+EZEOqnq0iKuW/xMy9/1ZTRwNs49CtWAxSLyjar+kGdFIqOAUQBNmzYtYhjGGGMCCaaxuBdOMtgsIqtFZI2IrA5iuR3AaT7jTYBdfub5VFV/c486FgAp+VekqhNVtaOqdkxISAhi08YYY4IVzBFB/2Kueylwhoi0AHYCV+O0Cfj6AHhRRKKBKjh9HjxbzO0ZY4wphmDuI/gJ55d9b/f5sSCXywLuAD4DNgDvqOo6ERktIqPdeTYAnwKrgW+BSaq6trg7Y4wxpuiCuXz0EaAj0BqYDMQAbwLdCltWVT8GPs437ZV84+OB8cGHbIwxpjQF00YwBBgM/AagqrvI25m9McaYCBZMIshwLyPN7Y+gemhDMsYYU5aCSQTviMi/gdoicgvwJfBqaMMyxhhTVoLpj2CCiFwIHMZpJ3hYVb8IeWThklvSetyh8MZhjDFlJJjG4urAXFX9QkRaA61FJEZVM0MfnjHGmFAL5tTQAqCqiDTGOS10IzAllEEZY4wpO8EkAlHVYzgVSP+pqkOAtqENyxhjTFkJKhGIyDnAMOAjd1owdyQbY4yJAMEkgj8CfwZmuncGtwTmhTQqY4wxZSaYq4a+Ar4SkVoiUtPtX+Cu0IdmjDGmLBR6RCAiHUVkDU49oLUiskpEzg59aMYYY8pCMOf6XwduU9WFACJyHk7NoeRQBmaMMaZsBNNGcCQ3CQCo6iLgSOhCMsYYU5YKPCIQkQ7u02/dEhPTceoNXQXMD31ohci9A9gYY0yJBDo19I9844/4PM/f5aQxxpgIVWAiUNVeZRmIMcaY8AjYRiAiiSIyVUSWichS93lSWQXnz54jJ8K5eWOMqXAKTAQicgkwE/gKuAkY6T5/330tLH49nM6WvUfDtXljjKlwArURPAZcqKrbfKatEpG5OJ3OfxDKwAoiwKRFW3kyHBs3xpgKKNCpoZh8SQAAd1pMqAIqTJ3qVZixfAf7tFa4QjDGmAolUCLIFJGm+SeKSDMgK3QhBVa/RlUys3P4T1bfcIVgjDEVSqBE8AjwpYiMEJEkt+H4RuBz4OGyCe9kVaOjuOCshvwn+0KOadVwhWGMMRVGgYlAVWcBVwK9cTqi+Q/QCxjqvhY2o3u05CA1eSe7RzjDMMaYCiFgrSFVXQVcX0axBO3sZnU5W75nUvYArvN8SbTkhDskY4yJWMHUGiqXRkV/xA5twCc5ncMdijHGRLSITQQXRi2npexiYtYg1ApeGGNMsQW6oewaEalXlsEURZQoIz0fs0ZbsjjHulA2xpjiCnRE0Ax4V0QWisg4EekiIlJWgQXjMs9C6nOIf2cPCncoxhgTsQJdNfSUqvYGBgCrcMpMrBCRaSJyvYg0LKsgCxIrmYyI/pSvclLZmHNauMMxxpiIVGgbgaoeUdWZqvoHVW0PPAEk4FxOGnbXeb4kjnQmZg0s3RWPi/99MMaYCqzIjcWqul5V/6GqF4UioKKqLb8x1DOf2TnnslvrhjscY4yJOBF71ZCvmz0fowivZ/ULdyjGGBNxKkQiOC1qHxdHLebN7AvYo3YqxxhjiqLQRCAiE0SkXVkEUxJ3R79PBjG8lHVpuEMxxpiIEswRwUZgoogsEZHRIlIuf3K3iPqFoZ75TMvuw/achHCHY4wxESOYq4YmqWo3nJpDzYHV7iWk5a5P47uj30fI4dmsy8MdijHGRIyg2ghExAO0cYd9OPcV3Csib4UwtiI7RQ5wg+dzZuacxw85jcMdjjHGRIRg2giewTk9NAB4UlXPVtW/q+rFQPtClu0nIt+LyCYRGRtgvk4iki0iVxQa8a7vAl7bf2v0/1GddCZkDS10VcYYY4I7IlgLpLg3lH2b77UCS3+6RxEvAf2BtsA1InJSUSB3vr8DnwUddQB15Qi3RH/E5zmdWJnTqjRWaYwxFVowiWCYqh7znSAicwBU9VCA5ToDm1R1i6pmAG8Bl/iZ707gPWBPcCEX7mbPJ9TjEOOzriqtVRpjTIUVqPporIjUBeqLSB0RqesOzYFGQay7MbDdZ3yHO813G42BIcArgVYkIqNEZJmILNt7rPCa0zUknduiP+DrnES+zi73V74aY0xYBToi+AOwHKeBeIX7fDnwAc4pn8L4q1Sa/1v8OeABVc0OtCJVnaiqHVW1Y0JccAVQh3nm0Ih9PJ11lfVXYIwxAQSqPvq8qrYA7lfVFj5Diqq+GMS6dwC+JUGbALvyzdMReEtEtgFXAC+LyKVF2oMCxEomf4x+j1V6Op/ldCyNVRpjTIUU6NRQb/fpThG5LP8QxLqXAmeISAsRqQJcDcz2ncFNLM1VtTkwA7hNVWcVa0/8uMyzkFayk39kDSVby1VXCsYYU24EOjXUw3282M9QaE8wqpoF3IFzNdAG4B1VXefenTy6RFEHKVpyuC/6XX7UJryffX5ZbNIYYyKOaISdQO/YyKPLRtUIev4cFYZkPMYurcucqvdTS44XfaPjAl0cZYwx5Z+ILFdVv+fJg7mh7G4RqSWOSSKyQkT6ln6YoRElymMxk9lHPM9mFX6/mjHGVDbB3Edwk6oeBvoCDYAbgadCGlUpS4nawrWeuUzNvoj1OU3DHY4xxpQrwSSC3FbWAcBkVV2F/0tDy7Ux0W9Tm6M8lHkjOdZwbIwxXsEkguUi8jlOIvhMRGoCOaENq/TVlt8YGz2d5dqaGdndwx2OMcaUG8EkgpuBsUAnt9REFZzTQxHnCs8COsr3PJV1DQe1evALWif2xpgKLJj+CHKAX4G2ItIdaAfUDnFcIRElyuMxr3OI6jxtdYiMMQaA6MJmEJG/A1cB64HcUhAKLAhhXCFzVtR2bvB8xuTsfgz1fEVq1OZwh2SMMWEVzKmhS4HWqjpAVS92h8Ehjiuk7ol+jwQO8dfMm+yOY2NMpRdMItgCxIQ6kLJUU47z15g3WastmJbdJ9zhGGNMWBV6agg4Bqx0+yA4kTtRVe8KWVRl4OKoxbwV1Yuns66in+dbEuRwuEMyxpiwCOaIYDbwOPA/fi9FvTyUQZUFEXgsegrpVOWhzJusVLUxptIq9IhAVaeKSDWgqap+XwYxlZnTo3ZxX/Q7PJV1LTOyu3NldES2fxtjTIkEU2voYmAl8Kk7nioiswMuFEFu8XxEZ9nAo1nXsz0nIdzhGGNMmQvm1NA4nP6HDwKo6kqgRcgiKmMeUZ6p8i8A7s281a4iMsZUOsEkgiw/ndRXqDPqTWQfj8ZMZam24d/ZhXa1YIwxFUowiWCtiFwLeETkDBH5J07DcYVyWdRCBkQt4dmsK1mb0yzc4RhjTJkJJhHciVNW4gQwHTgM/DGEMYWFCPy/mNeowxH+mHk76Vqhbp0wxpgCBVNr6JiqPqiqnVS1o/s8vSyCK2t15CgTYl5hkzbhqaxrwh2OMcaUiYCJQERucHsk+80dlonI9WUVXDh096xhhOdTpmT3Y2F2YrjDMcaYkCswEbhf+H8E7gMaAY2BPwF3V/Rk8ED0W7SSndyfOZoDGnz/yMYYE4kCHRHcBgxR1XmqekhVD6rqXOBy97UKq5pk8HzMSxygJndm3kmWBtOUYowxkSnQN1wtVd2Wf6I7rVaoAiovEqO28UT06yzKSeLprKvDHY4xxoRMoBITx4v5WoUxNPor1moLJmYPol3UNi4Jd0DGGBMCgRLBWSKy2s90AVqGKJ5y56HoN9iYcxp/yhxFq52HSGxsXVYaYyoW0QLKbopIwLuqVPWnkERUiI6NPLpsVNk24O7TWlx84v8RVbsJs+/oRr0aVct0+8YYU1IislxVO/p7rcA2AlX9KdAQunDLn/pymIlVnmHf0RPcMe07MrNzwh2SMcaUGrscJkhJUVv522VJLN6SxpMfbwh3OMYYU2qC6aHMuC7r0IR1uw7z2qKttGsUzxVnNwl3SMYYU2LB9EcwSETsyAFgXDx/XtqNc1vV4y8z1/DNlrRwR2SMMSUWzBf81cCPIvK0iJwV6oDKu2jJ4aVrO9C0bhy3TF3G2p35K3QbY0xkCabo3HVAe2AzMFlEFovIKBGpGfLoyqk61avwxs2dqVUthhte/5Yte4+GOyRjjCm2oE75qOph4D3gLeBUYAiwQkTuDGFs5dqp8dV44+bOAAx/7Vt2H6oU99gZYyqgYNoIBovITGAuEAN0VtX+QApwf4jjK9daJtRg6k2dOXw8k+smLWH/bxnhDskYY4osmCOCK4BnVTVZVcer6h5w+ikAbgppdBEgsXE8k27oyI4Dxxkx+VuOnsgKd0jGGFMkwSSC3aq6wHeCiPwdQFXnhCSqCNOlZT1eurYD63YdZtR/lpGemR3ukIwxJmjBJIIL/UzrX9qBRLoL2jZkwpXJ/G9zGndN/46MLLv72BgTGQJ1THOriKwB2ojIap9hK+CvGJ2/dfQTke9FZJOIjPXz+jCf9f5PRFKKvyvhN6R9Ex4d3I7P1//KLf9ZxvEMOzIwxpR/ge4sngZ8AvwN8P0SP6Kq+wtbsYh4gJdwjih2AEtFZLaqrveZbSvQQ1UPiEh/YCLQpYj7UK7ccG5zqkZH8eeZa7j+9SW8NqITtWJjwh2WMcYUKNCpIXU7obkdOOIzICJ1g1h3Z2CTqm5R1QycS0/zlPRX1f+p6gF39BugQtRsuLpzU168pgMrtx/kmonfsO/oiXCHZIwxBQqUCKa5j8uBZe7jcp/xwjQGtvuM73CnFeRmnCOQk7g3sC0TkWV7j/kvm13eDEw+lVev78jmvUcZ+u/F7Dpo9xkYY8qnQGWoB7mPLVS1pfuYOwTTMY34W63fGUV64SSCBwqIZaKqdlTVjglx/lZbPvVs3YA3bu7C3sMnuPKVxWzd91u4QzLGmJMEaizuEGgIYt07gNN8xpsAu/xsJxmYBFyiqhWuilun5nWZPqorxzOzufKVxazfdTjcIRljTB6BeiibF2A5VdXeAVcsEg38APQBdgJLgWtVdZ3PPE1x7li+XlX/F0zA4eihrEDjgi84t2nPUYa/toTDxzN57ur2XNi2YQgDM8aYvAL1UFbgVUOq2qskG1XVLBG5A/gM8ACvq+o6ERntvv4K8DBQD3hZRACyCgo00p3eoAbv33Yuo/6znFFvLOO+C8/k9l6n4+63McaETaAjgt6qOldELvP3uqq+H9LIChCpRwS50jOzGfveamat3MXApFMZf2UycVWsfyBjTGgV64gA6IFz2uZiP68pEJZEEOliYzw8e1UqbRvV4qlPNrJl32+8ev3ZNKkTF+7QjDGVVIFHBOVVpB8R+Jr//R7unP4dMZ4oXh7Wga4t65VSYMYYk1egI4JgylDXE5EXRGSFiCwXkedFxL6xSkHP1g2YdXs3asfFcN2kJby2aCs5OZGVmI0xkS+YonNvAXuBy3FKUu8F3g5lUJVJq4QazLq9Gz1bJ/D4h+sZMWUpew6nhzssY0wlEkwiqKuqj6vqVnd4Aqgd4rgqlVqxMbx6fUcevzSRb7em0e/5hXy+7pdwh2WMqSSCSQTzRORqEYlyh6HAR6EOrLIREYZ3bcaHd57HqfGxjHpjOX9+fzXHMqyjG2NMaAW6fPQIztVBAlQHcgvsRwFHVbVWmUSYT0VqLC5IRlYOz3zxA/9esJkW9arz3NWpJDepHZJtGWMqh0CNxXbVUGkIUUJYvDmNe99Zyd4jJ7i1Zytu73U6sTGekGzLGFOxleiqIXcFdUSks4h0zx1KN0Tjzzmt6vHp3d0ZnNKIf87dRN9nFzDv+z3hDssYU8EEc/noSGABTqmIR93HcaENy+SKj4vhmatSmXZLF6I9wo2Tl3Lbf5fzyyG7ssgYUzqCOSK4G+gE/OTWH2qPcwmpKUPntqrPJ3efz/19z2TOhj30+cd8Ji3cQla29Y1sjCmZYBJBuqqmA4hIVVXdCLQObVgRZly8M4RY1WgPd/Q+gy/u6UGnFnV54qMNXPzi13y9aV/It22MqbiCSQQ7RKQ2MAv4QkQ+wE+/AqbsNK0Xx+QRnfjXsA4cPp7JsElLGP7aEtbsCE2jtTGmYivSVUMi0gOIBz51+yEuc+XyqqFcIbp6KJD0zGze/OYnXpq3iQPHMhmYfCr3921Ni/rVyzwWY0z5Vdzqo74r6ACch3NfwdfhSgLmZLExHkae35KhnU5j0oItTFq0lU/X/sJVnU7j7j5n0LBWbLhDNMaUc8FcNfQwMBWnA5n6wGQR+WuoAzNFUys2hnv7tuarMb0Y1qUp7yzdTven5/HQrLX8nHYs3OEZY8qxQk8NicgGoL1Pg3E1YIWqnlUG8Z3ETg0F56e033hp3iZmfreT7BxlQNKpjO7RisTGoW/UNsaUPyW9oWwb4Ht+oSqwuRTiMiHUrF51nr4ihUUP9OaW81sy//u9DPrnIoa/toRFP+4j0u4oN8aETqBaQ//EaRNoinMfwRfu+IXAIlW9uqyC9GVHBMVzOD2T/37zM69/vZW9R07Q9tRaXNe1GYNTG1GjqnWVaUxFV6xaQyJyQ6CVqurUUoityMp1IvCnnCWH9MxsZn63k6n/28bGX45QvYqHS9o35trOTe20kTEVWImLzolIFeBMd/R7Vc0sxfiKxBJB6VBVvtt+kGlLfubD1btIz8whpUk813ZpysUpjYirYkcJxlQkJUoEItIT56qhbTglqU8DblDVBaUaZZAiLhHkKqcJAeDQsUxmfreDad/+zA+/HiWuiocL2zZkcEojzj8jgSrRQdUmNMaUYyVNBMuBa1X1e3f8TGC6qp5d6pEGwRJB6Kgqy386wPvf7eTjNbs5eCyT2nEx9E88lcEpjejSoi5RURLuMI0xxVDSRLBaVZMLm1ZWIj4R5NYkKueJISMrh0Wb9jJ75S4+X/8rxzKyaVirKv0TT+WCsxrSuUVdO1IwJoKU9M7i5SLyGvCGOz4MWF5awZnyqUp0FL3bNKR3m4Ycy8hizoY9zF61i+nf/syU/22jZtVoerRO4IKzGtKzdQK146qEO2RjTDEFkwhGA7cDd+G0ESwAXg5lUKZ8iasSzcUpjbg4pRHHM7JZtGkfX67/lTkb9/Dh6t14ooROzevQ/cwEzju9Pu0axeOxU0jGRIyAp4ZEJApYraqJZRdSYBF7aii/cn5qKBg5OcqqHQf5csOvzNmwh42/HAEgvloM57SsR7cz6tOtVT1a1K+OiCUGY8Kp2KeGVDVHRFaJSFNV/Tk04ZlIaTfILypKaN+0Du2b1mHMRW3YcySdxZvT+HrTPr7elMan634B4NT4WDo1r0vH5nXo0LQObU6pSbTH2heMKS+CaSyei3Nn8bfAb7nTVXVwaEPzr8IcEfgTYYkgEFXlp7RjLNq0j8Wb01j2035+PXwCgOpVPKQ2rc3ZzepydrM6JDWOp251a2MwJpRK2lj8aCnHYyoBEaF5/eo0r1+d67o2Q1XZefA4y386wLJtB1j+0wFenPsjOe7vkMa1q5HYuBZJjeNJbBxPUuN46tWoGt6dMKaSCFRiIhanofh0YA3wmqpmlWFsflXoI4L8KtARgj9H0jNZs+MQa3Y6w9qdh9jmUzL71PhYzmxYkzan1OTMhjVpfUpNTm9Qg9gYTxijNiYyFfeIYCqQCSwE+gNtcTqyN6ZU1IyN4dzT63Pu6fW90w6nZ7Ju52HW7jzE+t2H+f6XIyzekkZGVg4AUeJUVj29QQ1aJlSnZf3qtKjvPK9XvYo1ShtTDIESQVtVTQJw7yP4tmxCMiETAY3StWJjOKdVPc5pVc87LSs7h5/2H+OHX46w8ZcjfP/LETbvPcpX3+8lIzvHZ9loWiTUoFndOJrWjeO0utU4rW4cp9WJ49T4WGugNqYAgRKBt7CcqmbZL60wGhdEVdBy/OVeUtGeKFol1KBVQg36J53qnZ6do+w8cJwt+46ydd9vbNn7G1v2HeW77Qf4aM1usnN+P+0ZHSU0ql2NRrVjaRRfjVNrx3JqfDVOjXceG9WOJb5ajB1RmEopUCJIEZHD7nMBqrnjAqiq1gp5dJVdMAkg/7wRVsqiJDxRQtN6cTStF0fP1nlfy8rOYfehdLbvP8bP+4+x/cAxft5/nN0Hj7Nk635+OZyeJ1GAczd1g5pVaVCzKg1rxTrPa8WSULMq9WtUoV71qtRzH6tVsXYKU3EUmAhU1T7pFVVBScI38UR4Aon2RDmnherGca6f17NzlL1HTrDr0HF2H0xn96Hj7Dlygj2H09lz5AQ/7jnK15v2cTjd//URcVU81KtRhbrVq1InLoY6cVWo7T7WiYuhtjseXy2GWrHOY83YaDs9ZcolKzpf0ZTkKKK480QgT5RwSnwsp8THOn3wFSA9M5u9R06Q9lsGaUdPkHY0g32/OY9pR3OnZ7Bpz1EOHsvk6InAF9ZVr+Jxk0IMNWKjqRkbTY2qzmPN2BhqVI2metVoqlfxOI9VPVSv4kyLq+Ihrko01ap4iKviIcaSiiklQXVMU+yVi/QDngc8wCRVfSrf6+K+PgA4BoxQ1RWB1lmpLh8tT4JJBPmTUFGSRwVJOBlZORw8nsHBY5kcOp7JoWOZHE53nh8+nuV9fiTdSRpH07M4kp7FEff58czsoLcV4xGqxfyeHGJjPMTGRBEb7XHHo9xpHqpGR1E12n2M+f15lego9zXneRWPx3mMjqKKJ4oq0UIVj4eYaCHGE0WMx5ke7RGio8TaVCJISW8oK+5GPcBLOH0c7wCWishsVV3vM1t/4Ax36AL8y3005U1RjjTyL+Pvyz3Y9fk7XVWUhFPYabCSJp5863HaGWJpUDO2WKvLzM7hWEY2v53I4lhGFkdPZHPsRBZHT2RxLCPbHbI4npHN8Uxn/HhGNscys0n3GfYeySI905knPTOHE1nZnMjK8V6GW1piPEJ0lJMYfk8QvyeKGE8Unigh2hNFTJS4zwVPVBTRueNRQpT76BFn2kmDOz3KfR7lneaUOokSZ1wE7zIiQpTgzO++FiVCVJT76B3wzus7n79HIe/8Th78fZrw+7z4PM9d1lmXs4wz7fflIO983vECXvem4PzLFLDOQEJ5aqgzsElVtzjByFvAJYBvIrgE+I86hyXfiEhtETlVVXeHMC5T1krrdFVB6wnqqqpizFOcxBNoe4Wtb1w8MUC8OwRef4D3p6CkFw05HiHjr2mcyMrhRKaTHE5k5ZDx4jlkEEMG0WSo+0g0mVe8QWZ2DpnZOWRkK5lZOWR8+ley8JDR469kZeeQufB5MnOiyUodRWaWkrniv2TjIQsPmW0uITtHyfzhC7LwkN2sOycyc8jKySY7R8nKUbJzcsjKUXLcce/jkT1kEUU2HrKJIjs6jhxVsnOUnNCdyKiUQnZqSESuAPqp6kh3fDjQRVXv8JnnQ+ApVV3kjs8BHlDVZfnWNQoY5Y4mAmtDEnTZqQ/sC3cQJRDp8UPk70Okxw+Rvw+RFn8zVU3w90Iojwj8HYvkzzrBzIOqTgQmAojIsoLOc0WKSN+HSI8fIn8fIj1+iPx9iPT4fYXysoMdOB3d52oC7CrGPMYYY0IolIlgKXCGiLQQkSrA1cDsfPPMBq4XR1fgkLUPGGNM2QrZqSG3LMUdwGc4l4++rqrrRGS0+/orwMc4l45uwrl89MYgVj0xRCGXpUjfh0iPHyJ/HyI9foj8fYj0+L1Ceh+BMcaY8s9uTTTGmErOEoExxlRyEZUIRKSfiHwvIptEZGy44ykKETlNROaJyAYRWSciEdnJj4h4ROQ79x6QiOPetDhDRDa6f4tzwh1TUYnIPe5naK2ITHd7Eyy3ROR1EdkjImt9ptUVkS9E5Ef3sU44YyxMAfsw3v0crRaRmSJSO4whlkjEJAKfkhW5vaVdIyJtwxtVkWQB96nqWUBX4PYIiz/X3cCGcAdRAs8Dn6pqGyCFCNsXEWkM3AV0VNVEnAsxrg5vVIWaAvTLN20sMEdVzwDmuOPl2RRO3ocvgERVTQZ+AP5c1kGVlohJBPiUrFDVDCC3ZEVEUNXduQX1VPUIzhdQ4/BGVTQi0gQYCEwKdyzFISK1gO7AawCqmqGqB8MaVPFE4/QPEg3EUc7vvVHVBcD+fJMvwekOF/fx0rKMqaj87YOqfu7Tj/s3OPdBRaRISgSNge0+4zuIsC/SXCLSHGgPLAlzKEX1HPAnoHQrl5WdlsBeYLJ7emuSiFQPd1BFoao7gQnAz8BunHtvPg9vVMXSMPeeIfexQZjjKambgE/CHURxRVIiCKocRXknIjWA94A/qurhwuYvL0RkELBHVZeHO5YSiAY6AP9S1fbAb5T/UxJ5uOfSLwFaAI2A6iJyXXijqtxE5EGcU7//DXcsxRVJiSDiy1GISAxOEvivqr4f7niKqBswWES24ZyW6y0ib4Y3pCLbAexQ1dwjsRk4iSGSXABsVdW9qpoJvA9+O2Er734VkVMB3Mc9YY6nWETkBmAQMEwj+KasSEoEwZSsKLfcTnheAzao6jPhjqeoVPXPqtpEVZvjvPdzVTWifomq6i/AdhHJ7eG4D3nLokeCn4GuIhLnfqb6EGEN3q7ZwA3u8xuAD8IYS7G4HW89AAxW1WPhjqckIiYRuI0yuSUrNgDvqOq68EZVJN2A4Ti/pFe6w4BwB1UJ3Qn8V0RWA6nAk+ENp2jco5kZwApgDc7/cLkudSAi04HFQGsR2SEiNwNPAReKyI84nVc9FWgd4VbAPrwI1AS+cP+fXwlrkCVgJSaMMaaSi5gjAmOMMaFhicAYYyo5SwTGGFPJWSIwxphKzhKBMcZUcpYITNiIyHwRuSjftD+KyMuFLBPSDsPdip6rReSefNMvLW6hQBFJLcrlwm6V1NuKOp+INBKRGaU1v6kcLBGYcJrOyZUzr3anh4WInAKcq6rJqvpsvpcvxal8WxypON2yBqs2UGgiyD+fqu5S1StKcX5TCVgiMOE0AxgkIlXBW4yvEbBIRP4lIsvcuvuP+ltYRI76PL9CRKa4zxNE5D0RWeoO3fwsGysik0VkjVuArpf70udAA/cGofN95j8XGAyMd19r5Q6fishyEVkoIm3cea90+wpYJSIL3DvhHwOucpe9Kl8s7UTkW/e11SJyBs4NVq3caeNFpIaIzBGRFW7MuZV388/XPLdmfpDr9Z3fIyIT3PWvFpE7g/5LmsimqjbYELYB+Ai4xH0+FhjvPq/rPnqA+UCyOz4fpxY/wFGf9VwBTHGfTwPOc583xSnrkX+79wGT3edtcEo3xALNgbUFxDoFuMJnfA5whvu8C07ZDXDu+G3sPq/tPo4AXixgvf/EqVUDUAWolj8OnIJ5tdzn9YFNOIUY88/nHQ9yvb7z34pTCyva929gQ8UfogvJE8aEWu7poQ/cx5vc6UNFZBTOF+CpOKdkVge5zguAtk4pHgBqiUhNdfqByHUezhclqrpRRH4CzgSCqggrThXZc4F3fbZT1X38GpgiIu/gFIUrzGLgQXH6e3hfVX/0Wad3k8CTItIdpwx4Y6BhKazX1wXAK+rW2FfV/H0ImArKEoEJt1nAMyLSAaimqitEpAVwP9BJVQ+4p3z8dcfoWx/F9/Uo4BxVPR5guwG/EYMQBRxU1dSTglIdLSJdcDrxWSkiJ82Tb/5pIrLEnf8zERkJbMk32zAgAThbVTPFqQIbsIvKINfrS4jA0u6m5KyNwISVqh7FOd3zOr83EtfC6SvgkIg0xOme1J9fReQsEYkChvhM/xynQCHgXLHjZ9kFOF+uiMiZOKeQvi8k3CM4RcZQpy+JrSJypbsOEZEU93krVV2iqg8D+3DKp3uXzU9EWgJbVPUFnKqcyX7mj8fpDyLTbc9olj+mYq7X1+fAaHF6PkNE6hbyfpgKwhKBKQ+m4/Qf/BaAqq4CvgPW4SSIrwtYbizwITAXp7euXHcBHd0Gz/XAaD/Lvgx4RGQN8DYwQlVPFBLnW8AYt3G5FU4iuVlEVrmx5jbgjncbXNfiJJxVwDyc01UnNRYDVwFrRWQlTnvFf1Q1DfjabXQej9PpSUcRWeZud6P7XuWfr6jr9TUJp61ktbtP1xbyfpgKwqqPGmNMJWdHBMYYU8lZIjDGmErOEoExxlRylgiMMaaSs0RgjDGVnCUCY4yp5CwRGGNMJff/AUSnK4ZrDKtYAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "# Draw 10000 samples, each with 11 observations\n", + "n_observations = 11\n", + "n_samples = 10000\n", + "y = norm_dist.rvs(size=(n_observations, n_samples))\n", + "\n", + "# calculate the value of the statistic for each sample\n", + "null_dist_mc = statistic(y)\n", + "\n", + "# compare the MC and asymptotic distributions\n", + "plt.plot(y_grid, pdf)\n", + "plt.hist(null_dist_mc, density=True, bins=200)\n", + "plt.xlim(0, np.max(y_grid))\n", + "plt.xlabel(\"Value of test statistic\")\n", + "plt.ylabel(\"Probability Density / Observed Frequency\")\n", + "plt.legend(['Asymptotic Null Distribution', 'Monte Carlo Null Distribution (11 observations/sample)'])\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "3c15ba7a-53c4-42b4-833c-39d86e2ba969", + "metadata": {}, + "source": [ + "This is because the asymptotic null distribution was derived under the assumption that the number of observations approaches infinity (hence the name \"asymptotic\" null distribution). Apparently, it is quite different from the null distribution of the test statistic when the number of observations is 11.\n", + "\n", + "The true theoretical null distribution when the number of observations is 11 may not be possible to calculate analytically (in a way that can be expressed in terms of a finite number of common functions). So rather than comparing a test statistic against a theoretical null distribution to determine the $p$-value, the approach of the *Monte Carlo test* is to compare the test statistic against the Monte Carlo null distribution." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "17e06351-479d-49b6-989f-7a30bfde43ab", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Asymptotic p-value 0.03045746622458189\n", + "Monte Carlo p-value: 0.0085\n" + ] + } + ], + "source": [ + "res = stats.jarque_bera(x)\n", + "stat, p_asymptotic = res\n", + "count = np.sum(null_dist_mc >= stat)\n", + "p_mc = count / n_samples\n", + "print(f\"Asymptotic p-value {p_asymptotic}\")\n", + "print(f\"Monte Carlo p-value: {p_mc}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e27fd4f9-64eb-42ca-9131-b6f1cefa9698", + "metadata": {}, + "source": [ + "These $p$-values are substantially different, so we might draw different conclusions about the validity of the null hypothesis depending on which test we perform. Under the 1% threshold used above, the Monte Carlo test would suggest that there is evidence for rejection of the null hypothesis whereas the asymptotic test performed by `stats.jarque_bera` would not. In other cases, the opposite may be true. In any case, it seems that the Monte Carlo test should be preferred when the number of observations is small.\n", + "\n", + "`stats.monte_carlo_test` simplifies the process of performing a Monte Carlo test. All we need to provide is the obverved data, a function that generates data sampled under under the null hypothesis, and a function that computes the test statistic. `monte_carlo_test` returns an object with the observed statistic value, an empirical null distribution of the statistic, and the corresponding $p$-value." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "4869f8dd-4d59-432e-80be-b6ec321aef26", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Observed values of the statistic: 6.982848237344646\n", + "Monte Carlo p-value: 0.0076\n", + "Empirical null distribution shape: (10000,)\n" + ] + } + ], + "source": [ + "res = stats.monte_carlo_test(sample=x, rvs=norm_dist.rvs, statistic=statistic, \n", + " n_resamples=10000, alternative='greater')\n", + "print(f\"Observed values of the statistic: {res.statistic}\")\n", + "print(f\"Monte Carlo p-value: {res.pvalue}\")\n", + "print(f\"Empirical null distribution shape: {res.null_distribution.shape}\")" + ] + }, + { + "cell_type": "markdown", + "id": "46763f1a-a696-41b9-ace3-9124760b7e65", + "metadata": {}, + "source": [ + "Note that the $p$-value here is slightly different than `p_mc` above because the algorithm is stochastic. Nevertheless, `res.pvalue` is much closer to `p_mc` than to `p_asyptotic`: for small samples, the Monte Carlo null distribution generated from a sufficiently large number of random samples is often more accurate than the asymptotic null distribution, despite the error inherent in random sampling.\n", + "\n", + "Here, we also passed optional arguments to control the behavior of `monte_carlo_test`. As one might expect, the parameter `n_resamples` controls the number of samples to draw from the provided *random variate sample* function. Perhaps less obvious is the meaning of `alternative`, which controls which *alternative hypothesis* we are testing, that is, which tail of the null distribution the observed statistic value should be compared against. In this case, we are assessing the null hypothesis that the sample was drawn from a normal population against the alternative that the sample is drawn from a population which tends to produce a *greater* value of the test statistic. Perhaps more intuitively, the argument `'greater'` corresponds with the sign of the comparison against the null distribution (i.e., `null_dist_emperical >= statistic(x)` from above). Another options for `alternative` is `'less'` (i.e., `null_dist_emperical <= statistic(x)`). In some cases, a `'two-sided'` alternative is desired, which is twice the minimum of the \"one-sided\" $p$-values. (More on the choice of this convention below.)\n", + "\n", + "We can improve performance of `monte_carlo_test` by ensuring that our test statistic is \"vectorized\". That is, instead of requiring a one-dimensional sample array as input, the statistic should accept an $n$-dimensional array of samples in which each *axis-slice* (e.g. row, column) is a distinct sample. Our `statistic` function is already vectorized in some sense. Above, we wrote `null_dist_emperical = statistic(y)`, and `statistic` computed the statistic for each column of the two-dimensional `y`." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "6d0f8f55-5497-428c-a54e-8664c1cfd6d0", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(11, 10000)\n", + "(10000,)\n" + ] + } + ], + "source": [ + "print(y.shape) # 10000 samples, each with 11 observations\n", + "print(statistic(y).shape) # statistic for each column of y" + ] + }, + { + "cell_type": "markdown", + "id": "0c4da9ef-f7b2-4540-a195-b322dd089692", + "metadata": {}, + "source": [ + "However, `monte_carlo_test` requires that the statistic function accept an `axis` argument to compute the statistic along *any* axis. Only minor modifications are required:" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "b7d6797e-7dab-46ac-a097-ec41716cd8b8", + "metadata": {}, + "outputs": [], + "source": [ + "def statistic_vectorized(x, axis=0):\n", + " # Calculates the Jarque-Bera Statistic\n", + " # Compare against https://github.com/scipy/scipy/blob/4cf21e753cf937d1c6c2d2a0e372fbc1dbbeea81/scipy/stats/_stats_py.py#L1583-L1637\n", + " n = x.shape[axis]\n", + " mu = np.mean(x, axis=axis, keepdims=True)\n", + " x = x - mu # remove the mean from the data\n", + " s = stats.skew(x, axis=axis) # calculate the sample skewness\n", + " k = stats.kurtosis(x, axis=axis) # calculate the sample kurtosis\n", + " statistic = n/6 * (s**2 + k**2/4)\n", + " return statistic\n", + "\n", + "np.testing.assert_allclose(statistic_vectorized(y, axis=0), statistic_vectorized(y.T, axis=1))" + ] + }, + { + "cell_type": "markdown", + "id": "9f5001f8-c828-42f2-848c-8d79157bd159", + "metadata": {}, + "source": [ + "But `monte_carlo_test` becomes much faster:\n" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "198b8734-c516-4394-af01-614712138c10", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "7.15 s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)\n" + ] + } + ], + "source": [ + "# Before\n", + "%timeit -r1 -n1 stats.monte_carlo_test(sample=x, rvs=norm_dist.rvs, statistic=statistic, n_resamples=10000, alternative='greater')" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "20212a36-30ab-4486-81d8-fbe8683ecaa6", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "10.8 ms ± 583 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n" + ] + } + ], + "source": [ + "# After\n", + "%timeit stats.monte_carlo_test(sample=x, rvs=norm_dist.rvs, statistic=statistic_vectorized, n_resamples=10000, alternative='greater', vectorized=True)" + ] + }, + { + "cell_type": "markdown", + "id": "f376518b-23b7-45bf-9921-4f717378f0aa", + "metadata": {}, + "source": [ + "When a statistical test is already implemented in SciPy (like `stats.jarque_bera`), it becomes even easier to perform a Monte Carlo version of the test. We simply need to \"wrap\" it in another function which only returns the test statistic rather than the full result object." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "4fc52618-5476-4042-9bea-8b463c2767f3", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.008" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def statistic_scipy(x):\n", + " # `jarque_bera` returns a result obeject\n", + " res = stats.jarque_bera(x) \n", + " # Our wrapper returns only the statistic, as required by `monte_carlo_test`\n", + " return res.statistic\n", + "\n", + "res = stats.monte_carlo_test(sample=x, rvs=norm_dist.rvs, statistic=statistic_scipy, n_resamples=10000, alternative='greater')\n", + "res.pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "0ac75775-40e2-44c7-882b-009b2adef689", + "metadata": {}, + "source": [ + "Of course, besides enabling more accurate tests for small sample sizes, `monte_carlo_test` makes it easy to perform hypothesis tests *not* implemented in SciPy. For instance, suppose we want to assess whether our data is distributed according to a [`rayleigh` distribution](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rayleigh). Just as the \"normal distribution\" is really a *family* of distributions parameterized by mean and standard deviations, so `rayleigh` is a family of distributions rather than one specific distribution. In contrast with the normal distribution, however, there are no tests in SciPy specifically designed to determine whether a sample is drawn from a Rayleigh distribution. \n", + "\n", + "The closest options are tests like [`ks_1samp`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_1samp.html) and [`cramervonmises`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.cramervonmises.html), which can assess whether a sample is drawn from a *specific* distribution. If we want to use these tests to assess whether the sample is distributed according to *any* Rayleigh distribution, one approach is to fit a Rayleigh distribution to the data, and then apply `ks_1samp` and `cramervonmises` to test whether the data were drawn from the fitted Rayleigh distribution." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "7ecf4ac3-3430-4508-b42e-625e3d7f08cc", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABHpElEQVR4nO3dd3hUVfrA8e+bkEIPLUhP6BAglNCUJjZAFMvae2OxrLru+hN0Xcuq6+ruolgXFUUXxYIFV7CACFIlQXpNAkggQEgglPTk/f0xNziGlAnJZDLJ+3meeWbm3nPuvMfgvHPvOfccUVWMMcaYyhDg6wCMMcbUHJZUjDHGVBpLKsYYYyqNJRVjjDGVxpKKMcaYSlPH1wH4UvPmzTUiIsLXYRhjjF+Ji4s7pKotittXq5NKREQEsbGxvg7DGGP8iojsLmmfXf4yxhhTaSypGGOMqTSWVIwxxlSaWt2nYoypGrm5uSQlJZGVleXrUEw5hIaG0rZtW4KCgjyuY0nFGON1SUlJNGzYkIiICETE1+EYD6gqqampJCUlERkZ6XE9u/xljPG6rKwsmjVrZgnFj4gIzZo1K/fZpVeTioiMEZFtIhIvIpOL2S8iMs3Zv15E+jvb24nIIhHZIiKbROQ+tzpNReQ7EdnhPDdx2zfFOdY2EbnAm20zxpSPJRT/czp/M69d/hKRQOAV4DwgCVgtInNVdbNbsbFAF+cxGHjNec4D/qSqa0SkIRAnIt85dScDC1X1WSdRTQYeEpGewNVAFNAaWCAiXVU131ttNKcn9Xg2KxPT2JV6gvCGIbQOq0urxqG0alyXusGBvg7PGFMB3uxTGQTEq2oigIjMBiYA7kllAvCuuhZ1WSkiYSLSSlWTgWQAVT0mIluANk7dCcAop/5M4AfgIWf7bFXNBnaKSLwTwwovttF44GhWLj8lprE8IZXlCYfYuv9YiWXD6gXRvmk9LuvXhisHtqNesHX7mcqxf/9+7r//flavXk1ISAgRERG88MILdO3atULH3bVrF+PHj2fjxo3Exsby7rvvMm3atFPKFd5s3bx58xKP9cwzz/Dwww+X6/NHjRpFcnIyoaGhNGjQgBkzZtCtW7eT20NCQsjJyeHcc8/lqaeeIiwsDIDAwEB69+598jiff/45lTHDiDf/j20D7HF7n4TrLKSsMm1wEgqAiEQA/YBVzqaWTtJBVZNFJNztWCuLOdZviMhEYCJA+/bty9UgUz4Hj2bxxJebmb8xmQKFkDoBxEQ04cELujG0UzO6tWzIoePZ7DuSRXJ6JsnpWew7ksnGfUd5/MvNTF2wgxuGdODGMzsQ3jDU180xfkxVufTSS7npppuYPXs2AGvXruXAgQMVTiruYmJiiImJOe36p5NUAGbNmkVMTAzTp0/nwQcfZO7cub/ZnpOTw5QpU5gwYQKLFy8GoG7duqxdu/a0Yy2JN5NKcRfjii4zWWoZEWkAzAHuV9WjlfB5qOp0YDpATEyMLXvpBarKJ3FJ/O1/m8nKK+CO4R0Z1S2cfu3DCA367eWt+iF16NCs/inHiNudxvQlibzyQzzTlyRyWf823D48ks7hDauqGaYGWbRoEUFBQUyaNOnktr59+wJw/PhxJkyYwOHDh8nNzeWpp55iwoQJ7Nq1i7FjxzJs2DCWL19OmzZt+OKLL6hbty5xcXHceuut1KtXj2HDhp085g8//MA///lP/ve//5Gamso111xDSkoKgwYNwn2V3UsuuYQ9e/aQlZXFfffdx8SJE5k8eTKZmZn07duXqKgoZs2axX//+1+mTZtGTk4OgwcP5tVXXyUwsORLxCNGjOCFF144ZXtwcDDPPfccnTt3Zt26dURHR1f8P2oJvJlUkoB2bu/bAvs8LSMiQbgSyixV/dStzIHCS2Qi0go4WI7PM162Jy2Dhz/bwI87DjEwogn/uLwPHVs0KPdxBnRoyn9uaEpiynHeWrqTT+KSmL16DzefGcGUcd0JqWN9L/7qiS83sXlfWb8Ry6dn60Y8dlFUifs3btzIgAEDit0XGhrKZ599RqNGjTh06BBDhgzh4osvBmDHjh188MEHvPHGG1x55ZXMmTOH66+/nltuuYWXXnqJkSNH8uCDDxZ73CeeeIJhw4bx17/+la+++orp06ef3DdjxgyaNm1KZmYmAwcO5PLLL+fZZ5/l5ZdfPnn2sGXLFj788EOWLVtGUFAQd911F7NmzeLGG28ssZ1ffvnlby5puQsMDCQ6OpqtW7cSHR19MoEBREZG8tlnn5V43PLwZlJZDXQRkUhgL65O9GuLlJkL3OP0twwG0p1kIcBbwBZV/XcxdW4CnnWev3Db/r6I/BtXR30X4KfKb5YpTkGB8t7K3fzj660I8OSEKK4f3IGAgIqN+OnYogFPX9qbB87rykvfx/PO8l38/MthXr62P+2a1quc4E2tpqo8/PDDLFmyhICAAPbu3cuBAwcA15dt4RfvgAED2LVrF+np6Rw5coSRI0cCcMMNNzB//vxTjrtkyRI+/dT1e/jCCy+kSZOTA1WZNm3ayS/xPXv2sGPHDpo1a/ab+gsXLiQuLo6BAwcCkJmZSXh4OMW57rrrqFu3LhEREbz00kultrWQ313+UtU8EbkH+AYIBGao6iYRmeTsfx2YB4wD4oEM4Ban+lnADcAGEVnrbHtYVefhSiYfichtwC/AFc7xNonIR7g68/OAu23kV9XIzMnn9ndXsyw+lRFdW/DMpb1o26Ryv/CbNQjh8YujGNKxGQ9+vI7xLy3l31dGc06PlpX6Ocb7Sjuj8JaoqCg++eSTYvfNmjWLlJQU4uLiCAoKIiIi4uS9GSEhISfLBQYGkpmZiap6PNS2uHI//PADCxYsYMWKFdSrV49Ro0YVey+IqnLTTTfx97//vczPKew7KU1+fj4bNmygR48eHsV+urx6n4qqzlPVrqraSVWfdra97iQU1OVuZ39vVY11ti9VVVHVPqra13nMc/alquo5qtrFeU5z+7ynnWN1U9VTfzqYSpedl8/E92JZkZDKM5f2ZuYtAys9obgb0+sM/nfvMNo2qcttM2N5dv5W8vILvPZ5pmYYPXo02dnZvPHGGye3rV69msWLF5Oenk54eDhBQUEsWrSI3btLnNUdgLCwMBo3bszSpUsB1xd6cUaMGHFy3/z58zl8+DAA6enpNGnShHr16rF161ZWrvx1fFFQUBC5ubkAnHPOOXzyySccPOi6wp+WllZmbCXJzc1lypQptGvXjj59+pzWMTxld9Sb05aXX8B9H6zlxx2HePayPlw7uH2V3ODWoVl95tx5JtcObs/rixO49o1VHDhqc0qZkokIn332Gd999x2dOnUiKiqKxx9/nNatW3PdddcRGxtLTEwMs2bNonv37mUe7+233+buu+9m6NCh1K1bt9gyjz32GEuWLKF///58++23J0ebjhkzhry8PPr06cOjjz7KkCFDTtaZOHEiffr04brrrqNnz5489dRTnH/++fTp04fzzjuP5OTkYj+rJNdddx19+vShV69enDhxgi+++KLsShUk7tfYapuYmBi1RbpOT0GB8uAn65mzJolHx/fktmGezw1UmT77OYmHP91I84bBfPT7obRqXPz/4Ma3tmzZ4vXLLsY7ivvbiUicqhZ7vc3OVEy5qSpPfLmJOWuS+OO5XX2WUAAu7deW2ROHcORELte+sYqDdsZijE9ZUjHl9q9vtzNzxW7uGB7Jved09nU4RLcL451bB3LgaBbXvrmKQ8ezfR2SMbWWJRVTLv9ZnMDLi+K5ZlA7Hh7Xo9pMEjigQ1Nm3DyQpMMZXP/mKg6fyPF1SMbUSpZUjMeWbE/h7/O3Mr5PK566pHe1SSiFhnRsxps3DiTx0AlumLGK9MxcX4dkTK1jScV4JD0zl4fmrKdzeAP+eUU0gRW8qdFbhnVpzn+uH8C2/ce4acZPHMuyxGJMVbKkYjzy5JebOXgsm39dEX3K/F3Vzdndw3nl2v5s3JvObTNjycmz+1iMqSqWVEyZvtt8gDlrkrhrVCei24X5OhyPnB91Bv+6Mpqfdqbxt/9tLruCqfGSkpKYMGECXbp0oVOnTtx3333k5Lj63t555x3uueceH0d4qgYNip83LzAw8OTEk9HR0fz73/+moKD0H0+7du3i/fff90aYv2FJxZTq8Ikcpny6gR6tGvGH0V18HU65TOjbht+P6Mh7K3fz0eo9ZVcw1cPSF2Dnkt9u27nEtf00qSqXXXYZl1xyCTt27GD79u0cP36cRx55pEKhliYvL89rxy6ct2vTpk189913zJs3jyeeeKLUOpZUTLXw6BcbSc/M4d9XRhNcx//+uTx4QTeGdW7OXz7fyNo9R3wdjvFEm/7w8c2/JpadS1zv2/Q/7UN+//33hIaGcsstrukFAwMDmTp1KjNmzCAjIwNwTew4ZswYunXrdvIL+sSJE1x44YVER0fTq1cvPvzwQwDi4uIYOXIkAwYM4IILLjh5p/uoUaN4+OGHGTlyJE8//TQREREnzyAyMjJo164dubm5JCQkMGbMGAYMGMDw4cPZunWrq6k7dzJ06FAGDhzIo48+6lHbwsPDmT59Oi+//DKqyq5duxg+fDj9+/enf//+LF++HIDJkyfz448/0rdvX6ZOnVpiuQpT1Vr7GDBggJqSfblur3Z46H/68vc7fB1KhaQdz9aznl2og59eoAePZvk6nFpp8+bN5auQuFj1H5GqC59yPScurtDnv/jii3r//fefsr1v3766bt06ffvtt/WMM87QQ4cOaUZGhkZFRenq1av1k08+0dtvv/1k+SNHjmhOTo4OHTpUDx48qKqqs2fP1ltuuUVVVUeOHKl33nnnyfIXX3yxfv/99yfL3XbbbaqqOnr0aN2+fbuqqq5cuVLPPvtsVVW96KKLdObMmaqq+vLLL2v9+vWLbU9x28PCwnT//v164sQJzczMVFXV7du3a+H33KJFi/TCCy88Wb6kckUV97cDYrWE71X/++lpqsTBY1k8+vlGotuF8fsRHX0dToU0qR/Mf24YwJHMHO5+fw25NgFl9Rc5AmJugyXPuZ4jR1TocFrCzMLu28877zyaNWtG3bp1ueyyy1i6dCm9e/dmwYIFPPTQQ/z44480btyYbdu2sXHjRs477zz69u3LU089RVJS0sljXnXVVb95XXh2M3v2bK666iqOHz/O8uXLueKKK+jbty+///3vT57pLFu2jGuuuQZwTalf3jaCa/LIO+64g969e3PFFVeweXPxfYqelisvWwDcnEJVefjTjWTk5POvK6KpE+j/vz2iWjfmH5f34b7Za3n6qy08fnHVT79uymHnEoh9C0b8n+s5cniFEktUVBRz5sz5zbajR4+yZ88eOnXqRFxc3ClJR0To2rUrcXFxzJs3jylTpnD++edz6aWXEhUVxYoVK4r9rPr1f13J9OKLL2bKlCmkpaURFxfH6NGjOXHiBGFhYSWuZXI6938lJiYSGBhIeHg4TzzxBC1btmTdunUUFBQQGlr8UtxTp071qFx5+f+3hal08zfuZ8GWAzx4QTc6h5d/1cbqakLfNtw2LJJ3lu9iTlxS2RWMbxT2oVzxDox+xPXs3sdyGs455xwyMjJ49913AdfaIn/605+4+eabqVfPtVTDd999R1paGpmZmXz++eecddZZ7Nu3j3r16nH99dfz5z//mTVr1tCtWzdSUlJOJpXc3Fw2bdpU7Oc2aNCAQYMGcd999zF+/HgCAwNp1KgRkZGRfPzxx4DrR9y6desAOOuss5g9ezZQ8pT6RaWkpDBp0iTuueceRIT09HRatWpFQEAA7733Hvn5rmWlGjZsyLFjx07WK6lcRVlSMb+Rl1/AP7/ZRteWDbjlLN9NFOktU8Z2Z0jHpjz82Qa2HzhWdgVT9faucSWSwjOTyBGu93vXnPYhC6e+//jjj+nSpQtdu3YlNDSUZ5555mSZYcOGccMNN9C3b18uv/xyYmJi2LBhA4MGDaJv3748/fTT/OUvfyE4OJhPPvmEhx56iOjoaPr27VtqJ/dVV13Ff//7399cFps1axZvvfUW0dHRREVFnZyS/sUXX+SVV15h4MCBpKenl3hM97Xszz33XM4//3wee+wxAO666y5mzpzJkCFD2L59+8kzpz59+lCnTh2io6OZOnVqieUqyqtT34vIGOBFXCs/vqmqzxbZL87+cbhWfrxZVdc4+2YA44GDqtrLrc6HQDfnbRhwRFX7ikgEsAXY5uxbqaqTSovPpr4/1eyffmHypxuYfsMAzo86w9fheEXKsWzGvLCElo1C+fzus/xyVJu/sanv/Ve1mfpeRAKBV4CxQE/gGhHpWaTYWFxryXcBJgKvue17BxhT9LiqepU6q0ECc4BP3XYn6K8rRZaaUMypsnLzeWHBDvq1D+O8njV3md4WDUP4+2W92Zx8lBcWbPd1OMbUKN78iTYIiFfVRFXNAWYDE4qUmQC864xSWwmEiUgrAFVdAqRRAucs50rgA69EXwu9t2I3+49m8X8XdK92k0VWtvOjzuCqmHa8vjiB2F0l/jMzxpSTN5NKG8D9NuYkZ1t5y5RkOHBAVXe4bYsUkZ9FZLGIDC+ukohMFJFYEYlNSUnx8KNqvqNZubzyQzwjurZgaKdmvg6nSjx6UU/aNKnLAx+t43i29+5+Ni7evNRuvON0/mbeTCrF/dQtGqEnZUpyDb89S0kG2qtqP+AB4H0RaXTKwVWnq2qMqsa0aNHCw4+q+d5cksiRjFz+74JuZReuIRqE1OHfV/Zlz+EMnrL5wbwqNDSU1NRUSyx+RFVJTU0t91Bjb96nkgS0c3vfFth3GmVOISJ1gMuAAYXbVDUbyHZex4lIAtAVsJ74MqQcy+bNpTu5sE8rerVp7OtwqtTAiKZMGtmJ135I4JweLWt0X5IvtW3blqSkJOzqgH8JDQ2lbdu25arjzaSyGugiIpHAXuBq4NoiZeYC94jIbGAwkK6qyR4c+1xgq6qevNlARFoAaaqaLyIdcXX+J1ZCO2q8VxbFk51XwJ/O6+rrUHzij+d25YdtKUyes55+7UfQvEGIr0OqcYKCgoiMrHlD1M2pvHb5S1XzgHuAb3AN9f1IVTeJyCQRKRyZNQ/XF3888AZwV2F9EfkAWAF0E5EkEbnN7fBXc2oH/QhgvYisAz4BJqmq9cCWYU9aBrNW7ebKmLZ0bFFzbnQsj+A6AbxwVV+OZecxec4Gu0RjTAV49T6V6s7uU4EHPlrLV+uT+eHBUbRqXNfX4fjUmz8m8tRXW/jXFdFcPqB8p/zG1CY+uU/FVH/b9h/js5/3ctOZEbU+oQDcelYkAzo04amvNpN2IsfX4Rjjlyyp1GKv/hBP/eA63Dmyk69DqRYCAoS/X9ab49l5PPWVjQYz5nRYUqml9qdn8dX6ZK6MaUeT+sG+Dqfa6NqyIZNGduLTNXtZuuOQr8Mxxu9YUqml/rtyN/mq3HxmhK9DqXbuPrszkc3r8/BnG8jMqZyZW42pLSyp1EJZufnMWrWb83q0pH2zer4Op9oJDQrk6Ut78UtaBtO+31F2BWPMSZZUaqHPf97L4Yxcbh1m9w2U5MxOzbliQFumL0lkS/JRX4djjN+wpFLLqCozlu2kZ6tGDI5s6utwqrWHx/UgrG4Qkz/dQH5B7R16b0x5WFKpZZbFp7L9wHFuOSuixs9EXFFN6gfz14t6sm7PEd5bscvX4RjjFyyp1DJvL9tJ8wbBXBTd2teh+IWLo1szvEtznv9mG/uOZPo6HGOqPUsqtcjOQydYuPUg1w3uQGhQoK/D8QsiwtOX9CZflSe/tHtXjCmLJZVa5J1lOwkODOC6Ie19HYpfad+sHvec3ZmvN+3nxx02y64xpbGkUkukZ+bycVwSF0W3Jrxh+dZHMHD78I50aFaPx+duIievwNfhGFNtWVKpJT5avYeMnHxuOSvC16H4pdCgQB67qCcJKSd4Z/lOX4djTLVlSaUWyMsv4J3luxgU2bTWLcJVmUZ3b8k53cN5ccEODhzN8nU4xlRLllRqgQVbDrD3SCa3nmU3O1bUXy/qSW6+8vd5W3wdijHVkleTioiMEZFtIhIvIpOL2S8iMs3Zv15E+rvtmyEiB0VkY5E6j4vIXhFZ6zzGue2b4hxrm4hc4M22+ZOZy3fTtkldWyq3EnRoVp/fj+zI52v3sSox1dfhGFPteC2piEgg8AowFugJXCMiPYsUG4tr2d8uwETgNbd97wBjSjj8VFXt6zzmOZ/XE9eKkFFOvVedGGq1PWkZrEhM5eqB7QgMsJsdK8NdozrTJqwuj83dRF6+ddob486bZyqDgHhVTVTVHGA2MKFImQnAu+qyEggTkVYAqroEKM9ywBOA2aqarao7cS1RPKjCrfBzc9YkIQKX9reVDCtL3eBA/nJhD7buP8asVb/4OhxjqhVvJpU2wB6390nOtvKWKc49zuWyGSLSpDzHEpGJIhIrIrEpKTX7noOCAmXOmiTO6tScNmG2smNlGtPrDIZ1bs6/vt3GoePZvg7HmGrDm0mluGstRWfl86RMUa8BnYC+QDLwr/IcS1Wnq2qMqsa0aNGijI/ybz/tSmNPWia/s/XWK52I8PjFPcnIyef5r7f5Ohxjqg1vJpUkoJ3b+7bAvtMo8xuqekBV81W1AHiDXy9xlftYNd0ncUk0CKnDBVFn+DqUGqlzeENuOSuCj+L2sHFvuq/DMaZa8GZSWQ10EZFIEQnG1Yk+t0iZucCNziiwIUC6qiaXdtDCPhfHpUDh6LC5wNUiEiIikbg6/3+qjIb4oxPZeczbkMz4Pq2oG1zrxyt4zR/O6ULTesE8+eVmVG16fGPKTCoi8k8RiSrvgVU1D7gH+AbYAnykqptEZJKITHKKzQMScXWqvwHc5fa5HwArgG4ikiQitzm7nhORDSKyHjgb+KPzeZuAj4DNwNfA3apaa9eCnb9xPxk5+Xbpy8sahQbxwPld+WlXGvM37vd1OMb4nJT160pEbgduAeoAbwMfqGqNONePiYnR2NhYX4fhFVdPX8H+9CwW/XmUrZviZfkFyoXTfuR4dh4LHhhpM0CbGk9E4lQ1prh9ZZ6pqOqbqnoWcCMQAawXkfdF5OzKDdNUlj1pGaxMTON3A9paQqkCgQHCX8f3JOlwJm8ttXnBTO3mUZ+KcxNhd+dxCFgHPCAis70YmzlNdm9K1Tuzc3PO79mSVxbFc9DmBTO1mCd9Kv8GtgLjgGdUdYCq/kNVLwL6eTtAUz52b4rvPHJhD3LzC3juGxtibGovT85UNgLRqvp7VS06mqrW37Fe3di9Kb7ToVl9bj0rkk/iklifdMTX4RjjE54kletUNcN9g4gsBKgpHfY1id2b4lv3jO5M8wY2xNjUXiUmFREJFZGmQHMRaSIiTZ1HBNC6yiI0HrN7U3yvYWgQfzq/G7G7D/O/9aXecmVMjVTamcrvgThcnfNrnNdxwBe4Zh821Yzdm1I9XBnTjh6tGvHs/K1k5dbaW6VMLVViUlHVF1U1Evizqka6PaJV9eUqjNF46JO4PUQ0q8eADk3KLmy8pnCI8d4jNsTY1D6lXf4a7bzcKyKXFX1UUXzGQ/vTs1iZmMal/ezelOpgaKdmnNezJa8uiufgMRtibGqP0i5/jXSeLyrmMd7LcZlymr/Rdf3+wj6tyihpqsrD43qQnVfA1O+2+zoUY6pMnZJ2qOpjzvMtVReOOV3zNiTTrWVDOoc38HUoxhHZvD43Do3gneU7uXFoBD1aNfJ1SMZ4nSc3P94nIo2cmYTfFJE1InJ+VQRnPHPgaBaxuw8zrredpVQ3957TmYahQTz1lQ0xNrWDJ/ep3KqqR4HzgXBck0s+69WoTLnM35CMKlzYx+5NqW7C6gVz/7ldWBafyqJtB30djjFe50lSKez1HQe8rarrKH6VReMj8zbsp2vLBnQOb+jrUEwxrh/SgY7N6/PUV1vIzS/wdTjGeJUnSSVORL7FlVS+EZGGgP2fUU0cPJrF6t1pdumrGgsKDGDKuB4kppzg/VW/+DocY7zKk6RyGzAZGOhM1xKM6xKYqQbmb9zvuvRlSaVaO7dHOGd2asYLC7aTnpHr63CM8RpP1lMpAA4APUVkBBAFhHlycBEZIyLbRCReRCYXs19EZJqzf72I9HfbN0NEDorIxiJ1nheRrU75z0QkzNkeISKZIrLWebzuSYz+7qsNyXQJb0CXlnbpqzoTER65sAdHMnN56fsdvg7HGK/xZPTXP4BlwF+AB53Hnz2oF4hrOpexQE/gGhHpWaTYWFxryXcBJgKvue17BxhTzKG/A3qpah9gOzDFbV+CqvZ1HpOKqVujHDyaxepddunLX0S1bswVA9oyc8Uudh064etwjPEKTy5/XQJ0U9VxqnqR87jYg3qDgHhVTVTVHGA2MKFImQnAu+qyEggTkVYAqroESCt6UFX9VlXznLcrgVo70dXXm5xLX3bDo9/48/ndCAoM4O/zt/g6FGO8wpOkkggEncax2wB73N4nOdvKW6Y0twLz3d5HisjPIrJYRIYXV0FEJopIrIjEpqSklOOjqp+v1ifTObwBXe3Sl98IbxTKnSM78c2mA6xISPV1OMZUOk+SSgawVkT+4/R/TBORaR7UK27YcdG7vzwpU/zBRR4B8oBZzqZkoL2q9gMeAN4XkVNuYVbV6aoao6oxLVq08OSjqqWDx7L4yS59+aU7RnSkdeNQnvpqMwUFdkOkqVk8SSpzgb8By/l1+vs4D+olAe3c3rcF9p1GmVOIyE245h+7Tp3blFU1W1VTnddxQALQ1YM4/dI3mw7YqC8/FRoUyENju7Np31HmrEnydTjGVCpPRn/NBD4CVqrqzMKHB8deDXQRkUgRCQauxpWg3M0FbnRGgQ0B0lW11JWNRGQM8BBwsfuKlCLSwhkcgIh0xNX5n+hBnH5p3vpkOrWoT9eWNteXP7o4ujV924Xx/DfbOJGdV3YFY/yEJ6O/LgLWAl877/uKSNHkcAqnM/0e4BtgC/CRqm4SkUkiUjgyax6uL/544A3gLrfP/QBYAXQTkSQRuc3Z9TLQEPiuyNDhEcB6EVkHfAJMUtVTOvprgkPHs1m1M5ULe7eyae79lIjw6PieHDyWzX+W1NjfPqYWKnGWYjeP4xrJ9QOAqq4VkUhPDq6q83AlDvdtr7u9VuDuEupeU8L2ziVsnwPM8SQuf/f1xv0UKIyzUV9+bUCHJozv04rpSxK4ZlA7WjWu6+uQjKkwT/pU8lQ1vcg26130oXkbkunYoj7dbNSX35s8tjsFCs99vc3XoRhTKTxJKhtF5FogUES6iMhLuDrtjQ+kHs9mZaJd+qop2japx+3DIvns572s23PE1+EYU2GeJJU/4JqaJRv4ADgK3O/FmEwpFm49SIHCBVE2zX1NcdfZnWneIJi//c/WXDH+z5PRXxmq+oiqDnTu73hEVW3RbR9ZuOUArRqHEtXaVhGsKRqE1OFP53cjdvdh5m3Y7+twjKmQUpOKiNzkrPR4wnnEisiNVRWc+a2s3Hx+3HGI0d3D7dJXDXNlTDt6tGrEM/O2kJWb7+twjDltJSYVJ3ncD/wJaI1r+pT/A+6zxOIbKxNTycjJ59weLX0diqlkgQHCX8f3ZO+RTN6wIcbGj5V2pnIXcKmqLlLVdFU9oqrfA5fjdj+JqToLtxykblAgQzs183UoxguGdmrG2F5n8OoPCexPtyvMxj+VllQaqequohudbXZBv4qpKgu3HGB4l+aEBgX6OhzjJQ+P60G+Kv/4equvQzHmtJSWVDJPc5/xgs3JR9mXnmWXvmq4dk3rccdw1xDjNb8c9nU4xpRbaUmlh7O6YtHHBqB7VQVoXBZuOYgInN093NehGC+7a1RnwhuG8MSXNoux8T+lTdPSo8qiMGVauOUAfduF0aJhiK9DMV5WP6QOD43pzp8+Xsfna/dyWf9auw6d8UMlnqmo6u7SHlUZZG134GgW65LS7dJXLXJpvzZEtwvj2flbbRZj41c8uaPe+Nj3Ww8CcE4Pu/RVWwQECI9d5JrF+LUfEnwdjjEes6TiBxZuOUCbsLo2gWQt0799Ey7t14bpPyayJy2j7ArGVAOerKcyXkQs+fhIZk4+S+MPcW4Pu4u+NnpoTHcCRXjqq82+DsUYj3iSLK4GdojIcyJSrs57ERkjIttEJF5EJhezX5w17+OdkWX93fbNEJGDIrKxSJ2mIvKdiOxwnpu47ZviHGubiFxQnlirq2Xxh8jKLeDcntafUhud0TiUe0Z35ptNB1iyPcXX4RhTJk8mlLwe6Idrzfe3RWSFiEwUkVKvxThL+74CjAV6AteISM8ixcbiWva3CzAReM1t3zvAmGIOPRlYqKpdgIXOe5xjX41rRuUxwKuFywv7s4VbD9AgpA6DI+0u+trq9uGRRDSrx+NfbiInr8DX4RhTKo8ua6nqUVyrKs4GWgGXAmtE5A+lVBsExKtqoqrmOHUnFCkzAXhXXVYCYSLSyvnMJUBxywFPAGY6r2cCl7htn62q2aq6E9cSxYM8aV91VVCgLNxykBFdmxNcx65A1lYhdQJ57OIoElNOMGPZTl+HY0ypPOlTuVhEPgO+B4KAQao6FogG/lxK1TbAHrf3Sc628pYpqqWqJgM4z4VDok7nWNXaxn3pHDyWzTnd7dJXbXd2t3DO7dGSaQt3kJxuE1qY6suTn7+/A6aqah9VfV5VD4JrnRXg1lLqFderXPT2YE/KeMqjYzmX7mJFJDYlpXpfo16w5SABdhe9cTx2UU/yCpRn5tm8YKb68iSpJDuXok4SkX8AqOrCUuolAe3c3rcF9p1GmaIOFF4ic54PludYqjrdWWwspkWLFmV8lG8t2HyAAR2a0LR+sK9DMdVAu6b1uHNkJ75ct48VCam+DseYYnmSVM4rZttYD+qtBrqISKSIBOPqRJ9bpMxc4EZnFNgQIL3w0lYp5gI3Oa9vAr5w2361iISISCSuzv+fPIizWtp3JJPNyUc5x+6iN27uHNWJtk3q8tjcjeTmW6e9qX5KW6TrzsLJI4tMKLkTWF/WgVU1D7gH+AbYAnykqptEZJKITHKKzQMScXWqv4HbOi0i8gGwAugmIkkicpuz61ngPBHZgSvhPet83ibgI2Az8DVwt6r67RJ6i7Y5d9HbpS/jJjQokL+O78n2A8d5d4XNlmSqH1EtvgtDRBoDTYC/4wzbdRxT1eJGZfmdmJgYjY2N9XUYxZr4biyb9h1l6UNn202P5jdUlZvfXs2a3YdZ+OeRhDcM9XVIppYRkThVjSluX2mXv9RZkOtu4JjbAxFpWtlBml/l5BWwPCGVkd1aWEIxpxARHr84iuy8Ap61TntTzZSWVN53nuOAWOc5zu298ZI1vxzmeHYeI7pU74EExncim9fnjhGRfPrzXpYnHPJ1OMacVNrU9+Od50hV7eg8Fz46Vl2Itc+S7SnUCRDO7Gx30ZuS/WF0F9o3rcdfPttIdp7fdh+aGqa0jvr+pT2qMsjaZvH2FPp3aEKj0CBfh2KqsdCgQP52SS8SD52w6fFNtVHayo//KmWfAqMrORYDHDyWxaZ9R3nwgm6+DsX4gZFdW3BRdGteXZTARdGt6dSiga9DMrVciUlFVc+uykCMy4/bXdfHR3a1/hTjmUfH9+CHbQf5y2cbef+OwTa4w/hUaZe/RjvPlxX3qLoQa5clO1Jo3iCYnq0a+ToU4yfCG4by0JjurEhM5dM1e30djqnlSrv8NRLXJJIXFbNPgU+9ElEtll+gLNmewtndwgkIsF+bxnPXDmrPnDVJPD1vC6O7h9PEpvYxPlLa5a/HnOdbqi6c2m3j3nQOZ+Qywi59mXIKCBCeubQ3419ayt/nb+G530X7OiRTS3ky9X0zZ3XGNSISJyIvioiNdfWCJdtTEIHhXZr7OhTjh3q0asTtwyP5KDaJVYk24aTxDU8mlJwNpACX45oGPwX40JtB1VaLt6fQu01jmjUI8XUoxk/dd04X2oTV5eHPNti9K8YnPEkqTVX1b6q603k8BYR5Oa5aJz0jlzW/HLa76E2F1Auuw9OX9iIh5QTTFu7wdTimFvIkqSwSkatFJMB5XAl85e3AaptlCYcoUBjZzZKKqZhR3cK5vH9bXl+cyIakdF+HY2qZ0oYUHxORo8Dvcc0DluM8ZgN/rJrwao8l21NoGFqHfu3CfB2KqQH+Or4nzeoH8+An68jJs3VXTNUpbe6vhqrayHkOUNU6ziNAVe0mikqkqizensKwzs2pE+jJyaMxpWtcL4hnLu3N1v3HeHlRvK/DMbWIR99gItJERAaJyIjCh7cDq012HDxOcnqWDSU2lercni25pG9rXl0Uz6Z9dhnMVA1PhhTfDizBtYLjE87z454cXETGiMg2EYkXkcnF7BdnuHK8s6pk/7LqisiHIrLWeewSkbXO9ggRyXTb97onMVYHi7elAFhSMZXusYuiCKsXzIMfr7flh02V8ORM5T5gILDbmQ+sH65hxaUSkUDgFVzr2fcErhGRnkWKjcW1lnwXYCLwWll1VfUqVe2rqn2BOfz2zv6Ewn2qOgk/sWRHCl3CG9AmrK6vQzE1TJP6wTx1SRSbk4/yus1kbKqAJ0klS1WzAEQkRFW3Ap5MoTsIiFfVRFUt7OCfUKTMBOBddVkJhIlIK0/qimvWvCuBDzyIpdrKyMljVWKanaUYrxnTqxUX9mnFtO93sG3/MV+HY2o4T5JKkoiEAZ8D34nIF8A+D+q1Afa4H8fZ5kkZT+oOBw6oqvtg/EgR+VlEFovI8OKCEpGJIhIrIrEpKWWecHndqsQ0cvILbFZi41VPXhxFw9AgHvxkHXl2Gcx4UZlJRVUvVdUjqvo48CjwFnCJB8cubkZE9bCMJ3Wv4bdnKclAe1XtBzwAvC8ip4xSU9XpqhqjqjEtWvj+i3zx9hRC6gQwKLKpr0MxNVizBiE8OSGK9UnpvLLILoMZ7/F09Fd/EbkX6AMkOZekypIEtHN735ZTz3BKKlNqXRGpA1yG23QxqpqtqqnO6zggAejqQZw+tSz+EIMimxIaFOjrUEwNN75Pay7p25pp3+9gzS+HfR2OqaE8Gf31V2Am0AxoDrwtIn/x4NirgS4iEikiwcDVwNwiZeYCNzqjwIYA6aqa7EHdc4GtqprkFmcLp4MfEemIq/M/0YM4febA0Sx2HDzOsM42gaSpGk9e0oszGoXyxw/Xcjw7z9fhmBrIkzOVa4CBqvqYMx3+EOC6siqpah5wD64hyFuAj1R1k4hMEpHCkVnzcH3xxwNvAHeVVtft8Fdzagf9CGC9iKwDPgEmqWqaB+3zmWXxrlUez7KkYqpIo9Agpl7Vlz1pGTz55aayKxhTTqUt0lVoFxAKZDnvQ3BdWiqTqs7DlTjct73u9lqBuz2t67bv5mK2zcE1xNhvLI0/RJN6QbbKo6lSgyKbcueoTryyKIGzu4UztncrX4dkapASk4qIvISrczwb2CQi3znvzwOWVk14NZeqsiz+EGd2bm6rPJoqd/+5XflxxyEmf7qBfu2bcEbjUF+HZGqI0i5/xQJxwGfAw8Ai4AfgEWC+1yOr4RJSjnPgaLb1pxifCAoM4IWr+pKTV8CfPl5LQUHRwZXGnJ7SlhOeWfja6SwvHEm1TVVzvR1YTbd0h6s/xZKK8ZWOLRrw14t6MuXTDcxYtpPbh3f0dUimBvBk9NcoYAeuaVNeBbbbhJIVtywhlfZN69GuaT1fh2JqsasHtuO8ni157uttNumkqRSejP76F3C+qo5U1RHABcBU74ZVs+XlF7AyIdVGfRmfExH+cXkfmtQP4u5ZaziaZRchTMV4klSCVHVb4RtV3Q4EeS+kmm/93nSOZefZpS9TLTStH8zL1/Znz+FM/u/j9bgGZRpzejxJKnEi8paIjHIeb+DqwDenadmOQ4jA0E7NfB2KMQAMjGjK5DHd+XrTft5autPX4Rg/5klSmQRsAu7FNQ3+ZmebOU1L4w8R1boRTesH+zoUY066fXgk5/dsybPztxK7q1rfN2yqsVKTiogEAHGq+m9VvcyZXHKqqmZXUXw1TkZOHmt+OWz9KabaERGevyKaNk3qcs/7P3PouP1vbsqv1KSiqgXAOhFpX0Xx1Hg/7UwjN1+tP8VUS43rBvHqdf05nJHD/bPXkm/3r5hy8uTyVytcd9QvFJG5hQ9vB1ZTLYs/RHCdAAZG2FT3pnqKat2Yv03oxdL4Q7y4cEfZFYxx48ncX094PYpaZGl8KjEdmthU96Zau3JgO1bvSuOl73fQr30YZ3cL93VIxk+UeKYiIqEicj9wBdAdWKaqiwsfVRVgTXLoeDZbko9af4rxC09O6EX3Mxpx7wc/E3/wuK/DMX6itMtfM4EYYAMwFtdNkKYCliekAjY1i/EPdYMDeePGAYTUCeC2mas5fMKTtflMbVdaUumpqter6n+A3+FaE95UwLIdh2gUWodebRr7OhRjPNK2ST3+c0MMyelZTPpvHDl5tr69KV1pSeXkfA3OolmmAlSVpfGHOLNTcwJtqnvjRwZ0aMJzl/dh1c40Hv18o91xb0pVWlKJFpGjzuMY0KfwtYgc9eTgIjJGRLaJSLyITC5mv4jINGf/ehHpX1ZdEXlcRPaKyFrnMc5t3xSn/DYRucCz/wRVY3dqBnuPZHJWF7v0ZfzPJf3a8IfRnfkwdg9v/mh33JuSlTb1fYWGJznrxb+Ca1GvJGC1iMxV1c1uxcbiWku+CzAYeA0Y7EHdqar6zyKf1xPXMsNRQGtggYh0VdX8irSjsiyNt6nujX/747ldSUg5zjPzt9CxRX3O6dHS1yGZasiT+1RO1yAgXlUTVTUHmA1MKFJmAvCuuqwEwkSklYd1i5oAzFbVbFXdiWvd+0GV2aCKWJ5wiDZhdYloZlPdG/8UECD864q+9GrdmHs/+JktyR5dsDC1jDeTShtgj9v7JGebJ2XKqnuPc7lshog0KcfnISITRSRWRGJTUlLK057TVlCgLE9IZWinZohYf4rxX64RYTE0CK3Dre+sZu+RTF+HZKoZbyaV4r49i/bwlVSmtLqvAZ2AvkAyvw519uTzUNXpqhqjqjEtWrQopkrl27L/KEcycjnTZiU2NcAZjUOZcfNAjmflccNbq0i1OcKMG28mlSSgndv7tsA+D8uUWFdVD6hqvjMv2Rv8eonLk8/ziRXO/Sk21b2pKaJaN+atmwey93AmN7+9mmO2uJdxeDOprAa6iEiks8b91UDROcPmAjc6o8CGAOmqmlxaXafPpdClwEa3Y10tIiEiEomr8/8nbzWuPFYkpNKxeX1aNa7r61CMqTSDIpvy2vX92ZJ8lDvejSUrt1qMiTE+5rWk4tzbcg/wDbAF+EhVN4nIJBEpXI9lHpCIq1P9DeCu0uo6dZ4TkQ0ish44G/ijU2cT8BGu9V6+Bu6uDiO/8vILWLUzjSF2lmJqoNHdW/LPK6JZmZjGPe//TF6+3RxZ20ltvpEpJiZGY2NjvfoZa/cc4ZJXlvHytf0Y36e1Vz/LGF+ZuXwXj83dxOX92/L87/oQYDf41mgiEqeqMcXt82SWYlMByxNc96cM6WhnKqbmuunMCI5k5DJ1wXYa1w3i0fE9bKRjLWVJxctWJKTSrWVDmjcI8XUoxnjVved05khmDjOW7SRA4JELLbHURpZUvCgnr4DVu9K4eqAtnGlqPhHhr+N7ogpvLt1Jdl4BT1wcZZfCahlLKl60ds8RsnILbCixqTVEhMcu6klIUAD/WZxITl4Bz1zW2yZRrUUsqXjR8oRDiMCQSEsqpvYQESaP6U5InUCmLdxBTn4Bz/+uD3UCvXkHg6kuLKl40YqEVHq1bkzjekG+DsWYKiUiPHBeV0LqBPD8N9vIySvghav7EmSJpcazpOIlWbn5/PzLEW4+K8LXoRjjM3ef3ZmQOgE89dUWsvMKePnafoQGVWgCdFPN2c8GL4nbfZicfOtPMeb24R3524QoFmw5wPVvriLNliWu0SypeMnyhEPUCRAGRjT1dSjG+NwNQyN45dr+rN+bzmWvLiMx5bivQzJeYknFS5YnpNKnbWMahNgVRmMALuzTig/uGMKxrDwue205P+1M83VIxgssqXjB8ew81ielc2YnW+XRGHcDOjThs7vOomn9YK5/cxWf/7zX1yGZSmZJxQtW70wjv0CtP8WYYrRvVo9P7zyTfu3DuP/Dtby4YAe1eQ7CmsaSihcsTzhEcGAAAzo0KbuwMbVQWL1g3rttMJf1b8PUBdu55/2fbU2WGsKSihesSEylf4cwGzppTCmC6wTwryuieWhMd77etJ+LX17G5n227r2/s6RSyY5k5LBp31GGdrT+FGPKIiLcOaoT798+mBPZeVz66jI+XP2LXQ7zY15NKiIyRkS2iUi8iEwuZr+IyDRn/3oR6V9WXRF5XkS2OuU/E5EwZ3uEiGSKyFrn8bo321aSlYlpqMKZna0/xRhPDe7YjHn3DScmogkPzdnAnz9eT0ZOnq/DMqfBa0lFRAKBV4CxQE/gGhHpWaTYWFzL/nYBJgKveVD3O6CXqvYBtgNT3I6XoKp9ncckfGBlYip1gwKJbhvmi483xm81bxDCu7cO5r5zuvDpz0lc8soy4g8e83VYppy8eaYyCIhX1URVzQFmAxOKlJkAvKsuK4EwZw36Euuq6rfOcsMAK4G2XmxDuS1POERMRBOC69iVRWPKKzBA+ON5XXn31kGkHs9h3LSl/GdxAvkFdjnMX3jzm68NsMftfZKzzZMyntQFuBWY7/Y+UkR+FpHFIjK8uKBEZKKIxIpIbEpKimct8dCh49lsP3DchhIbU0HDu7Rg/v3DObtbC/4+fyuXvbacHQfsrMUfeDOpFLeAQtGfGyWVKbOuiDwC5AGznE3JQHtV7Qc8ALwvIo1OOYjqdFWNUdWYFi1alNGE8lmZmArAUFs62JgKC28YyuvXD+Cla/rxS+oJLpy2lFd/iCcvv8DXoZlSeDOpJAHt3N63BfZ5WKbUuiJyEzAeuE6dYSKqmq2qqc7rOCAB6FopLfHQysRU6gcH0rtN46r8WGNqLBHhoujWfPfASM7tGc5zX2/jsteWs22/nbVUV95MKquBLiISKSLBwNXA3CJl5gI3OqPAhgDpqppcWl0RGQM8BFysqhmFBxKRFk4HPyLSEVfnf6IX23eKFQmpDIxsaosRGVPJmjcI4dXrBvDKtf3ZeziTcdN+5PG5mziSYTMeVzdem+1QVfNE5B7gGyAQmKGqm0RkkrP/dWAeMA6IBzKAW0qr6xz6ZSAE+E5EAFY6I71GAE+KSB6QD0xS1Sqbse7gsSwSUk5wZUy7sgsbY07LhX1aMbRTM/757TbeXbGLz9fu5YHzunLtoPb2Y66akNp8k1FMTIzGxsZWyrHmrtvHvR/8zBd3n0V0u7BKOaYxpmRbko/y5JebWZGYSteWDXh0fE+Gd6ncflJTPBGJU9WY4vZZaq8kKxNTaRhSh6jWp4wNMMZ4QY9WjXj/jsG8fv0AsnILuOGtn7h95mqb6sXHLKlUkpXWn2JMlRMRxvQ6g2//OIKHxnRnVWIa46b9yB3vxrJxb7qvw6uV7BuwEhw4mkXioRM2lNgYHwkNCuTOUZ1YOnk095/bhVWJqYx/aSm3z1zN+qQjvg6vVrGkUgkK708ZYknFGJ9qXDeI+8/tytLJo/nTeV1ZveswF7+8jFve/onl8YdsosoqYGvdVoKViak0DK1DT+tPMaZaaBQaxB/O6cLNZ0Xw3srdvPnjTq59cxVdwhtw49AOXNq/rS317SV2plIJViSkMjiyKYEBxU0EYIzxlYahQdw1qjPLJ4/mn1dEExoUyKNfbGLIMwt5fO4mElKO+zrEGsdSdQUlp2eyKzWD64d08HUoxpgShAYF8rsBbbm8fxvW7jnCuyt28/6qX3hn+S4GRTRlQr/WXNi7FWH1gn0dqt+zpFJBqxJd91daf4ox1Z+I0K99E/q1b8LD43rwUewePl2TxCOfbeTxuZsY1S2cS/u1YXT3cFu59TRZUqmgFQmpNK4bRM9W1p9ijD9p0TCEu8/uzF2jOrFp31E+/3kvc9ft47vNB2gYUofzerbknB4tGdG1OQ1Dg3wdrt+wpFJBK3emMiiyKQHWn2KMXxIRerVpTK82jZkyrgcrElL5fO1eFmw5wKc/7yUoUBgc2YxzeoRzbo+WtGtaz9chV2uWVCpg35FMdqdmcNPQCF+HYoypBIEBwrAuzRnWpTl5+QWs+eUIC7ccYMGWAzzx5Wae+HIzncMbMLRjM4Z0bMbgjk1p3iDE12FXK5ZUKsDuTzGm5qoTGMCgyKYMimzKlHE92HXoBAu3HmTJ9hQ+XZPEeyt3A9C1ZQOGdGzGoMim9G0XRpuwujiT3dZKllQqYEVCKmH1guh+RkNfh2KM8bKI5vW5bVgktw2LJDe/gI1701mZmMbKxFQ+iUvi3RWuJNOsfjB92jamT9swotu5nmvT2YwllQpYudN1f4r1pxhTuwQFBpwcRXbnqE7k5hewed9R1icdYV1SOuuTjvDD9hQKb+Bv0TCEri0b0LVlw5OPLi0b0KgGDgCwpHKakg5nsCctk9vOivR1KMYYHwsKDCC6XRjR7cK4wdl2IjuPjXvT2bA3nW37j7H9wDFm/7SHzNz8k/VaNAwholk9OjSrT0SzerR3nts1qUdYvSC/vIxmSeU0rSy8P6WT9acYY05VP6QOgzs2Y7Bbn2tBgbL3SCbb9h9j24Fj7Dp0gt2pGfy4I4VP4rJ/Uz80KIDWjetyRuNQWjWuS+uwUM5oHErzBiE0bxBCiwYhtGgYQt3g6nU/jVeTirP074u4Vm98U1WfLbJfnP3jcK38eLOqrimtrog0BT4EIoBdwJWqetjZNwW4DdfKj/eq6jfeatuKhFSa1g+ma3gV9qcsfQHa9IfIEb9u27kE9q6BYfdXbv3iyn55Hyhw8Yu/7gdYNg3Outf1uvBYxR23op9fnrZWhuoQQ3WMxZSP298uIEBo17Qe7dJjOTdoDVxx/8limTn5/JKWwc5DJ9h7JJPkI5kkp2exLz2TZfGHOHgsi4Ji5sOsHxxI84YhhNULpkm9IMLqBhFWL5iwekE0qRdMw9A6NAwNcp7r0Cg0iAYhrtfeWKrDa0nFWS/+FeA8IAlYLSJzVXWzW7GxuNaS7wIMBl4DBpdRdzKwUFWfFZHJzvuHRKQnrrXso4DWwAIR6aqq+XjBykQf9Ke06Q8f3wxXvOP6ctm55Nf3lV2/uLIbP3Xt6325a//s61zvRz706+urZ5V83Ip+fnnaWhmqQwzVMRZTPh7+7eoGB9LtjIZ0K2HgT15+ASnHszl0LIdDx7Ndr49nk3Ism9TjORzOyCH1eA4JKcc5ciKXY9l5pYY1ttcZvHb9gEppojtvnqkMAuJVNRFARGYDEwD3pDIBeFdd81GvFJEwEWmF6yykpLoTgFFO/ZnAD8BDzvbZqpoN7BSReCeGFZXdsD1pGew9ksnvR3as7EOXLnKE6x/ixzdDzG0Q+9av/1Aru35xZa+e5dpXuK1QlttiSDt/LPm4Ff388rS1MlSHGKpjLKZ8KulvVycwgFaN69KqcV2PyufmF5CemcuxrDyOZRU+//q6vZdu4vRmUmkD7HF7n4TrbKSsMm3KqNtSVZMBVDVZRMLdjrWymGP9hohMBCYCtG/fvhzN+VV2Xj4XRLXkzE7NT6t+hUSOcP3DXPIcjPi/8n+plKd+SWXdt0Hxr0s6bmV8flWqDjFUx1hM+fjgbxcUGHCy/6UqeXPq++KuCxW9IlhSGU/qns7noarTVTVGVWNatGhRxiGL1zm8If+5IYbO4Q1Oq36F7Fzi+qUz4v9czzuXeK9+cWXdt6163fUo+rq041b086tadYihOsZiyqc2/e1U1SsPYCjwjdv7KcCUImX+A1zj9n4b0Kq0uoVlnNetgG3FHR/4BhhaWowDBgxQv5K4WPUfka7n4t5XZv3iyj7T1vVIXPzb98teOnVfccet6OeXp62VoTrEUB1jMeVTA/92QKyW9N1f0o6KPnBdWksEIoFgYB0QVaTMhcB8XGcZQ4CfyqoLPA9Mdl5PBp5zXkc55UKceolAYGkx+l1S+XFq8V/UP06t/PrFlZ17r+oX9/52f+Ji1fcu//V14bGKO25FP788ba0M1SGG6hiLKZ8a+LcrLamIenHNZhEZB7yAa1jwDFV9WkQmAajq686Q4peBMbiGFN+iqrEl1XW2NwM+AtoDvwBXqGqas+8R4FYgD7hfVeeXFl9MTIzGxsZWapuNMaamE5E4VY0pdp83k0p1Z0nFGGPKr7SkYmvUG2OMqTSWVIwxxlQaSyrGGGMqjSUVY4wxlaZWd9SLSAqw28sf0xw45OXP8AVrl/+pqW2zdlW9Dqpa7N3jtTqpVAURiS1plIQ/s3b5n5raNmtX9WKXv4wxxlQaSyrGGGMqjSUV75vu6wC8xNrlf2pq26xd1Yj1qRhjjKk0dqZijDGm0lhSMcYYU2ksqVSAiMwQkYMisrGYfX8WERWR5m7bpohIvIhsE5ELqjba8impbSLyByf+TSLynNt2v2hbce0Skb4islJE1opIrIgMctvnL+1qJyKLRGSL87e5z9neVES+E5EdznMTtzrVvm2ltOt5EdkqIutF5DMRCXOr47ftctvvv98fJc2Jbw+P1owZAfQHNhbZ3g7XImG7gebOtp78dr2XBMpY76W6tQ04G1gAhDjvw/2tbSW061tgrPN6HPCDH7arFdDfed0Q2O7E/xy/XX/oH/7UtlLadT5Qx9n+j5rSLue9X39/2JlKBajqEiCtmF1Tgf/jt8sZTwBmq2q2qu4E4oFBxdStFkpo253As6qa7ZQ56Gz3m7aV0C4FGjmvGwP7nNf+1K5kVV3jvD4GbAHa4GrDTKfYTOAS57VftK2kdqnqt6qa5xRbCbR1Xvt1u5zdfv39YUmlkonIxcBeVV1XZFcbYI/b+yR+/UfkL7oCw0VklYgsFpGBznZ/b9v9wPMisgf4J66lqcFP2yUiEUA/YBXQUlWTwfVFBoQ7xfyubUXa5e5WXCvIgp+3qyZ8f9TxdQA1iYjUAx7BdWp+yu5itvnbeO46QBNcSz8PBD4SkY74f9vuBP6oqnNE5ErgLeBc/LBdItIAmINr5dOjrsVViy9azLZq27ai7XLb/giulV5nFW4qprpftAtXO/z++8POVCpXJ1zXO9eJyC5cp+RrROQMXL8s2rmVbcuvl1n8RRLwqbr8BBTgmvTO39t2E/Cp8/pjfr2s4FftEpEgXF9Qs1S1sD0HRKSVs78VUHjJ0m/aVkK7EJGbgPHAdep0PODf7aoZ3x++7tTx9wcQQZGOerd9u/i1oy2K33a0JVJNO9pKahswCXjSed0V1+m4+FvbimnXFmCU8/ocIM7f/mbO3+Fd4IUi25/ntx31z/lT20pp1xhgM9CiyHa/bleRMn75/eHzAPz5AXwAJAO5uH5J3FbSPwrn/SO4Rm1swxltVF0fxbUNCAb+C2wE1gCj/a1tJbRrGBDn/E+7Chjgh+0ahutyyHpgrfMYBzQDFgI7nOem/tS2UtoVj+tHTeG212tCu4qU8cvvD5umxRhjTKWxPhVjjDGVxpKKMcaYSmNJxRhjTKWxpGKMMabSWFIxxhhTaSypGOMhEZkqIve7vf9GRN50e/8vEXmglPpPisi5ZXzG4yLy52K2h4nIXaXUq+tMnRMoIhGFszCLyM0i8nIx5YNFZImI2KwaplJZUjHGc8uBMwFEJADXbAJRbvvPBJaVVFlV/6qqC07zs8OAEpMKrvmvPlXVfE8Opqo5uO5bueo04zGmWJZUjPHcMpykgiuZbASOiUgTEQkBegA/i8gA56whzjmbKZwm5R0R+Z3zepyzHshSEZkmIv9z+5yeIvKDiCSKyL3OtmeBTs6aL88XE9t1wBclxN1ORL521uF4zG375049YyqNnfoa4yFV3ScieSLSHldyWYFrptihQDquu6MVeAmYoKopInIV8DSuMwkARCQU+A8wQlV3isgHRT6qO661axoC20TkNVxTrPRS1b5F4xKRYKCjqu4qIfRBQC8gA1gtIl+paiyupDiwhDrGnBZLKsaUT+HZypnAv3EllTNxJZXlQDdcX+DfOTMEB+KaFsZddyBRXetigGvqmIlu+79S15o12SJyEGhZRkzNgSOl7P9OVVMBRORTXFOExKpqvojkiEhDda3pYUyFWVIxpnwK+1V64/qlvwf4E3AUmIFrosBNqjq0lGOUOB+9I9vtdT5l/3+aCYSWsr/oXEzu70OArDKOb4zHrE/FmPJZhmu69TRVzVfVNFyd6ENxXQ7bBrQQkaHgmt5cRKKKHGMr0NFZnAk86yw/huty2ClU9TAQ6FxWK8554lqrvi6ulR+XObE1A1JUNdeDzzfGI5ZUjCmfDbguN60ssi1dVQ85o6p+B/xDRNbhmn32TPcDqGomrpFcX4vIUuAArstnJXIuXy0TkY0ldNR/i+uyVnGWAu85scxx+lPA1W8zr7TPNaa8bJZiY3xARBqo6nFxdby8AuxQ1akVOF4/4AFVvaEcdT4FpqjqttP9XGOKsjMVY3zjDhFZC2wCGuMaDXbaVPVnYJGIBHpS3hkx9rklFFPZ7EzFGGNMpbEzFWOMMZXGkooxxphKY0nFGGNMpbGkYowxptJYUjHGGFNp/h9AUSF6/lgbyAAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "dist_family = stats.rayleigh\n", + "params = dist_family.fit(x)\n", + "dist_specific = dist_family(*params)\n", + "\n", + "z = np.linspace(dist_specific.ppf(0), dist_specific.isf(0.001))\n", + "plt.plot(z, dist_specific.pdf(z))\n", + "plt.plot(x, np.zeros_like(x), 'x')\n", + "plt.legend(('Candidate PDF', 'Observed Data'))\n", + "plt.xlabel('Weight (lb)')\n", + "plt.ylabel('Probability Density')\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "bf116179-6679-4ca4-b938-549886b92b0d", + "metadata": {}, + "source": [ + "To the eyes of the author, this does not look like a terrific fit. The mode of the Rayleigh distribution is too far to the right compared to cluster of observations around 160 lb. Also, according to this Rayleigh distribution, there is zero probability that any weights could be less than ~135 lb, which does not seem realistic. However, the `ks_1samp` and `cramervonmises` tests are both inconclusive, with relatively large $p$-values." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "09860ff5-3a29-4d41-bf9f-d69770ab9aa3", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "KstestResult(statistic=0.26884627441317677, pvalue=0.3412228239139401)" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "stats.ks_1samp(x, dist_specific.cdf)" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "09867af4-cd95-4286-8eb7-3e79e5b97b7a", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "CramerVonMisesResult(statistic=0.17536330558707267, pvalue=0.32395064743536117)" + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "stats.cramervonmises(x, dist_specific.cdf)" + ] + }, + { + "cell_type": "markdown", + "id": "7666ae1d-8c0b-48df-b40d-06050da64edc", + "metadata": {}, + "source": [ + "A much more powerful test of the null hypothesis that the data is distributed according to *any* Rayleigh distribution is the [Anderson-Darling Test](https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test). [`scipy.stats.anderson`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.anderson.html) implements the test for some families of distributions, but not for the Rayleigh distribution. A simple implementation is included below." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "e9eb2df7-4560-4b17-98f5-e9611a9223a5", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.0425042504250425" + ] + }, + "execution_count": 18, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def statistic(x):\n", + " \"\"\"Compute the Anderson-Darling statistic A^2\"\"\"\n", + " # fit a distribution to the data\n", + " params = dist_family.fit(x)\n", + " dist = dist_family(*params)\n", + " \n", + " # compute A^2\n", + " x = np.sort(x)\n", + " n = len(x)\n", + " i = np.arange(1, n+1)\n", + " Si = (2*i - 1)/n * (dist.logcdf(x) + dist.logsf(x[::-1]))\n", + " S = np.sum(Si)\n", + " return -n - S\n", + "\n", + "params = dist_family.fit(x)\n", + "dist = dist_family(*params)\n", + "res = stats.monte_carlo_test(x, rvs=dist.rvs, statistic=statistic, alternative='greater')\n", + "res.pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "db20efd8-6b9f-41d2-a44b-3b26e847ab22", + "metadata": {}, + "source": [ + "Although this does not meet the threshold for significance used above (1%), it does begin to cast doubt on the null hypothesis." + ] + }, + { + "cell_type": "markdown", + "id": "eb114e39-ea07-451d-b6d1-1c965fca311e", + "metadata": {}, + "source": [ + "As we can see, `monte_carlo_test` is a versatile tool for comparing a sample against a distribution by means of an arbitrary statistic. Provided a statistic and null distribution, it can replicate the $p$-value of any such tests in SciPy, and it may be more accurate than these existing implementations, especially for small samples:\n", + "\n", + "- [`skewtest`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.skewtest.html)\n", + "- [`kurtosistest`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kurtosistest.html)\n", + "- [`normaltest`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.normaltest.html)\n", + "- [`shapiro`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.shapiro.html)\n", + "- [`anderson`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.anderson.html)\n", + "- [`chisquare`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chisquare.html)\n", + "- [`power_divergence`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.power_divergence.html)\n", + "- [`cramervonmises`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.cramervonmises.html)\n", + "- [`ks_1samp`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_1samp.html)\n", + "- [`binomtest`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binomtest.html)\n", + "\n", + "In addition, `monte_carlo_test` can be used to perform tests not yet implemented in SciPy, such as [the Lilliefors Test](https://www.tandfonline.com/doi/abs/10.1080/01621459.1967.10482916) for normality.\n", + "\n", + "However, there are other types of statistical tests that do not test whether a sample is drawn from a particular distribution or family of distributions, but instead test whether multiple samples are drawn from the same distribution. For these situations, we turn our attention to [Permutation Tests](https://nbviewer.org/github/mdhaber/scipy/blob/resampling_tutorial/doc/source/tutorial/stats/notebooks/resampling_tutorial_2.ipynb)." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.5" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_2.ipynb b/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_2.ipynb new file mode 100644 index 0000000..c56bb99 --- /dev/null +++ b/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_2.ipynb @@ -0,0 +1,386 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "c2bf7028-42c0-4310-8bf8-e7e5a9bf27fd", + "metadata": {}, + "source": [ + "## Permutation Tests\n", + "### Exact Tests\n", + "Consider the following experiment from [An Introduction to the Bootstrap](https://books.google.com/books?id=MWC1DwAAQBAJ&printsec=frontcoverhttps://books.google.com/books?id=MWC1DwAAQBAJ&printsec=frontcover). A new medical treatment is intended to prolong life after a form of surgery. Sixteen mice are randomly assigned to either a treatment group or control group under the constraint that only seven treatments are available. All mice receive the surgery, but only the treatment group will receive the treatment being studied. The survival time of each mouse after surgery is recorded below." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "823b24e2-d26a-4f79-94cb-cc1eb27c8e5b", + "metadata": {}, + "outputs": [], + "source": [ + "# survival times measured in days\n", + "import numpy as np\n", + "x = np.array([94, 197, 16, 38, 99, 141, 23]) # treatment group\n", + "y = np.array([52, 104, 146, 10, 51, 30, 40, 27, 46]) # control group" + ] + }, + { + "cell_type": "markdown", + "id": "e19f78a9-e1c0-4d8a-aeb6-3fa944752948", + "metadata": {}, + "source": [ + "The difference in the mean life after treatment between the two groups suggests that the treatment has a prolonging effect, as hypothesized." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "5b3b1ead-55f7-456f-b015-9e36d0380c9d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "30.63492063492064\n" + ] + } + ], + "source": [ + "def statistic(x, y):\n", + " return np.mean(x) - np.mean(y)\n", + "print(statistic(x, y))" + ] + }, + { + "cell_type": "markdown", + "id": "9d016372-3ebe-49e5-a70c-923d8e9a1650", + "metadata": {}, + "source": [ + "It is possible that the treatment has no effect in reality; perhaps the apparent prolonging effect of the treatment is due to the inherent variability in survival times and chance alone. This possibility is typically assessed using Student's t-test. A common formulation of the test begins with the null hypothesis that the survival times `x` and `y` are sampled at random from normal distributions $X$ and $Y$ with means $\\mu_x$ and $\\mu_y$ and a common standard deviation, $\\sigma$. To test the null hypothesis that $\\mu_x = \\mu_y$ against the alternative that $\\mu_x > \\mu_y$, we perform the independent sample t-test with `stats.ttest_ind`." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "b993796d-05c7-4b75-bc70-7cd974da76b0", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Ttest_indResult(statistic=1.1208453991208167, pvalue=0.14060629239765005)" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from scipy import stats\n", + "stats.ttest_ind(x, y, equal_var=True, alternative='greater')" + ] + }, + { + "cell_type": "markdown", + "id": "33e2913b-ac57-493a-a6c8-20f11ab9b933", + "metadata": {}, + "source": [ + "The probability of observing such an extreme test statistic under the null hypothesis (due to chance alone) is greater than 14%, so these data do not seem inconsistent with the null hypothesis. The *point estimate* of the statistic (~30 days) suggested a life-prolonging effect, but such a value of the statistic could quite easily have been observed due to chance alone.\n", + "\n", + "Although the t-test tends to be rather robust to violations of its underlying assumptions (e.g., $X$ and $Y$ do not need to be strictly normally distributed for the test to be reasonably accurate), it is possible to perform a hypothesis test which requires no such assumptions at all. \n", + "\n", + "Instead, let the null hypothesis be that the samples `x` and `y` are drawn a single distribution ($X = Y = Z$), and test this against the alternative that the two sample are drawn from distributions which would tend to produce greater values of `statistic`. \n", + "\n", + "The complete population of mice survival times in the study is really:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "8888960d-bb51-4549-a01c-d1e70f880744", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[ 94 197 16 38 99 141 23 52 104 146 10 51 30 40 27 46]\n" + ] + } + ], + "source": [ + "z = np.concatenate([x, y])\n", + "print(z)" + ] + }, + { + "cell_type": "markdown", + "id": "6204af8f-b858-4d5a-8bed-b4ef1bc00c30", + "metadata": {}, + "source": [ + "Since the mice were randomly divided into the two groups under the constraint that there were only seven treatments available, any selection of seven mice from `z` to form the treatment group `x` was equally likely; the remaining mice would form the control group `y`. Furthermore, if the null hypothesis is true, the mice survival times would be *unaffected by the grouping*. Therefore, each value of the statistic obtained from the possible groupings is equaly likely.\n", + "\n", + "We begin our hypothesis test by calculating the value of `statistic` for all possible *permutations*[[2]](#cite_note-2) of mice into the the two groups, forming an exact null distribution.\n", + "\n", + "[[2]](#cite_note-2) Here and below, we will refer to the the ways of rearranging samples as \"permutations\" even when the word is not stricly appropriate in the technical sense. " + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "c7fed236-d06e-478b-b580-cbdd468730eb", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Text(0, 0.5, 'Observed Frequency')" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAd4ElEQVR4nO3dfZxdVX3v8c+XYHiw0EgTaMxDJ9IATX0hhCGhCFWeNEEkYHmUXihY0yjprXq1BPFay+vevlBQbimYGErAeIUAlofUhgJigSvXQAaEkACRIYAMiRDsLU+hYPB3/9hrYOfkPOw9zJ45M/N9v17nNXuvvdY6v3OYzI+99t5rKSIwMzMrarvBDsDMzIYWJw4zMyvFicPMzEpx4jAzs1KcOMzMrJTtBzuAgTB27Njo6OgY7DDMzIaU+++//4WIGFdbPiISR0dHB11dXYMdhpnZkCLp6XrlHqoyM7NSnDjMzKwUJw4zMyvFicPMzEpx4jAzs1KcOMzMrBQnDjMzK8WJw8zMSnHiMDOzUkbEk+Nm/a1jwb/ULX/qgo8NcCRmA89nHGZmVooTh5mZlVJp4pA0S9I6Sd2SFtQ5LkmXpOOrJU3PHVsi6XlJaxr0/UVJIWlslZ/BzMy2VlnikDQKuAyYDUwDTpU0rababGBqes0FFuaOXQXMatD3JOAo4Bf9G7WZmbVS5cXxGUB3RKwHkLQMmAM8kqszB1gaEQGslDRG0viI2BgRd0vqaND3xcBfAzdXF75Zeb5obiNBlUNVE4Bncvs9qaxsna1IOhZ4NiIealFvrqQuSV2bNm0qHrWZmTVVZeJQnbLoQ523K0s7A+cBX2315hGxOCI6I6Jz3LhtFrAyM7M+qnKoqgeYlNufCGzoQ528PYEpwEOSeus/IGlGRPzyHUdsVqPR0JPZSFblGccqYKqkKZJGA6cAy2vqLAdOT3dXHQS8GBEbG3UYEQ9HxO4R0RERHWSJZ7qThpnZwKkscUTEFmA+cCvwKHBdRKyVNE/SvFRtBbAe6AYuBz7b217SNcBPgb0l9Uj6VFWxmplZcZVOORIRK8iSQ75sUW47gLMbtD21QP8d7zBEMzMryU+Om5lZKU4cZmZWihOHmZmV4sRhZmalOHGYmVkpXsjJbAB4DisbTnzGYWZmpThxmJlZKU4cZmZWiq9xmA0iX/uwochnHGZmVooTh5mZleLEYWZmpfgahxlesMmsDJ9xmJlZKU4cZmZWihOHmZmV4sRhZmalOHGYmVkpThxmZlZKpYlD0ixJ6yR1S1pQ57gkXZKOr5Y0PXdsiaTnJa2paXOhpMdS/RsljanyM5iZ2dYqe45D0ijgMuAooAdYJWl5RDySqzYbmJpeM4GF6SfAVcClwNKarm8Hzo2ILZK+DpwLnFPV57Dhxc9rmL1zVT4AOAPojoj1AJKWAXOAfOKYAyyNiABWShojaXxEbIyIuyV11HYaEbfldlcCJ1T2CcwGSbME5wkQbbBVOVQ1AXgmt9+TysrWaeYs4JZ6ByTNldQlqWvTpk0lujQzs2aqTByqUxZ9qFO/c+k8YAvw/XrHI2JxRHRGROe4ceOKdGlmZgVUOVTVA0zK7U8ENvShzjYknQEcAxyRhrnMzGyAVHnGsQqYKmmKpNHAKcDymjrLgdPT3VUHAS9GxMZmnUqaRXYx/NiI2FxF4GZm1lhlZxzprqf5wK3AKGBJRKyVNC8dXwSsAI4GuoHNwJm97SVdA3wYGCupB/ibiLiC7E6rHYDbJQGsjIh5VX0Os3bjVQNtsGkkjPR0dnZGV1fXYIdh74Bvo+07JxTrK0n3R0RnbbmfHDczs1KcOMzMrBSvAGg2zPmaiPU3n3GYmVkpThxmZlaKE4eZmZXixGFmZqW0TByS3j8QgZiZ2dBQ5IxjkaT7JH3WiyaZmVnLxBERhwCnkU1G2CXpaklHVR6ZmZm1pULXOCLiceArZJMLfgi4JC3f+okqgzMzs/ZT5BrHvpIuBh4FDgc+HhF/kLYvrjg+MzNrM0WeHL8UuBz4ckS81lsYERskfaWyyMzMrC0VSRxHA69FxJsAkrYDdoyIzRHxvUqjMzOztlMkcfwIOBJ4Je3vDNwGHFxVUGZWPc9hZX1V5OL4jhHRmzRI2ztXF5KZmbWzIonjVUnTe3ckHQC81qS+mZkNY0WGqj4HXC9pQ9ofD5xcWUQ2YnmVP7OhoWXiiIhVkvYB9gYEPBYRv648MjMza0tFJzk8ENgX2B84VdLpRRpJmiVpnaRuSQvqHJekS9Lx1TVDYkskPS9pTU2b3STdLunx9PM9BT+DmZn1gyIPAH4PuAg4hCyBHAhss3h5nXajgMuA2cA0soQzrababGBqes0FFuaOXQXMqtP1AuCOiJgK3JH2zcxsgBS5xtEJTIuIKNn3DKA7ItYDSFoGzAEeydWZAyxNfa+UNEbS+IjYGBF3S+qo0+8c4MNp+7vAnWRToZiZ2QAoMlS1BvjdPvQ9AXgmt9+TysrWqbVHRGwESD9370NsZmbWR0XOOMYCj0i6D3i9tzAijm3RTnXKas9aitTpE0lzyYa/mDx5cn90aWZmFEscX+tj3z1kU7H3mghs6EOdWs/1DmdJGg88X69SRCwGFgN0dnb2SzIyM7Ni63HcBTwFvCttrwIeKND3KmCqpCmSRgOnAMtr6iwHTk93Vx0EvNg7DNXEcuCMtH0GcHOBWMzMrJ8Uuavq08APgO+kognATa3aRcQWYD5wK9mU7NdFxFpJ8yTNS9VWAOuBbrIZeD+be99rgJ8Ce0vqkfSpdOgC4ChJjwNHpX0zMxsgRYaqzia7Q+peyBZ1klTognRErCBLDvmyRbntSP3Xa3tqg/JfAUcUeX8zM+t/Re6qej0i3ujdkbQ9/XQB28zMhp4iieMuSV8GdkprjV8P/HO1YZmZWbsqkjgWAJuAh4G/IBt68sp/ZmYjVJFJDn9DduH68urDMbPB5gWerJWWiUPSk9S5phER76skIjMza2tF56rqtSNwIrBbNeGYmVm7K/IA4K9yr2cj4n8Bh1cfmpmZtaMiQ1XTc7vbkZ2B7FJZRGZm1taKDFV9M7e9hWz6kZMqicbMzNpekbuqDhuIQMzMbGgoMlT1hWbHI+Jb/ReOmZm1u6J3VR3I2zPbfhy4m60XYDIzsxGi6EJO0yPiZQBJXwOuj4g/rzIwG74aPWBmZkNDkSlHJgNv5PbfADoqicbMzNpekTOO7wH3SbqR7Any44GllUZlZmZtq8hdVf9T0i3AoanozIj4WbVhmZlZuyoyVAWwM/BSRPw90CNpSoUxmZlZGyuydOzfAOcA56aidwH/u8qgzMysfRU54zgeOBZ4FSAiNuApR8zMRqwiieONtDZ4AEh6d7UhmZlZOyuSOK6T9B1gjKRPAz+i4KJOkmZJWiepW9KCOscl6ZJ0fHV+QsVGbSXtJ2mlpAcldUmaUSQWMzPrH03vqpIk4FpgH+AlYG/gqxFxe6uOJY0CLgOOAnqAVZKWR8QjuWqzganpNRNYCMxs0fYbwN9GxC2Sjk77Hy7+kc3M7J1omjgiIiTdFBEHAC2TRY0ZQHdErAeQtAyYA+QTxxxgaRoKWylpjKTxZA8YNmobwK6p/W8DG0rGZWZm70CRBwBXSjowIlaV7HsCW89n1UN2VtGqzoQWbT8H3CrpIrKhtoPrvbmkucBcgMmTJ5cM3czMGilyjeMwsuTxRLoO8bCk1QXaqU5Z7drljeo0a/sZ4PMRMQn4PHBFvTePiMUR0RkRnePGjSsQrpmZFdHwjEPS5Ij4Bdl1iL7oASbl9iey7bBSozqjm7Q9A/irtH098I99jM/MzPqg2RnHTQAR8TTwrYh4Ov8q0PcqYKqkKZJGA6fw9tTsvZYDp6e7qw4CXoyIjS3abgA+lLYPBx4vEIuZmfWTZtc48sNF7yvbcURskTQfuBUYBSyJiLWS5qXji4AVwNFAN7AZOLNZ29T1p4G/l7Q98J+k6xhmZjYwmiWOaLBdWESsIEsO+bJFue0Azi7aNpX/BDigL/GYWd81WkflqQs+NsCR2GBrljg+IOklsjOPndI2aT8iYtfGTc3MbLhqmDgiYtRABmLDj1f6Mxueik6rbmZmBjhxmJlZSU4cZmZWihOHmZmV0uzJ8Zdpchuu76oyMxuZmt1VtQuApPOBXwLfI7sV9zS8AqCZ2YhVZKjqoxHx7Yh4OSJeioiFwJ9UHZiZmbWnIonjTUmnSRolaTtJpwFvVh2YmZm1pyKJ45PAScBz6XViKjMzsxGo5UJOEfEU2ep7Zmbb8BxWI0/LMw5Je0m6Q9KatL+vpK9UH5qZmbWjIkNVlwPnAr8GiIjVZOtjmJnZCFQkcewcEffVlG2pIhgzM2t/RRLHC5L2JD0MKOkEYGOlUZmZWdtqeXGcbKGlxcA+kp4FniR7CNDMzEagIonj6Yg4UtK7ge0i4uWqg7KhxetumI0sRYaqnpS0GDgIeKXieMzMrM0VOePYG/g42ZDVFZJ+CCxLa3+bmdXl5zuGr5ZnHBHxWkRcFxGfAPYHdgXuKtK5pFmS1knqlrSgznFJuiQdXy1pepG2kv4yHVsr6RtFYjEzs/5R5IwDSR8CTgZmA6vIpiBp1WYUcBlwFNADrJK0PCIeyVWbDUxNr5nAQmBms7aSDiN7kn3fiHhd0u7FPqqZmfWHlolD0pPAg8B1wJci4tWCfc8AuiNifepnGdkf/HzimAMsjYgAVkoaI2k80NGk7WeACyLidYCIeL5gPGZm1g+aDlWl//O/MiKOj4hrSiQNgAnAM7n9nlRWpE6ztnsBh0q6V9Jdkg5sEPtcSV2SujZt2lQibDMza6Zp4oiIN4HD+ti36nVZsE6zttsD7yG7y+tLwHWStqkfEYsjojMiOseNG1c8ajMza6rINY7/K+lS4FrgrTOOiHigRbseYFJufyKwoWCd0U3a9gA3pOGt+yT9BhgL+LTCzGwAFEkcB6ef5+fKAji8RbtVwFRJU4BnySZGrF3HYzkwP13DmAm8GBEbJW1q0vam9N53StqLLMm8UOBzmJlZPyiyHkefhqoiYouk+cCtwChgSUSslTQvHV8ErACOBrqBzcCZzdqmrpcAS9I0728AZ6SzDzMzGwBF7qraA/g74L0RMVvSNOCPIuKKVm0jYgVZcsiXLcptB9mDhYXapvI3gD9t9d5mZlaNIkNVVwFXAuel/Z+TXe9omThsePGcVNYf/ET50FdkrqqxEXEd8BvIhpGANyuNyszM2laRxPGqpN/h7fU4DgJerDQqMzNrW0WGqr5AdvfTnpLuAcYBJ1QalZmZta0id1U9kOaq2pvswbx1EfHryiMzM7O21HKoStKJwE7pdtjjgGvzs9iamdnIUuQax3+PiJclHQJ8FPgu2Sy2ZmY2AhVJHL13UH0MWBgRN5M9rW1mZiNQkcTxrKTvkK3BsULSDgXbmZnZMFTkrqqTgFnARRHxH2m9jC9VG5YNJj/oZ2bNFLmrarOkp4DZkmYB90TEbZVHZmYjip8oHzqK3FX1VbIL4r9DNn35lZK+UnVgZmbWnooMVZ0K7B8R/wkg6QLgAeB/VBmYmZm1pyIXuZ8Cdszt7wA8UUk0ZmbW9hqecUj6B7L5qV4H1kq6Pe0fBfxkYMIzM7N202yoqiv9vB+4MVd+Z2XRmJlZ22uYOCLiuwCSdgR+n+xs44neax1mZjYyNRuq2p5s5b+zgKfJrodMlHQlcJ4nOhz6/LyGmfVFs4vjFwK7AVMi4oCI2B/YExgDXDQAsZmZWRtqljiOAT4dES/3FkTES8BngKOLdC5plqR1krolLahzXJIuScdX52fdLdD2i5JC0tgisZiZWf9oljgiIqJO4Zuk1QCbkTQKuAyYDUwDTpU0rababGBqes0lzbrbqq2kSWR3d/2iVRxmZta/mt1V9Yik0yNiab5Q0p8CjxXoewbQHRHrU7tlwBzgkVydOcDSlKBWShqT5sLqaNH2YuCvgZsLxGFmQ5inImk/zRLH2cANks4iuyU3gAOBnYDjC/Q9AXgmt98DzCxQZ0KztpKOBZ6NiIckNXxzSXPJzmKYPHlygXDNzKyIZrfjPgvMlHQ48Idky8beEhF3FOy73l/12iGuRnXqlkvaGTgP+EirN4+IxcBigM7OzpZDa2ZmVkyR2XF/DPy4D333AJNy+xOBDQXrjG5QvicwBeg925gIPCBpRkT8sg8xmplZSVUuyLQKmCppiqTRwCnA8po6y4HT091VBwEvRsTGRm0j4uGI2D0iOiKigyzxTHfSMDMbOEVmx+2TiNgiaT5wKzAKWBIRayXNS8cXASvIbu3tBjYDZzZrW1WsZmZWXGWJAyAiVpAlh3zZotx2kF2EL9S2Tp2Odx7l8OcnxM2sP3ntcDMzK8WJw8zMSnHiMDOzUpw4zMysFCcOMzMrxYnDzMxKceIwM7NSKn2Ow8ysKs2eT/LMudXyGYeZmZXixGFmZqV4qGoY8dQiZjYQfMZhZmalOHGYmVkpThxmZlaKE4eZmZXii+NmNuw0ulHEz3f0D59xmJlZKU4cZmZWihOHmZmV4sRhZmalVJo4JM2StE5St6QFdY5L0iXp+GpJ01u1lXShpMdS/RsljanyM5iZ2dYqSxySRgGXAbOBacCpkqbVVJsNTE2vucDCAm1vB94fEfsCPwfOreozmJnZtqo845gBdEfE+oh4A1gGzKmpMwdYGpmVwBhJ45u1jYjbImJLar8SmFjhZzAzsxpVJo4JwDO5/Z5UVqROkbYAZwG31HtzSXMldUnq2rRpU8nQzcyskSofAFSdsihYp2VbSecBW4Dv13vziFgMLAbo7Oysfd8hzbPgmtlgqjJx9ACTcvsTgQ0F64xu1lbSGcAxwBERMaySgplZu6sycawCpkqaAjwLnAJ8sqbOcmC+pGXATODFiNgoaVOjtpJmAecAH4qIzRXGb2bDjKci6R+VJY6I2CJpPnArMApYEhFrJc1LxxcBK4CjgW5gM3Bms7ap60uBHYDbJQGsjIh5VX0OMzPbWqWTHEbECrLkkC9blNsO4OyibVP57/dzmGZmVoKfHDczs1KcOMzMrBQnDjMzK8WJw8zMSnHiMDOzUrx0rJmNeH6+oxwnjjbmqUXMrB15qMrMzErxGYeZWUkjfWjLicPMrAEPF9fnxNEG/MtpZkOJr3GYmVkpPuMwM+snI+XahxPHAPFwlJkNF04c/cwJwsyGO1/jMDOzUnzGYWZWseF27cNnHGZmVorPOPrI1zLMbKTyGYeZmZVSaeKQNEvSOkndkhbUOS5Jl6TjqyVNb9VW0m6Sbpf0ePr5nio/g5mZba2yoSpJo4DLgKOAHmCVpOUR8Uiu2mxganrNBBYCM1u0XQDcEREXpISyADinqs/hISkzq0p/XTRv9neqigvwVV7jmAF0R8R6AEnLgDlAPnHMAZZGRAArJY2RNB7oaNJ2DvDh1P67wJ1UmDjMzAZau/8Pa5WJYwLwTG6/h+ysolWdCS3a7hERGwEiYqOk3eu9uaS5wNy0+4qkdX35EHWMBV7op74GkuMeWEM1bhi6sTvuOvT1d9T89+oVVpk4VKcsCtYp0rapiFgMLC7TpghJXRHR2d/9Vs1xD6yhGjcM3dgd98Cp8uJ4DzAptz8R2FCwTrO2z6XhLNLP5/sxZjMza6HKxLEKmCppiqTRwCnA8po6y4HT091VBwEvpmGoZm2XA2ek7TOAmyv8DGZmVqOyoaqI2CJpPnArMApYEhFrJc1LxxcBK4CjgW5gM3Bms7ap6wuA6yR9CvgFcGJVn6GBfh/+GiCOe2AN1bhh6MbuuAeIshuazMzMivGT42ZmVooTh5mZleLEUYCkayU9mF5PSXowlXdIei13bNEgh7oNSV+T9GwuxqNzx85NU7qsk/TRwYyzlqQLJT2WpqK5UdKYVD4UvvOmU+20C0mTJP2bpEclrZX0V6m84e9Mu0j/Dh9O8XWlsraejkjS3rnv9EFJL0n63FD4vmv5GkdJkr5JdvfX+ZI6gB9GxPsHOayGJH0NeCUiLqopnwZcQ/aE/3uBHwF7RcSbAx5kHZI+Avw43SjxdYCIOKfdv/M0Xc7PyU2XA5xaM9VOW0i3s4+PiAck7QLcDxwHnESd35l2IukpoDMiXsiVfQP499x0RO+JiLacVSL9njxL9mDzmbT5913LZxwlSBLZP6prBjuWfjAHWBYRr0fEk2R3ts0Y5JjeEhG3RcSWtLuS7FmeoeCtqXYi4g2gd7qcthMRGyPigbT9MvAo2awNQ9UcsmmISD+PG7xQWjoCeCIinh7sQPrCiaOcQ4HnIuLxXNkUST+TdJekQwcrsBbmpyGfJbnT90bTvbSjs4Bbcvvt/J0Ppe/1LelMbn/g3lRU73emnQRwm6T70/RCUDMdEVB3OqI2cQpb/w9ou3/fW3HiSCT9SNKaOq/8/y2eytb/sTcCkyNif+ALwNWSdh3IuKFl7AuBPYH9Urzf7G1Wp6sBHbcs8p1LOg/YAnw/FbXFd97EoH+vZUn6LeCfgM9FxEs0/p1pJx+MiOlkM2yfLemPBzugopQ91HwscH0qGgrf91a8AmASEUc2Oy5pe+ATwAG5Nq8Dr6ft+yU9AewFdFUY6jZaxd5L0uXAD9NukSlhKlXgOz8DOAY4Is2g3DbfeROD/r2WIeldZEnj+xFxA0BEPJc7nv+daRsRsSH9fF7SjWRDhM9JGp8mP23n6YhmAw/0fs9D4fuu5TOO4o4EHouInt4CSePSRS4kvY9sXZH1gxRfXekfUK/jgTVpezlwiqQdJE0hi/2+gY6vEUmzyKbLPzYiNufK2/07LzLVTltI1+yuAB6NiG/lyhv9zrQFSe9OF/OR9G7gI2QxDpXpiLYauWj377sen3EUVzsmCfDHwPmStgBvAvMi4t8HPLLmviFpP7LhkqeAvwBI079cR7bGyRbg7Ha5oyq5FNgBuD37+8bKiJhHm3/nLabLaTcfBP4L8LDSLebAl4FT6/3OtJE9gBvT78X2wNUR8a+SVjG40xG1JGlnsjvu8t9p3X+j7cy345qZWSkeqjIzs1KcOMzMrBQnDjMzK8WJw8zMSnHiMDOzUpw4bEiRdKdqZvJNM4x+u0WbzorjuiZNGfH5mvLj0oSSfelzvzIzpUoaI+mzZetJeq+kH/RXfRv+nDhsqLmG7JmavHrP2AwYSb8LHBwR+0bExTWHjwP6lDjIpqAoM8X2GKBl4qitFxEbIuKEfqxvw5wThw01PwCOkbQDvDU533uBn0haKKlL2doSf1uvsaRXctsnSLoqbY+T9E+SVqXXB+u03VHSlcrWgfiZpMPSoduA3ZWtpXBorv7BZHMSXZiO7Zle/5om5/s/kvZJdU9M83Q9JOnu9NT5+cDJqe3JNbH8oaT70rHVkqYCFwB7prILJf2WpDskPZBi7p0DrLZeh6Q1JfrN1x8l6aLU/2pJf1n4v6QNXRHhl19D6gX8CzAnbS8ALkzbu6Wfo4A7gX3T/p1kazdAtu5Bbz8nAFel7auBQ9L2ZLJpOGrf978BV6btfcieTt4R6ADWNIj1KuCE3P4dwNS0PZNszRGAh4EJaXtM+vlnwKUN+v0H4LS0PRrYqTYOsqeqd03bY8mmzledem/tF+w3X/8zZHNdbZ//b+DX8H55yhEbinqHq25OP89K5Scpm2J7e2A82RDR6oJ9HglMS9NYAOwqaZfI1qnodQjZH1Yi4jFJT5NNsPhSkTdQNgvtwcD1uffZIf28B7gqTQNzQ4HufgqcJ2kicENEPJ7r8623BP5O2cyxvyGb3n2Pfug370hgUaS1U6KNpn+x6jhx2FB0E/AtSdOBnSJbwW4K8EXgwIj4f2kIasc6bfNz7OSPbwf8UUS81uR9m/4FLWA74D8iYr9tgoqYJ2km8DHgwTR3UUMRcbWke1P9WyX9OdtO9ngaMA44ICJ+rWzVvHrfSdl+80SbTxtv/c/XOGzIiYhXyIaflvD2RfFdgVeBFyXtQTZ1dT3PSfoDSduRzUTa6zZgfu9Ogz/cd5P9MUbSXmRDWutahPsysEuK+yXgSUknpj4k6QNpe8+IuDcivgq8QDY1+1ttaymbGXh9RFxCNivsvnXq/zbwfEoahwG/VxtTH/vNuw2Yp2zZASTt1uL7sGHAicOGqmuAD5AtzUpEPAT8DFhLllDuadBuAdl6Bz8mWzSn138FOtMF3keAeXXafhsYJelh4FrgzyJbH6SZZcCX0sX0PckSz6ckPZRi7b1gfWG6wLyGLEE9BPwb2fDZNhfHgZOBNcpmtd0HWBoRvwLuSRfZLyRb/KpTUld638fSd1Vbr2y/ef9Idq1ndfpMn2zxfdgw4NlxzcysFJ9xmJlZKU4cZmZWihOHmZmV4sRhZmalOHGYmVkpThxmZlaKE4eZmZXy/wHyovxJGFZxogAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "from itertools import combinations\n", + "import matplotlib.pyplot as plt\n", + "\n", + "def null_distribution(z, nx):\n", + " # z is the population of mice survival times in the study\n", + " # nx is the number of mice in the treatment group\n", + " z = set(z)\n", + " null_distribution = []\n", + " for x in combinations(z, nx):\n", + " y = z - set(x)\n", + " stat = statistic(list(x), list(y))\n", + " null_distribution.append(stat)\n", + " return null_distribution\n", + "\n", + "null_dist = null_distribution(z, len(x))\n", + "plt.hist(null_dist, density=True, bins=50)\n", + "plt.xlabel(\"Value of test statistic\")\n", + "plt.ylabel(\"Observed Frequency\")" + ] + }, + { + "cell_type": "markdown", + "id": "13914661-4e02-47db-82d0-a2302cc1e695", + "metadata": {}, + "source": [ + "We complete the hypothesis test by comparing the observed value of the test statistic to the rest of the null distribution." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "d0b5ba29-952e-4652-8f35-58928e76e6ad", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.1409965034965035\n" + ] + } + ], + "source": [ + "pvalue = np.sum(null_dist >= statistic(x, y) ) / len(null_dist)\n", + "print(pvalue)" + ] + }, + { + "cell_type": "markdown", + "id": "90fd3def-2996-4366-a661-76c3216cd634", + "metadata": {}, + "source": [ + "Approximately 14% of the values of the null distribution are greater than the observed value of the statistic, so there is a 14% probability of observing such an extreme value of the statistic even if the the treatment had no effect at all. \n", + "\n", + "Given data and a statistic function, `stats.permutation_test` performs the same test automatically." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "8ff08473-77e4-4bc9-864e-2f06f19f2948", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "PermutationTestResult(statistic=30.63492063492064, pvalue=0.1409965034965035, null_distribution=array([ 30.63492063, 38. , 51.20634921, ..., -11.01587302,\n", + " -45.55555556, -34.88888889]))\n" + ] + } + ], + "source": [ + "# `alternative` is 'greater' because we are interested in the percentage of values in the\n", + "# null distribution that are greater than the observed value of the test statistic.\n", + "# `n_resamples` is `np.inf` to ensure that all possible permutations are used\n", + "# Note that `(x, y)`, a tuple, is a single argument.\n", + "res = stats.permutation_test((x, y), statistic, alternative='greater', n_resamples = np.inf)\n", + "assert res.pvalue == pvalue\n", + "print(res)" + ] + }, + { + "cell_type": "markdown", + "id": "8b5bb238-7db9-40eb-8221-d703c3d26b70", + "metadata": {}, + "source": [ + "It returns the observed value of the test statistic, the null distribution, and the $p$-value. They are related as above:" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "f303201f-cf56-46c1-8ee3-428afca93eb3", + "metadata": {}, + "outputs": [], + "source": [ + "assert np.sum(res.null_distribution >= res.statistic ) / len(res.null_distribution) == res.pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "ae116861-ded9-4728-a1b1-4c9c34c50fdd", + "metadata": {}, + "source": [ + "Note that the exact $p$-value from the permutation test matches the $p$-value from the t-test quite closely. (As we shall see, Ronald Fisher introduced permutation tests primarily to support the use of the t-test in applications where the underlying normality assumptions were not strictly true [[4](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2458144/https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2458144/)].)" + ] + }, + { + "cell_type": "markdown", + "id": "ae96fea0-1255-48ac-9d7f-e56f1db10ae2", + "metadata": {}, + "source": [ + "### Randomized Tests\n", + "The number of possible permutations grows rather quickly as the number of observations increases. Specifically, if $n_x$ and $n_y$ are the number of observations in `x` and `y`, respectively, than the number of possible permutations is $\\frac{(n_x + n_y)!}{n_x! n_y!}$. " + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "d864ec56-9438-45f2-a41a-cb30e3c1ac35", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "11440\n" + ] + } + ], + "source": [ + "from math import factorial as f\n", + "n_x = len(x)\n", + "n_y = len(y)\n", + "assert len(res.null_distribution) == f(n_x + n_y) / (f(n_x) * f(n_y))\n", + "print(len(res.null_distribution))" + ] + }, + { + "cell_type": "markdown", + "id": "432cff2d-f08e-4be3-99f2-c97a8f751e83", + "metadata": {}, + "source": [ + "When the number of possible permutations is too large, it is common to use a randomly-sampled subset of the possible permutations instead. As with `monte_carlo_test` the maximum number of resamples used by `permutation_test` is controlled using the `n_resamples` parameter." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "e9625f6e-d527-4e78-89fc-e24a603a2811", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.1494" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# use only 4999 randomly-sampled permutations\n", + "res = stats.permutation_test((x, y), statistic, alternative='greater', n_resamples = 4999)\n", + "res.pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "e1937a92-5ad8-4164-a6c0-5ee01e6702b0", + "metadata": {}, + "source": [ + "If the number of distinct permutations of the data is less than or equal to `n_resamples`, `permutation_test` performs an exact test, computing the value of the test statistic for each distinct permutation exactly once. If the number of distinct permutations exceeds `n_resamples`, `permutation_test` computes the value of the statistic for `n_resamples` random permutations, and the $p$-value is computed as:" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "8e7e5525-5cbf-43df-86b2-7af081ad96e4", + "metadata": {}, + "outputs": [], + "source": [ + "pvalue = (np.sum(res.null_distribution >= res.statistic ) + 1) / (len(res.null_distribution) + 1)\n", + "assert pvalue == res.pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "459d7ca3-f719-4b8e-a855-8e47846c3ec0", + "metadata": {}, + "source": [ + "Note that `1` is added to both the numerator and denominator when performing the randomized test [[3]](https://www.degruyter.com/document/doi/10.2202/1544-6115.1585/html). This can be thought of as including the observed value of the test statistic in the null distribution, and it ensures that the $p$-value of a randomized test is never zero." + ] + }, + { + "cell_type": "markdown", + "id": "cb30657f-b7db-4762-95c2-1eebb2a05a64", + "metadata": {}, + "source": [ + "A wide variety of common hypothesis tests can be performed as permutation tests. We continue with several other examples to explore the flexibility of `permutation_test`, beginning with [Independent-Sample Tests](https://nbviewer.org/github/mdhaber/scipy/blob/resampling_tutorial/doc/source/tutorial/stats/notebooks/resampling_tutorial_2a.ipynb)." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.5" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_2a.ipynb b/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_2a.ipynb new file mode 100644 index 0000000..76ca6f0 --- /dev/null +++ b/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_2a.ipynb @@ -0,0 +1,578 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "57c13277-a239-452b-a202-e499e0a068b0", + "metadata": {}, + "source": [ + "### Independent-Sample Tests\n", + "#### Two-sample Test\n", + "In [Individual Comparisons by Ranking Methods](https://www.jstor.org/stable/3001968#metadata_info_tab_contents), Wilcoxon considers two sprays designed to kill flying insects. A subset of the data, the percentage of flies killed in repeated trials of two treatments, is recorded below." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "fe264441-ebf5-47f0-83fb-81b824751b2d", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "x = np.array([61, 62, 67, 63, 56, 58])\n", + "y = np.array([60, 68, 59, 72, 64])" + ] + }, + { + "cell_type": "markdown", + "id": "8525c260-6699-4f90-b20c-85e0d948bb3a", + "metadata": {}, + "source": [ + "In the paper, Wilcoxon describes a test to assess whether the two samples are drawn from the same population that is now commonly described as a *nonparametric* version of the independent sample t-test - that is, a version of the t-test that does not make the normality (or any particular distributional) assumption. \n", + "\n", + "Suppose we want to test that null hypothesis that the samples are drawn from the same distribution against the alternative that they are drawn from different distributions which tend to produce samples with a lower values of the statistic. Under certain assumptions, this can be argued as evidence that the location of the distribution underlying `x` is less than the location of the distribution underlying `y`. We pass the data into [`scipy.stats.mannwhitneyu`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html) with `alternative='less'`." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "9fc1c550-67a2-4483-b760-e119b16e6b2d", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.16450216450216448" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from scipy import stats\n", + "_, pvalue = stats.mannwhitneyu(x, y, alternative='less')\n", + "pvalue # p-value is greater than our threshold; test is inconclusive" + ] + }, + { + "cell_type": "markdown", + "id": "4b709400-3601-4ce5-9a36-538498abab45", + "metadata": {}, + "source": [ + "Like the mean comparison test in Efron's example, this is an example of an \"independent sample\" test of the null hypothesis that group labels (`x`, `y`) are entirely random. In fact, because `mannwhitneyu` claims to produce an exact value of the statistic, we would expect `permutation_test` to return precisely the same $p$-value (using `mannwhitneyu` only to compute the statistic)." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "f526ac5a-b0a8-4c60-a47d-324c22215431", + "metadata": {}, + "outputs": [], + "source": [ + "def statistic(x, y):\n", + " # return just the Mann-Whitney U statistic\n", + " return stats.mannwhitneyu(x, y, alternative='less').statistic\n", + "\n", + "# \"independent\" is the default `permutation type`, so we are not required to pass it here\n", + "# We pass `alternative='less'` because lesser values of the statistic are more extreme\n", + "res = stats.permutation_test((x, y), statistic, permutation_type='independent', alternative='less')\n", + "np.testing.assert_allclose(res.pvalue, pvalue, atol=1e-15)" + ] + }, + { + "cell_type": "markdown", + "id": "5c346fa8-1b34-4443-8220-c018b6d0f5a8", + "metadata": {}, + "source": [ + "Just as with `monte_carlo_test`, vectorizing the `statistic` function can greatly improve the speed of the test." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "4f2dae73-2bf1-4c66-89b3-ac1ff0487e4c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "272 ms ± 4.61 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" + ] + } + ], + "source": [ + "# Before\n", + "%timeit stats.permutation_test((x, y), statistic, alternative='less')" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "b5c8a5c2-e958-4164-bd19-0b9fffc342b5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "56.5 ms ± 859 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n" + ] + } + ], + "source": [ + "# After \n", + "def statistic_vectorized(x, y, axis=0):\n", + " # return just the Mann-Whitney U statistic\n", + " return stats.mannwhitneyu(x, y, axis=axis, alternative='less').statistic\n", + "\n", + "%timeit stats.permutation_test((x, y), statistic_vectorized, alternative='less', vectorized=True)" + ] + }, + { + "cell_type": "markdown", + "id": "a751fc04-e796-4d8e-bf5a-3c47da9e4323", + "metadata": {}, + "source": [ + "Although `mannwhitneyu` provides an exact $p$-value for the data above, `permutation_test` comes in handy when there are ties in the samples. As the [`mannwhitneyu` documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html) states,\n", + "> `'exact'`: computes the exact p-value by comparing the observed statistic against the exact distribution of the statistic under the null hypothesis. **No correction is made for ties.**\n", + "\n", + "The complete data set in Wilcoxon's original paper had ties." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "ccadcdf2-1341-48c6-bcc3-92be7b80b725", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.014763014763014764 0.01351981351981352\n" + ] + } + ], + "source": [ + "x = [60, 67, 61, 62, 67, 63, 56, 58]\n", + "y = [68, 68, 59, 72, 64, 67, 70, 74]\n", + "res1 = stats.mannwhitneyu(x, y, method='exact', alternative='two-sided')\n", + "# By default, only 9,999 random permutations are used. \n", + "# We pass n_resamples=np.inf to ensure that all 12,870 possible permutations are used\n", + "res2 = stats.permutation_test((x, y), statistic_vectorized, alternative='two-sided', vectorized=True, n_resamples=np.inf)\n", + "print(res1.pvalue, res2.pvalue)" + ] + }, + { + "cell_type": "markdown", + "id": "7c85b266-314c-490f-9a22-a5c1e1400496", + "metadata": {}, + "source": [ + "The two $p$-values are similar despite the ties, but only `permutation_test` is truly \"exact\" in this case. Either way, our 1% threshold for statistical significance is not met, and the test is inconclusive." + ] + }, + { + "cell_type": "markdown", + "id": "6cfb7ac5-6ec6-43a4-ae3b-f5ef32050f17", + "metadata": {}, + "source": [ + "#### Multi-sample Test\n", + "`scipy.stats.kruskal` is a many-sample extension of the Mann-Whitney U test, but SciPy provides only an approximate (asymptotic) $p$-value. It is possible to perform an exact version of the test using `permutation_test` very small samples, and a randomized test using a subset of the possible permutations may yield more accurate results than the approximation implemented by `kruskal`, especially if there are ties or the sample size is small. Using the (artificial) data for milk cap production from [Kruskal and Wallis' original paper](https://www.tandfonline.com/doi/abs/10.1080/01621459.1952.10483441), we have:" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "d8879fa7-9576-4d84-acfb-85962ebb0b02", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "KruskalResult(statistic=5.656410256410254, pvalue=0.059118869289796136)" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "x = [340, 345, 330, 342, 338]\n", + "y = [339, 333, 344]\n", + "z = [347, 343, 349, 355]\n", + "stats.kruskal(x, y, z)" + ] + }, + { + "cell_type": "markdown", + "id": "246840a9-f7fe-4af6-942e-ad97540e5730", + "metadata": {}, + "source": [ + "At the expense of some time, the exact p-value for this data is given by `permutation_test`." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "5b274945-72d2-4597-a2dd-e9d7efbe293e", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.048629148629148626" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def statistic(x, y, z, axis=0):\n", + " return stats.kruskal(x, y, z, axis=axis).statistic\n", + "\n", + "res = stats.permutation_test((x, y, z), statistic, vectorized=True, alternative='greater', n_resamples=np.inf)\n", + "res.pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "fc877591-9f75-4990-a035-a6a48b4e4597", + "metadata": {}, + "source": [ + "Note that we passed `alternative='greater'` into `permutation_test` but not into `kruskal`. This is because the `kruskal` statistic is inherently one-sided test: data generated under the null hypothesis tends to generate small positive values, and data with greater values are more exceptional. This raises the point that setting up a permutation test requires some study of the underlying statistic and SciPy's implementation. Another example of this is shown in the next section." + ] + }, + { + "cell_type": "markdown", + "id": "2eb247ee-5dc9-4951-9124-11bd3a88c510", + "metadata": {}, + "source": [ + "#### Gotchas\n", + "Suppose that we wish to perform the two-sample Kolmogorov-Smirnov test to test the null hypothesis that two samples were drawn from the same distribution against the alternative that the distribution $X$ underlying sample `x` is [stochastically greater](https://en.wikipedia.org/wiki/Stochastic_ordering) than the distribution $Y$ underlying sample `y`. Roughly speaking, this is the alternative that $X$ \"tends to be\" greater than $Y$.\n", + "\n", + "Here, we'll use randomly-generated data that best illustrates some confusing (but important) points. We choose shapes of the samples to generate the `RuntimeWarning` reported in [gh-14019](https://github.com/scipy/scipy/issues/14019)." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "08329d40-cd24-482b-9a40-24e2ed90b23e", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABDv0lEQVR4nO3dd3hcV5n48e87TSNp1HuxLfca98ROcSokdnohpAdIQja7S4clQGgLywK7/AIBkg2BQEggPSEJ6dWkufduy5Ysq/feZ87vjzs2ku7Ilm1JI43ez/PMY2veO3deXc28c+acc88VYwxKKaVGP0e4E1BKKTU4tKArpVSE0IKulFIRQgu6UkpFCC3oSikVIbSgK6VUhNCCPghE5EER+d4g7Wu8iDSLiDP480oRuWMw9h3c32si8pnB2l8kE5FCEflEuPM4GhE5V0SKh3D/g/baVkNPC/oxBN/UbSLSJCL1IvKxiNwlIkeOnTHmLmPMjwe4r6MWCGNMkTHGZ4zxD0LuPxSRv/TZ/wpjzJ9Pdt8hnusREekMfhjVishbIjKjRx5dwWPYJCJ7ReS3IpLV4/Hnikgg+PjDt78Pdp6DRURyReQ5EakWkQYR2SYinw13Xscj+Df7rz735YmIEREXDO5rWw09LegDc5kxJg6YAPwMuBt4eLCf5PCbaBT7H2OMD8gFKoFHesSeCh7DZOAqIBPY0LOoA6XBD7PDt8uGK/ET8BhwCOs1kQLcClSENaMIFQHvi2GjBf04GGMajDEvAdcBnxGROdC7pSMiqSLycrA1XysiH4iIQ0QeA8YDfw+2Pr/ZozV0u4gUAe/2bSEFTRaRtcGW4Isikhx8LtvX7cMtJRFZDnwHuC74fFuC8SNdOMG8visiB0WkUkQeFZGEYOxwHp8RkaJgS/SeAR6nVuBxYE6IWJcxZkfwGFYBXz/W/kTkNBFZLyKNIlIhIvf2s11S8NhXiUhd8P+5PeIrReTHIvJR8JvCmyKS2iN+S/BY1Azgdz0VeMQY02KM6TbGbDLGvNZjX8+ISHnwb/a+iMzuEXtERB4Qq/urOZhPpoj8Kpj3bhFZ0GP7QhH5tojsDMb/JCLefo5BdvCbQ5WIFIjIl451fI/mRF/bwe0vF5Edwe1XisjMHvtdKCKbgn+HZ0TkqR7Pc66IFIvI3SJSDvxpgH/b/xLrG3SziPxdRFJE5K/B1806EckLbisi8svga75BRLZK8L082mlBPwHGmLVAMbAsRPjrwVgakIFVVI0x5hagCKu17zPG/E+Px5wDzAQu6ucpbwVuA7KBbuDXA8jxdeC/sVrGPmPMvBCbfTZ4Ow+YBPiA3/bZ5ixgOnAB8P2eb8r+iIgPuAnYdJT8/MCLhD6Gfd0H3GeMiQcmA0/3s50D+BNWq3k80Ib997kR+ByQDniAbwRzngX8H3AL1nFOwfqm0Z/VwP0icr2IjA8Rfw2YGnyejcBf+8Q/DXwXSAU6gFXB7VKBZ4G+H1o3Yb0+JgPTgo/tRaxuwL8DW4AcrL/ZV0Skv9fV8Rrwa1tEpgFPAF8Jbv8qVsH3iIgH+BvWN7jk4HZX9XmuzGBsAnAnA/vbXo/198vBOk6rgo9JBnYBPwhudyFwNtZxTMRqXNSc8FEZQbSgn7hSrBdKX11AFjAh2Br9wBx7wZwfBlt6bf3EHzPGbDfGtADfAz4twUHTk3QTcK8x5oAxphn4NnC99P528J/GmDZjzBasQhHqg+Gwb4hIPZCP9eHw2WM8f99jmB1szR2+fTp4fxcwRURSjTHNxpjVoXZmjKkxxjxnjGk1xjQBP8H6sOzpT8aYvcFj/TQwP3j/p4CXjTHvG2M6sI5z4Ci5Xwt8ENyuQEQ2i8ipPXL5ozGmKbivHwLzJPjtJ+hvxpgNxph2rOLWbox5NPhB9xSwgN5+a4w5ZIypDf5eN4TI6VQgzRjzI2NMpzHmAPB7rELXn2/0PObA1qNsezyv7euAV4wxbxljuoBfANHAGcBSwAX8Orif54G1fR4fAH5gjOkIvv4G+rfdb4xpwPpA3W+MedsY0w08wz+PaRcQB8wAxBizyxhTdpTfe9TQgn7icoDaEPf/L1ZBe1NEDojItwawr0PHET8IuLFacicrO7i/nvt2YbW+Divv8f9WrELdn18YYxKNMZnGmMuNMfuP8fx9j2Fp8PGHb4db4rdjtaZ2B786XxpqZyISIyK/C3abNALvA4l9Pvz6+32y6XGcgx+e/bbajDF1xphvGWNmYx2vzcALwa/zThH5mYjsD+ZRGHxYz79Zz/72thA/9z3OfV8D2SHSmkCfD0WsVnRGiG0P+0XPYw7MPcq2x/Pa7vXaMsYEgr9DTjBW0ufDoO97oCr4YQcM+G87oGNqjHkXq3V/P1AhIg+JSPxRfpdRQwv6CQi2xHKAD/vGgq2yrxtjJgGXAV8TkQsOh/vZ5bFa8ON6/H88VgujGmgBYnrk5cT6ejvQ/ZZiFYGe++5mGAb3gt0Dl2G1co/KGLPPGHMDVvfFz4FnRSQ2xKZfx+oeWhLsnjn78NMNIKUyehxnEYnB6nY5JmNMNVYLNBvrG8eNwBXAJ4AEIO848uhP39dAaYhtDgEFfT4U44wxF5/E8x5xnK/tXq8tEZHg71CCdaxzgvcdNq73w237O5m/bajf5dfGmEXAbKzGwn+cyH5GGi3ox0FE4oOtwyeBvxhjtoXY5lIRmRJ8sTYC/uANrEI56QSe+mYRmRUsMj8Cng1+Nd8LeEXkEhFxY/WrRvV4XAWQJz2mWPbxBPBVEZkY7Pc+3OfefQI5DoiIuIP98E9g9ZOGHODs85ibRSQt2MqrD94dalpnHFZLrF6sgeMfhNimP88Cl4rIWcE+3h9xlPeHiPxcROaIiEtE4oB/BfKNMTXBPDqwWvgxWMf1ZP27WFMlk7Fa3U+F2GYt0BgcTIwOflOY07Mr6GQc52v7aeASEbkg+Nr8OtYx+Rirb9sPfCF4/K4ATjvG05/M37bv73GqiCwJ5tUCtBP69TTqaEEfmL+LSBNWC+gerCL0uX62nQq8DTRjvXAfMMasDMZ+Cnw3+HX4G8fx/I9hDSCVA17gS2DNugH+DfgDVsunBWvQ6rBngv/WiMjGEPv9Y3Df7wMFWC/sLx5HXsfjOhFpxirIL2EVu0XGmFAtzb6WAzuCj78PuL7n1/EefoXVT1uNNWj5+kCTM9bMm3/Hmp1TBtTR+1j2FYPV910PHMBqjV4ejD2K1d1QAuwM5nKyHgfeDD7XAeC/+m4Q/JC/DGtcoADrOPwB61vCYBjwa9sYswe4GfhNMI/LsAZNO40xncDVWF1p9cHtXsYq+P35FSf4tw0hHmtsoQ7r71SD9Q1r1JNjj9cppcJJRAqBO4wxb4c7l6EiImuAB40xfwp3LqOZttCVUsNORM4Ra+69S6ylKOZycq1uhTWjQSmlhtt0rH52H7Af+FSkTB0MJ+1yUUqpCKFdLkopFSHC1uWSmppq8vLywvX0Sik1Km3YsKHaGJMWKha2gp6Xl8f69evD9fRKKTUqicjB/mLa5aKUUhFCC7pSSkUILehKKRUhdB66UmrM6erqori4mPb2UCtIjAxer5fc3FzcbveAH6MFXSk15hQXFxMXF0deXh69F30cGYwx1NTUUFxczMSJEwf8OO1yUUqNOe3t7aSkpIzIYg4gIqSkpBz3Nwgt6EqpMWmkFvPDTiQ/LehKKRUhtKArpVSE0EFRNTq899P+Y+d9e/jyUGoE0xa6UkoNs3Xr1jF37lza29tpaWlh9uzZbN++/aT3qy10pdSY9p9/38HO0sZB3ees7Hh+cNnsfuOnnnoql19+Od/97ndpa2vj5ptvZs6cOSf9vFrQlVIqDL7//e9z6qmn4vV6+fWvfz0o+9SCrpQa047Wkh5KtbW1NDc309XVRXt7O7GxsSe9Ty3oamw72mAr6ICrGjJ33nknP/7xjykoKODuu+/mt7/97UnvUwu6UkoNs0cffRSXy8WNN96I3+/njDPO4N133+X8888/qf1qQVdKqWF26623cuuttwLgdDpZs2bNoOxXpy0qpVSE0IKulFIRQgu6UkpFCC3oSikVIbSgK6VUhNCCrpRSEUILulJKRQgt6EopFSG0oCul1DD73ve+x3333Xfk53vuuWdQFujSM0WVOhq9sEbke+1bUL5tcPeZeQqs+Fm/4dtvv52rr76aL3/5ywQCAZ588knWrl170k+rBV0ppYZZXl4eKSkpbNq0iYqKChYsWEBKSspJ71cLulJqbDtKS3oo3XHHHTzyyCOUl5dz2223Dco+tQ9dKaXC4KqrruL1119n3bp1XHTRRYOyT22hK6VUGHg8Hs477zwSExNxOp2Dsk8t6Eodw6oDNSHvX929l69+ctowZ6MiRSAQYPXq1TzzzDODtk/tclFKqWG2c+dOpkyZwgUXXMDUqVMHbb/aQlfqJPzyrb1HjWsLXoUya9YsDhw4MOj71Ra6UmpMMsaEO4WjOpH8tKArpcYcr9dLTU3NiC3qxhhqamrwer3H9TjtclFKjTm5ubkUFxdTVVUV7lT65fV6yc3NPa7HDKigi8hy4D7ACfzBGBNyJr6InAqsBq4zxjx7XJkopdQwcbvdTJw4MdxpDLpjdrmIiBO4H1gBzAJuEJFZ/Wz3c+CNwU5SKaXUsQ2khX4akG+MOQAgIk8CVwA7+2z3ReA54NRBzVCp4dJaC6095pxH+cCXCUDAQEVXNIc6Y2nyuxnnaaHLH8Dt1GEoNXIMpKDnAId6/FwMLOm5gYjkAFcB53OUgi4idwJ3AowfP/54c1Vq8HW3Q+Uua7W9plJ72JtMcdcSftO6nCKT1ismxftJjYvitLxkJqfFIiLDlbVSIQ2koId6lfYdGv4VcLcxxn+0F7Ux5iHgIYDFixePzOFlNXYcWAnrfg+dLRCTCpPOg4TxIEJzt/D+wTZS6zZzreM1roh6i7e9F1Hrm0acq5uiDh8feM8hv6KZV7aVkZMYzdlTU0mPP75ZCUoNpoEU9GJgXI+fc4G+TZnFwJPBYp4KXCwi3caYFwYjSaUGlb8bVv4UPvh/EJMMs66G+GwINkZK2tzcumUi+1u8XJW1jEbnGhY3vcPF7a9QLbs4kLyMdF873eNTWJKXzPbSBlYfqOXJdYc4b3o6p+QmhPkXVGPVQAr6OmCqiEwESoDrgRt7bmCMOTJcLCKPAC9rMVcjUlcbPH4dFPwD5t8MvnRweo6E9zRFceuGSbT6HTxx6n5OT25h1YEodnlXkN20hfEN6/FVVLMj7VIAHA5hbm4i0zPjeH17Oe/uqaSt28+pE5K0C0YNu2OO6BhjuoEvYM1e2QU8bYzZISJ3ichdQ52gUoPG3w3P3gYF78Plv4Er7+9VzNfXxXDt2skAPHOaVcyPEKE0fj470i/F5W9jZtWrRHU1HAlHuZxcOjebGZlxrNpfw/v7qkfsSSsqcg1oHrox5lXg1T73PdjPtp89+bSUGmSBALz0RdjzKlz8C1h4a69wUauH2zfmkerx8+jiA+RGd4XcTVNUJntSL2Rm1etctfMrPDfnfrqcMQA4HcKFszLwupxsPlSP16UzYNTw0lecGhve/gFseRzO/Q6c9vleoZZuB3dumgDAnxYV9FvMD2v0ZrM39Xwymndy6e67EdN9JCYinD0tlRmZcawuqOXd3RWD/7so1Q8t6Cry7X4FPv41LL4dzvlmr5Ax8B/bc9nb7OU384qYENM5oF3WRefx9pTvkFe/miWH/tgrJiKcPyOdNF8UX35yM4XVLf3sRanBpQVdRbamCqurJfMUWP7TIzNZDnuwII1XKxL51rQyzk5tPq5d78i4gp1pF7Pk0MNkNW7tFXM7HVw6NwunQ/iXxzbQ0tHdz16UGjxa0FXkMgZe/Ddrnvk1D4Mrqld4b0UT9+ZncElGPZ/Pqz6hp3hv0n/QFJXJ8n3fx93duyUeH+3mtzcsZG9lE//7xp4T/jWUGigt6CpyrX0I8t+GC/8L0qb3CvkDhruf24rPFeBHs0r6NtwHrNPl4/VpPyK+vYzzCn5hi581NZVbl07gz6sK2Xyo/sSeRKkB0oKuIlN9Ebz1fZh6IZx6hy382KpCNhXV8/0ZpaR4/Cf1VKXx81ib+1lmV77MhLqPbfFvXDSdjDgv33puK13+wEk9l1JHowVdRaY3vwsIXPpLW795SX0b//PGHs6elsaVWfWD8nRrxt1BnXc85xbciyPQe5ZMnNfNj6+cw+7yJn7/weBfdkypw/QCFyryFLwPO1+E8+6BhFx476e9wj/YmIfx+/hJ5vsn3NXSV8Dh5h8Tv8qVu77K/LKn2Jhzc6/4J2dlsGJOJve9vY+L52SRt+3XR9/hed8enMTUmKItdBVZ/N3w2t2QOB7O+KIt/HFNLG9XxfOlSRWMO8Z88+NVkHwWBUlnsPTQH4jprLHFf3j5bFwO4eev7x7U51XqMC3oKrJs+BNU7rQGQt3RvULGwM/3ZpHl7eRzE05sVsux/CPvq7gC7Zx58AFbLCPey+fPnsRr28vZVB8d4tFKnRwt6CpytNXDez+BvGUw83Jb+NWKBLY0xvDVKRV4nUOzzkpdTB6bsq5nduXfSWu2T1W8Y9kkUn0efr43C13qRQ02Legqcqz+P2irg4t+YhsI7QrA/+7LZLqvjWuy64Y0jTXj7qDDFcfpRb+zxXxRLr54/lRW1/n4R7VvSPNQY48OiqrI0FYHqx+AGZdC1jxb+MniFApbo/jjwgKcfQZCVx2w93efjE6Xjw3ZN3Jm0YM8/vwLVMT1vgSvP2BId7fxg+1pvHduMw5dZVcNEi3oKjKsegA6GuFc++yQdr9w3/50Tktq5rzUpkF7yqVFD/UbcwY6aXMlsPTQ73lx1i97xxzCp1MO8Nvy2fy9LJErsusHLSc1tmmXixr9Wmut7paZl0PmHFv4mZJkqjvdfG1KxaBNUzwWv8PDhuybmFT3IRlNO2zx032VjPM089sD6QS0L10NEi3oavRb/QB0NsE5d9tCXf4ADxaksTCxhSVJw7vq4Zasa4+00vtyCFyeVMS+Fi9vV8UPa14qcmlBV6NbVzusfhBmXRGydf7S5lJK2j38+8TKYWudH9bp8rEh5yYm1X0UupUeV8m46A4eOJCmM17UoNCCrka3sk1W6/zs/7CFAgHD//1jP+M9zUQ3FbLqQE3I21DanPlp2p1xLC55zBZziuHOvCo2N8SyqjZ2SPNQY4MWdDV6BfxQsgEmnWetd97HmzsryK9s5vLkg8PeOj+syxXLtsyrmFLzHvHtJbb4tTl1pHq6eKAgPQzZqUijBV2NXpU7obMZzviCLWSM4YGV+UxIiWGpryoMyf3TpqzrMCIsLH3CFvM6DbdPqObDmji2NujZo+rkaEFXo5MxULwWYtNg8gW28PqDdWwtbuDzyybhlPB2ULdEpbMn9SJmV7xEVHejLX7z+BriXH5+V5gWhuxUJNGCrkanukJoqYLcU21nhQI88lEhCdFurl6YM/y5hbAh5yY8gTbmlj9vi8W5AlyXU8vrFQmUtbvDkJ2KFFrQ1ehUvBY8sZA+yxYqrW/j9R3lXH/qOGI8I+PcuerYaRxMOI35ZU/b1ksHuHV8DQEDfz2UHIbsVKTQgq5Gn5YqqCuA7EXgsBfsx1YfxBjDLadPCENy/duQcxO+ziqmV7915L7DM21KystYGFvNnw8m8Y/8On751t4wZqpGKy3oavQp3QTihOz5tlB7l58n1hZx4axMcpNihj+3oziYeDq10ROYW/5syPjyxBKa/B5WNeuMF3VitKCr0aW7Ayq2Q/pMcNsL9oubS6hv7eKzZ+YNf27HIsLWzGvIbtpGbKd9PfbZ0XXkeFp4vT4Xo2caqRMwMjoYlRqoyh3g74TshUfuOtw9YYzh8bVFpPo8fJxfzar91klDS8OSaGg70y/lzIP3k9G8kwPJZ/eKicDyxGIerpxOWUN7mDJUo5m20NXoYYzV3eLLgLgsW7i0oZ3q5k7mjUtEwnUm0TF0uOLYnbac1NZ8nIEOW/ysuHJiHV1sOVQ//MmpUU8Luho9GoutAdHshSGnKm4vacDjdDA9Iy4MyQ3c1sxP4TR+0lr22WJeR4Bl8eXkVzVT02wv+EodjRZ0NXqUbgJnVMipiu1dfvZVNjMjMw63c2S/rCt9M2jypJPZvJNQq3KdH19KwMBzG4vDkJ0azUb2K1+pwzpboGq3tWaL037yza6yRvwBw5ychDAkd/zKfbOI7m4gvqPUFhsX1UpWgpcn1h7SwVF1XLSgq9GhfBuYAGQvsIWMMWwvaSQz3ktaXFQYkjt+NTET6XJEkdG8O2T8lJwECqpbWH2gdpgzU6OZFnQ18hkD5VshIRdiUmzh0vp2als7mZMzei4UYcRFdcxUktsKcfntM1qmpvuI97p4Ym1RGLJTo5UWdDXyFa2CtlrItF/8GWB7qTUYOm2ED4b2VRk7HQcBUlvtg6Mup4OrF+by+vZyals6w5CdGo20oKuRb+Oj1mBo2nRbqL7TOWoGQ/tq9STT5Ekjo2VPyMHRG04bT6c/wHMbdHBUDcyATiwSkeXAfYAT+IMx5md94lcAPwYCQDfwFWPMh4Ocqxrt3vtp/7Hzvh36/vYG2PGCdWao02MLv1iWOKoGQ/uqjJ3O5LoP8XVW0RzV+5T/6ZlxLJqQxJPrirhj2cQRO7dejRzHbNKIiBO4H1gBzAJuEJG+88beAeYZY+YDtwF/GOQ81Vi17Vnobuu3u+WZ0iTSfFGjZjC0r5qYyfjFRXrLnpDxaxflsr+qhU16opEagIF8Rz0NyDfGHDDGdAJPAlf03MAY02z+Ob8qFtC5VmpwbHoMMuZAXKYttKvJy/bGGGZlj57B0L78Dg810ZNIbd0fclndS+Zm4XU7eGa9druoYxtIQc8BDvX4uTh4Xy8icpWI7AZewWql24jInSKyXkTWV1WF97JgahQo32adTLTglpBnhj5TkoRHAkzPHF2DoX1V+qbjNF2ktB6wxeK8bi6ek8XLW0pp6/SHITs1mgykoIfquLO1wI0xfzPGzACuxOpPtz/ImIeMMYuNMYvT0vRyW+oYNv3V6jef+2lbqDMgvFCaxCfSG4l2O8OQ3OBp8mTQ5kogvTX0GuifWpxLU0c3b+woH+bM1GgzkIJeDIzr8XMuYD+9LcgY8z4wWURSTzI3NZb5u2DbMzBtOcTYr+LzblUctV0urs2pC0Nyg0yEqtipxHeUh7zm6NKJKeQmRfPMhkMhHqzUPw1klss6YKqITARKgOuBG3tuICJTgP3GGCMiCwEPUDPYyaoI1ncGTPU+aK22WughZsc8U5JMelQXy1Ka2BwYphyP09Kihwa8bVXMVMY1rCetZR/FCYt6xRwO4VOLcrnvnX0U17WOuAt3qJHjmC10Y0w38AXgDWAX8LQxZoeI3CUidwU3uwbYLiKbsWbEXGd0EQp1Miq2WRewSJpoC1V2uFhZHcc12XW4RtfU8351unw0ROWQ1rI35Jz0axbmYgw8t6EkDNmp0WJA89CNMa8Cr/a578Ee//858PPBTU2NWV1tUJMfvGaovX/8xbJE/Ea4JjsCult6qIqdytTalcR32PvKxyXHcMbkFJ7bWMyXLpiic9JVSBHSvlERpXKntRBX5pyQ4edLk5iX0MoUX2StF14bnYdf3KT1Mzh6zcJcimpb2XAwsj7I1ODRS9Cpkad8G8SmW1cm6mNno5ddTdH8aGbkdT0EHG5qYiaS0lqAy9925NJ6h3V2B3A5hB++tIMLZmbw1U9OC1OmaqTSFroaWVqqoLncWvc8hL+VJeGWAJdm1g9vXsOkMmYaTtPF1Jr3bDGPy8GUdB97K5vp9o/QkWAVVtpCVyNLxXZAQl6VqDsAL5Qmcl5aE8meyDzJpikqk3ZnHDMrX2FX+sW2+IzMOHaXN1FQ3WJrwfelLfixR1voauQwAajYCcmTwBNrC39Y46Oq083VETYY2osI1bFTGN+wjtiOSlt4XHIMsVFOdpU3hSE5NdJpQVcjR30RdDZZa7eE8HxpEonubs5Li+xiVhUzFcEwo/oNW8whwozMeA7WtNDa2R2G7NRIpgVdjRwVO6wTiVKm2EJN3Q7eqEzgssx6ohyRfYpDuzuBMt8cZla+FjI+IzOOgIG9Fc3DnJka6bSgq5HB3wXVe6yLWIS4CPRrFQl0BBxclV0//LmFwa70FaS17iO1xX41o9TgcsG7yuzLBKixTQdF1chQsw/8nZAeurvlhdJE8mI6WJDQaosdzyn2o8We1As5p+BeZla9xgexU23xGZlxfLCvmtqWTpJj7Rf+UGOTttDVyFCxA6LiIXG8LVTW7mZVrY8rs+pCraIbkdrdiRQmnsGMqtcRY5/RMz0jDgH26OCo6kELugq/zhaoPWBNVQxRsV8qS8QgXJlVP/y5hdGu9IvxdVYxrmG9LRYb5WJccgy7yxvRZZPUYVrQVfhV7gJMv7Nb/laayIKEFvJiO4c3rzA7kLyMDmfsUQdHG9u7KWtoH+bM1EilBV2FX+UO6zT/WPsS+ruavOxujh4zg6E9+R1R7E35BFNq3sPltxftyWk+XA5ht3a7qCAt6Cq8WmuhqQzSZ4cMv1CaiEtMxJ7qfyy701fgCbQyqfZ9W8zjcjA5zcfeiib8Ae12UVrQVbhV7rD+TZ9pCwWMtVTuuamRe6r/sRTHL6DJk87Mqv67XTq6AxTWtAxzZmok0oKuwscYa3ZLYh5E2S/0vLo2lvIOD1dG8qn+xyIOdqctZ0L9KqK77MdhfHIM0W6ndrsoQAu6CqfGUmivh4x+ulvKkvA5/XwibWyfQLMrbQVO42da9Vu2mMMhTM+Io6C6hY6usfktRv2TFnQVPpU7wOGCVPuqgO1+4bWKBJZnNOB1ju3+4ZrYKVTFTGFG1esh49Oz4vAHDPlVuhTAWKcFXYVHwA9VuyBlKriibOF3q+Jp6nZy5Ric3RLK7rQVZDdtI6Gt2BbLiIsiMdqt3S5KC7oKk7oC69qh/XS3/K0skfSoLk5P1lYnwO60CzEIM6rtrXQRYUZmHMV1bTS1d4UhOzVS6FouKjwqdoArGpIm2kL1nU5WVsXxmfE1OIMnjq46UDPMCY4szVGZFMcvZGbla6zJvd12Ru30zDhWF9Syt6KZRROSwpSlCjdtoavh191hLcaVPhMcTlv4lYoEuoxjbM9uCWFX+gqS2ovIaN5piyXGeMiM97K7fGwPII91WtDV8KveC4Hufk8merEskamx7cyO01Pae8pPOZ9u8Rx1Tnp1cyfVzR3DnJkaKbSgq+FXuQO8iRCfbQsdanOzts7HldljZ2XFgepwxXEgeRnTqt9CjP1qRVMzfIigg6NjmBZ0Nbw6mqDu4FFXVgS4fIytrDhQu9OWE9tVy/j6tbZYjMfFhOQY9pQ36QqMY5QOiqrhdWRlxd7dLasO1GAM/PXgZGZ46ykuK8c+QU8VJp1BtyOKZYW/Iatpuy0ecKfzm47ZlNS3hSE7FW7aQlfDq3IHxGVBTIotVNjho7QrlrPiy8OQ2Ojgd3iojp5IclshjoB9iuKi2GrcTl2BcazSgq6GT0s1NFf0Oxj6QVMmLgmwxFc1zImNLtWxU3GabpLbCm2xKEeAKek+9lU0065LAYw52uWihsUv39rLtXvWk4OwoS2Drj7zyv1G+LgpnYUx1fic9gE/9U9NngzanT7SWvOpDnm90Xh2lTXxzq5KLpmbFYYMVbhoC10NDxMgrTWfem8OXc4YW3h7axIN/ijOiq8IQ3KjjAjVMVNIaC/B7bdfNDs3KZrYKCd/21QShuRUOGlBV8Mit3ETUf5mqmPsLUqAD5syiHV0MT9mbJ8ROlBVsVMRDKmt+baYQ6wVGFfuqaS2ZWxdtm+s04KuhsXMylfxi5va6DxbrD3gZF1zGkt9lbgdOt1uINrdiTR70khrsRd0sLpdugOGV7aWDnNmKpy0oKuh19XG1Jp3qInOI+CwD9usa06lwzi1u+U4VcVMIbarhujOWlss1edhekacdruMMVrQ1dDb8ypR/haqQgzgAXzYlEmaq41p3oZhTmx0q46ZTAAhrXWfLSYiXLkgh41F9RRW6+Xpxgot6GrobXmKJk86jVH2GRe13R62tSaxLL4Ch57qf1y6ndHUe8eR2rofTMAWv3JBNiLwvLbSx4wBFXQRWS4ie0QkX0S+FSJ+k4hsDd4+FpF5g5+qGpWaqyD/bXanLQexv9w+asrAIJwVpycTnYjq2ClE+VuI7yizxbISojlrSirPbywmENCxibHgmAVdRJzA/cAKYBZwg4jM6rNZAXCOMWYu8GPgocFOVI1S258D42dX2gpbyBh4vzGTad4Gsjx6qvqJqPNOoFvcpLXYu10Arl6YQ3FdG+sK7f3sKvIMpIV+GpBvjDlgjOkEngSu6LmBMeZjY8zhxatXA7mDm6YatbY8AZmnUBM7xRYq7PBR3OljmZ7qf8ICDhc1MZNI6WcpgItmZxLjcfL8Ru12GQsGUtBzgEM9fi4O3tef24GQCzaLyJ0isl5E1ldV6endEa9iJ5Rthvk3hQwfPtV/qa9yePOKMFWxU3GarpBLAcR4XKyYk8Ur28p0KYAxYCAFPdRQVcgOORE5D6ug3x0qbox5yBiz2BizOC0tbeBZqtFpy+PgcMEp19pC3Ub4qCmDRbF6qv/JavJk0u6MI71lb8j4NYtyaO7o5s2dOi000g2koBcD43r8nAvYzlYQkbnAH4ArjDF6ut9Y5++GrU/D1IsgNtUW3tqSTKPfwzIdDD15IlTFTiO+oxRPt32VxaUTU8hO8PL8Rl2QONINpKCvA6aKyEQR8QDXAy/13EBExgPPA7cYY0I3E9TYsv9da2XF+TeEDL/flEmcs5N5sTpYNxispQAIOTjqcAhXLczh/b1VVDbqZf0i2TELujGmG/gC8AawC3jaGLNDRO4SkbuCm30fSAEeEJHNIrJ+yDJWo8OWxyE62Wqh99Hkd7GhJZWz4ipwiU6nGwwdrjgaorKsk4xCXK3omoW5BIzOSY90A5qHbox51RgzzRgz2Rjzk+B9DxpjHgz+/w5jTJIxZn7wtngok1YjXFsd7H7V6jt3eWzhj5oy6DYOzom3z51WJ64qdhrR3Y1kNW21xSal+Vg8IYln1h/Sy9NFMD1TVA2+7c+DvwPm3xgyvLIxi4lRTUyI0lPSB1NN9ET84mJ25csh49cuzmV/VQubDtUPb2Jq2GhBV4Nv8+PWRaCz7CcMF3b4ONgRp63zIRBwuKmJnsi06rdw+e195ZfMzSba7eSZ9YdCPFpFAi3oanBV7oKS9dbcc7HPeP1HozX3/Mw4nUI3FKpipxHlb2Fqzbu2mC/KxYpTMvn7ljLaOnVOeiTSS9CpwbXxMXC4Yd71tlC3ET5s1LnnA7G06MRWz2iMyqLem8vsiheBr9ji1y4ax/MbS/jKk5uYkRXf736++slpJ/T8Kry0ha4GT3cnbH0Spq8IOfd8Q3MqzQEP52p3y9ARYXv65Yxr3Ag1+23hJROTGZ8cw46yxjAkp4aaFnQ1ePa8Cq01sPDWkOGVjVkkOTuYG6Nzz4fSzvRLCOCATX+xxRwO4VOLcimua6Ohzb72ixrdtMtFDZqCtx4k1ZPOwwW5mMLe55c1tXexpTWZK5IO6rrnQ6wlKp3CpDOYtPlxOO8ecPZ+m39qUS73vrWXnaWNnD45JUxZqqGgLXQ1OBqKyatfzY70SzHitIV3lDZiEM5L0O6W4bA943JoLof8t22x7MRo8lJi2FHWoOukRxgt6GpwbH4cwbAz4zJbKGAMO0obOSWmlnS3nno+HAqSlkFsGmx6LGR8Tk4CLR1+Cmv0XIBIol0u6uQFArDpMRqisplZ+aotvLklmeaOeZyfqVegHy4Bhwvm3QCrH4CmCojL6BXPS4kl1uNke2kjk9J8YcpSDTZtoauTt/9dqC+iwjcjZPjdhmzinZ0s9lUPc2Jj3MLPQKA7ZCvd6RBmZcdTWN1CU7sOjkYKbaGrk7f+jxCbRm10ni1U3+1hY0sKK5KKdSGuYfbLTQGuSTiVhI/+wJ/aL7GNbczOTmBdYR07SxtZMkkHRyOBttDVyWkogb2vwYKbQw6G/qMxEz8Ozo/X7pZw2Jp5DQkdZUyoW22LJUS7GZ8cw/bSRgK6YFdE0IKuTs7GR63lWhd+xhYKGKu7ZVZ0nV4EOkz2J59DizuZueXPhYzPyY6nuaObgzWtw5yZGgpa0NWJ83fDxj/DlAsgeaItvLk1hcruaD6ZoGtwh0vA4WJ7xhVMrPuIuA771aEmpfmI8TjZWlw//MmpQacFXZ24va9DUxks+lzI8Fv1OSQ6O3QwNMy2Z1yJYJhT/oIt5nQIc7ITKKxp1TNHI4AWdHXiNvwJ4rJg2nJbqKLLy5bWZC5IKNXB0DBr9GZTkHQGcypexBGwL4p2Sk4CIrCtuCEM2anBpAVdnZia/dZZiAs/Yzu1HODthhwEw/kJOhg6EmzLvBpfVzWTa1faYj6vi8mpPnaUNtDtDwx/cmrQaEFXJ2bt761lchfbu1s6Aw5WNmRxqq+aZFdnGJJTfRUknUm9N4f5ZU+HjM/NTaC9O8DeiuZhzkwNJi3o6vh1NFkr+c2+CuIybeFVzek0B9w6GDqCGHGyJfNachs3kda8xxbPTYomOdbD1pL64U9ODRot6Or4bX4COptgyV22kDHwZn0OOZ4WZkXXD39uql87Mi6ny+FlftlTtpiIMDcngYrGDsobdL2d0UoLujo+gQCs/R3kLIbcRbbwnvYEDnTEc1FCcagr0Kkw6nDFsTP9EmZUvUF0V50tPjMrHo/TwWa9iPSopQVdHZ/970JNfsjWOcBrdbn4HF2cHW+f86zCb3PWp3GZzpBTGD0uB7Nz4tlX2URZg54INhppQVfHZ82D4MuAWVfYQhVdXta1pHFBQilRDp0tMRLVxkziYOIS5pU/G3IK4/zcRIyBP398MAzZqZOlBV0NXNUeyH8LFt8GLo8t/Hp9Lg4MFyYWhyE5NVCbsq4jrrOSqTXv2GLx0W6mpPt4fM1BWjr0Qt6jjRZ0NXAf/wZcXjj1Dluosb2LlQ1ZnB5XqVMVR7iCpDOpic5jUclj1ih2HwvGJ9LY3s2zG/SDebTR5XNVb+/9NPT9HU2w+XFr3nlsqi381NpDtBsXFyceGuIE1bEsLXqo39jq8XeCONiQczMX5v8X4xvWUpS4pNc2WQnRLByfyJ8+KuDmpRNw6kVgRw1toauBKdkAJgCn/7st1OUP8MjHhcyIrmeiV09MGQ12p62gxZ3C4pLQl6i7Y9kkCmtaeXtXxTBnpk6GFnR1bN0dULoJ0qZD8iRb+MXNpZTUt3F5kg6kjRZ+h4dN2dczoX5NyBONLpyVwbjkaB5YuR+ja6WPGlrQ1bGVbQZ/B+QusYX8AcMDK/OZmRXP/Jja4c9NnbCtmdfQ6Yix+tL7cDkd3HXOZLYcqmfV/powZKdOhBZ0dXQBP5Ssh4TxEJ9lC7+5o5wDVS3827mT9USiUabDFcfWzKuYXv02ce1ltvg1C3NJj4vi/pX5YchOnQgdFFVHV7HNGhCdtsIWMsbwwMr95KXEcPEpWaxdH4b81HHpO2AqJgAYLt19NwXJZ1mDpkFet5PPL5vET17dxaaiOhaMTxrmbNXx0ha66l/AD0WrrDXPk+xXJPpgXzXbShr413Mn60yIUarT5aMydjrpLXvwdNsHtG9cMp6EaDf3v7c/DNmp46UtdNW/iu3Q3gBTPkmo/pT738snK8HLVQtyw5CcGiwl8fNIb9lDTtOWI/f98q29R/4/IzOOt3dVcM/ftpHqiwLgq5+cNux5qmPTFroK7XDr3JcJyZNt4Y/zq1lTUMvnl03C49KX0WjW6YqjKnYa6c278XXYpynOH5eI2ymsLdBB75FuQO9EEVkuIntEJF9EvhUiPkNEVolIh4h8Y/DTVMOucge018OEM22tc2MM/++tvWTGe7lxyfjw5KcGVUn8fMCwuORRW8zrdrJgXBL7KpupauoY9tzUwB2zoIuIE7gfWAHMAm4QkVl9NqsFvgT8YtAzVMPPBIKt8wxImWILr9xbxYaDdXzxgil43c4wJKgGW4crjurYqZxS/gKxHVW2+MLxiUS5HKw6oFMYR7KBtNBPA/KNMQeMMZ3Ak0CvpfaMMZXGmHWAXjY8EpRvg7a6flvn9765l3HJ0Vy7aFyYElRDoTh+AYKfJcUP22JRbicLJyRRUN2iS+uOYAMZFM0Bei7QUQzYzzAZABG5E7gTYPx4/ao+Ivm7oPBDiMuGlKm28Js7K9hW0sAvrp2nfecRpsMVz7aMq5hb/jc2Zt9AffSEXvH5uYlsLqpn1YGaXoOmfemAafgM5B0Zaj7aCZ0LbIx5yBiz2BizOC0t7UR2oYZayQbr8nKTzrW1zv0G7n1zL5NSY7lyfnZ48lNDas242+l2eDjz4IO2mMflYHFeEodq2yiuaw1DdupYBlLQi4Ge361zgdKhSUeFVWut1XeePBkS7d+gnitJYk9FE1/95DRcTm2dR6JWTyobc25iWs3bZDTtsMXn5iTgi3LxYX61rvEyAg2ky2UdMFVEJgIlwPXAjUOalQqPD++11myZdK4t1NLt4L93p5MZ72VfRVPIr9xLhyFFNfQ2ZN/E3PLnOOvgb3lu9gO9vqm5nA5On5zCWzsr2FPRxIzM+DBmqvo6ZjPLGNMNfAF4A9gFPG2M2SEid4nIXQAikikixcDXgO+KSLGI6F96NKkvgjUPQcYpEGvvDvtdQRr1/ijOnpaK6KItEa3T5WNN7u2Mb1hPXv3HtvjMzDjS46L4KL+GLr9eanAkGdCZosaYV4FX+9z3YI//l2N1xajR6s3vgjggb5ktVNrm5qHCNE73VXBVw3vQEIb81LDamnk188ue4pyCX1KUcBoBh/tITEQ4e2oaz24sZmNRHUsmpoQxU9WTdoQqOLASdr4Iy74OXvsXq1/kZxIAbkg9MOypqfAIONysnPh1ktsOsqD0SVs8JymayWmxrC+so1mvPTpiaEEf6/xd8Oo3ISkPzviiLbyhLobnS5O4bUI1ae724c9PhU1h8pnsT1rG0kN/CHmy0VlTUjHGWgZCjQxa0Me6Nb+D6j2w/Ofg9vYKdQXgOztzyPJ28sVJlWFKUIXTPyZ+DYfpZtnBX9tiiTEeFoxPZFd5k05jHCG0oI9lTeWw8mcw9UKYvtwWfvhgGnuao/nhjFJiXTr4NRY1ROeyPudmZla9Tk7DRlv8tInJJES7eWd3Jd0BfY2Emy6fG4ne++nR4+d9G4yBV74O/k5Y/jPbJofa3PwqP4NPpDVwUUbjECWqRoN1uZ9jZtVrfGL/f/OX+X/F74g6EnM7HZw3PY0XNpdSvuVtPpVSCO+FGCQ979vDl/AYpi30sWrnC7D7ZeuNltJ7eVxj4Ie7cnCI4T9n6jlkY12308vbk79DcttBlhb93hafkBLLtAwfL9ZNoLQzOgwZqsO0oI9FLTXwyjcgaz6cbh8IfakskXeq4vnqlApyonW9NQVFSUvZnn45i0v+QnrzLlv87KlpRImf31fMwK8nkIaNFvSx6PVvWVciuvIBcPbudStrd/O9XdksTGzhc+N19oL6p/cnfoVWdxIX7vsxjkDvD/rYKBe3pOWzuz2RhwtTw5Sh0j70saZ6L+x4Hs79NmTM7hW69809vLs2gw4/3JKwjXWFukyq+qcOVxzvTP4WV+z+BksOPcyqCXf1ip8dV86G5lT+Z18m8e3FTIhqORJb3f3PpSJ0Ncahoy30saS9Efa8Cplz4ayv2cJbihvY3pbMzWn7yfRoMVd2B1LOYWfaJZxW/CdyGjb0ionAHel7iHV0cX/5LLoCukTEcNOCPlaYAOx+yfr32kfA5ekV3lvRxIf51cyPqeGCeB0IVf17d9J/0ODNZcXe7+Ptqu8Vi3d1cWfGHg51+ni6ZlJ4EhzDtKCPFQc/goZia855n1ktzR3d3PWXDXicDu7M2N13GXSleulyxfLq9J8Q3VXHhft+ZE2L6mFhbA2fSCjh5frxrG/W/vThpAV9LKg7CAc/how51q0HYwx3P7uVgzWtXHxKJkmuzjAlqUaTSt8MPsj7EpPrPmBB6RO2+C2p+UyMauSBipmU61TGYaMFPdK11VtzzmOSrdZ5H3/8qJBXtpXxzYumk5sUM+zpqdFrc9Z15Cefy9mFv2Zc/dpeMY8jwFeztuPEcG/ZHF1md5hoQY9k3R2w4znAwOxrwNm733zV/hp++uouLpyVwZ1na3+nOk4ivD71h9TG5HHpnm/j7eq9rnKau4MvZO6kuDOWd3ZV6hWOhoFOW4xUxsDuv0NLNcy9zmqh97Cvool/eWw9eamx/OLT8/SiFQqApUUP9RtbPf5O231drlhenPn/uHHLZ5he/SbbM67A7/hnw2FebC2fTingqYpJJES7OX2yrp0+lLSgRyJj4MC7UJMPUz5pLY0btOpADe+07OSp9YfwBwzLpqTy8AcF4ctVjXqN3hz+PuPnfGr7vzKt+m12p12EEeeR+BVJB9nmmc/awlrivFpyhpJ2uUSioo+heB3kLILshb1CbQEnL24ppb3LzxXzsomPdvezE6UGriRhEQeSlpHYUcLUmves6bFBInD+jHQmpMTw7p5K3tutSzEPFS3okWbN76DwA+ss0Mmf6HWB35ZuB/9TOpfq5g4unpNFerz3KDtS6vhU+aZTmLiUlLYCJtd90Gs6o9MhXDwni1RfFP/61w18vF+XlRgKWtAjycbH4LVvQspUmH6JrZh/bmMee9viWT47k7zU2DAmqiJVWdwpHIpfSHrLXvLqV/Uq6h6XgyvmZTMuKYbbHlmnVzoaAlrQI8Xq/4OXvgCTzoNZV1gXfA46XMw31MfyhcxdTMuIC2OiKtIVxy+k1HcKWc07mFz3fq/ul9goF0/cuZTxyTHc9ud1fKRFfVDpCMVINZCLVIDVAlr5U/jHz2HmZXDNw/DBvUc2q+xwccfGPLY3RnPf3CJS2ypZPYRpK4UIBxOX4He4Gde4EWegk3Xjbjsy+yXVF8UTn1/Kjb9fw22PrONX181nRfUj/e/vZC6OMdD3UYTQFvpo5u+CV75mFfP5N8GnHgHXP68mk98cxdVrprCvxcvvFhzksqyG/vel1GASoThhEQWJS0lpK+TKnV8mqsc89RRfFE/cuZTZ2fH82+Mbeaggte8KAuoEaEEfrZqr4NErYP0f4cwvw+W/7bW2+Uc1Pq5eM5l2v/DUqfv5ZLpeRk4Nv/K4U8hPPoecxs3cuPUzpLTkH4klx3p4/PNLuXhOFv+9N5vv7sqhU1doPCna5TLKrDpQQ2xnNXmrzyKmq463pv4nu7kY3rHeKAFjKN0cx/O1eWR7Wrk7eystNe2sqglz4mrMqoqdxvqcW7hs991cv/U2mPYQzLocAK/byW9uWMC45i08WJDOjkYvv5lXxDi9UtYJ0Rb6aGICZDduZk7Fi2AMT53ye3anX3wk3NLRzQubS3iudiJnxlXwX+M2kOZuD2PCSlnK4ufx13mPUhMzCZ6+Bf7+ZehoBsDhEL41rZwH5h1kf4uXSz6eyhsV8WHOeHTSgj5atNXD5seZ0LCOuugJ/HX+X6j0zQSsFRN3lTXyl9UHKa1v58703fxbxi68Dn94c1aqh5aodJ455SGri3DDn+HBs6BozZH4xZkNvHz6PibEdPIvm/P42rZc6jqdR9mj6ku7XMLpWCPwAIFuOLTWOvtTnOxLPpfqmCm0uxMBaGjr4t3dlRTVtpKV4OWCGemcV/vuUXd5tPU6lBoKvV5zk1Jg3g2w5xX444XWxconng3uGCbEdPLskv38Zn86Dxak84/qeL6fUcLl87J1vaEB0II+UhkDNftg/7vQXg+p02Hy+VSXdgPQ3uVnXWEtWw414HDAudPSmJubYL3oa8ObulLHlDgeFt0GhR9CyXqo2g15yyBrPlEOJ9+YWsGlmQ3cvSOXLz+5mUdXHeQ7F89g0YTkY+97DNOCPtIYA7X74eCH0FQOMSkw9/ojC2y1+ut5qyGHvxUU0tEdYGZWHKdPSiHOq2uyqFHGFQVTLoCsuZD/NuS/BcVrYfwZkDGHGXHtPL8kn6dib+KXb+/lmv9bxUWzM/jSBVOZnZ0Q7uxHJC3oI0Wg22qlFK+H5nLwJsC0FdYVhhxOKtpd/PFgKo8WzaIt4GJCipczJ6eSFhd17H0rNYKsOtB3ypUDfJ8k0VXMuMYN+Pa+Rvv+DyiPm01l7HQq8tq5dlEum4rq+Si/hjd2VHDWlFT+5ZxJnDUlVbtietCCHm4t1VCxHcq3QlcrRCcfKeRdOHmvOp6ni5N5rzoOY2CJr5LLkoqomHZjuDNXavCIUB89jnpvLonth8hp3Exe/RrGNawnuruBnemX4c47hQdvWcTja4r400cF3PLwWiamxnLt4lw+tTBXF5sDJFxXEVm8eLFZv359WJ47rIyB6n2w51VYfT80VwICKVMgZxGd8Xk8vNOwrjmN9S2pNPo9JDo7ODu+nPPjS8nw6DRENTbEdNaQ2byD1NZ8nMZPu9OHN3sOpE6jIzaLl8uTeKokmbV1PpwOYemkZNxOB5PTfPiirLZq3wkAp0/qc4GNUXjqv4hsMMYsDhXTFvpwaKqwZqkUfgj73oL6g9b9cdkEJn+CAzHz+KApk48K4lhdG0uz34lXulkQW8MZcZXMj63BJXpetBpbWj0pHEg+m8LEpSS3HSS1dT/eQ2vg0Gqi3NFckzyJa8ZPoOiae3gq38lr2ys4UN3Cyj1VpMVFMT4phujuZKZHN4yZKbzaQh9MxlhdKFW7oGyLdSvZaA1yAsYdQ2vOmRxIPJOPHQv5cE8ZmxtiaOq25tpOiO7gzJRmcvwlzImpw+PQC+sq1dPp42KgrgBq9kPdAehqswK+TEzuIt6pz2Jj13hWN2eypTEGv3EgGHI9LUzxNnJBboAZvnam+dqJdwciroU+oIIuIsuB+wAn8AdjzM/6xCUYvxhoBT5rjNl4tH2OyoJujDWFsKkcmsqgsRTqDlot7toCTPVepL3+yOY1zjTynZPZxAw+7p7GmvZxdASs4i0CEzxNTPE2MsXbyMzoetL1rE6ljqpXl4kx0FptXcylaBWUbsbU5CNYNa3T4aVB4ikhg/3+DLZ351AQSKfSJFFhkvBERTE+O5u8lFjGp8SQleAlKyGazAQvqT4PvijXiBxwPakuFxFxAvcDnwSKgXUi8pIxZmePzVYAU4O3JcD/Bf8dfMZY6yv3vQX8mOAt4O/GGD/+7m7wdxEI3kxXB4HuTgLdHQS62jHdHQQ62zCdrQQ6WzGdrdDRBJ3NSEcTjo4GHJ2NuDrq8XTU4e2qw2m6e6Xjx0G1pFBk0tnTvYj9Jpt8k8OOQB61xON0CInRbhJj3cxJ9ZDis27JMR7OKvnDkBwipcYEEYhNg1Nvt27A/a9tIq11HymtB0huLWRS3QdM7z7IXHZwjav3N94AQlN5PDVlcVQFfDSaGEqJYbeJpZUo2iUGR5QPR1QMzqgY3FGxuDzRuKO8eKK81r8eLx5PFG6PB4/bg9vjxu1y43a5cLrduJ0unC6X9bPTeeQWE+UmJmrwpxoPpA/9NCDfGHMAQESeBK4Aehb0K4BHjdXcXy0iiSKSZYwpG+yEN77+CAvXfCVkTIK3w+sZnMjh8huhBS/NRFNrYmkklkYTR43Jpo446iSBRlcaLVGptHoz6I7NJi42msQYN6m+KKbERVG1v4aLolz4vC6i3c4R+SmvVCTqcsVSGj+f0vj5AHQU+ayAMXj8LSzKdFlryHQ24ehsJSFtOgmtNUxoqaG7tR7TVomjsxFnd6vVeOvGurUMbp6rsm7m9H+5f3B3ysAKeg5wqMfPxdhb36G2yQF6FXQRuRO4M/hjs4jsOa5s/ykVGMJLnZzwuuFDnNdJGam5aV7HR/Oy+c7RgsfI6/VBzmWgHkjlrgdO9HhN6C8wkIIeqnnZt+N9INtgjHkIOOmFRERkfX99SOE0UvOCkZub5nV8NK/jM9byGshqi8XAuB4/5wKlJ7CNUkqpITSQgr4OmCoiE0XEA1wPvNRnm5eAW8WyFGgYiv5zpZRS/Ttml4sxpltEvgC8gTVt8Y/GmB0iclcw/iDwKtaUxXysaYufG7qUgUHothkiIzUvGLm5aV7HR/M6PmMqr7CdWKSUUmpw6RWLlFIqQmhBV0qpCDHqC7qIfENEjIikhjsXABH5sYhsFZHNIvKmiGSHOycAEflfEdkdzO1vIpIY7pwARORaEdkhIgERCfv0MhFZLiJ7RCRfRL4V7nwOE5E/ikiliGwPdy6Hicg4EXlPRHYF/4ZfDndOACLiFZG1IrIlmNd/hjunnkTEKSKbROTlwd73qC7oIjIOa0mConDn0sP/GmPmGmPmAy8D3w9zPoe9BcwxxswF9gIjZVWi7cDVwPvhTqTHMhcrgFnADSIyK7xZHfEIsDzcSfTRDXzdGDMTWAr8+wg5Xh3A+caYecB8YHlw9t1I8WVg11DseFQXdOCXwDcJcRJTuBhjGnv8GMsIyc0Y86YxRxaiWY11rkDYGWN2GWNO9IzhwXZkmQtjTCdweJmLsDPGvM8Iu1qsMabs8CJ8xpgmrCKVE96swFiagz+6g7cR8T4UkVzgEmBIFnIatQVdRC4HSowxW8KdS18i8hMROQTcxMhpofd0G/BauJMYgfpbwkIdg4jkAQuANWFOBTjSrbEZqATeMsaMiLyAX2E1QodkbewRfYELEXkbyAwRugdrAYcLhzcjy9HyMsa8aIy5B7hHRL4NfAH4wUjIK7jNPVhflf86HDkNNK8RYkBLWKjeRMQHPAd8pc831LAxxviB+cGxor+JyBxjTFjHH0TkUqDSGLNBRM4diucY0QXdGPOJUPeLyCnARGBLcCXDXGCjiJxmjCkPV14hPA68wjAV9GPlJSKfAS4FLjDDeALCcRyvcNMlLI6TiLixivlfjTHPhzufvowx9SKyEmv8IdwDymcCl4vIxYAXiBeRvxhjbh6sJxiVXS7GmG3GmHRjTJ4xJg/rjbhwOIr5sYjI1B4/Xg7sDlcuPQUvUnI3cLkxpjXc+YxQA1nmQgUFL2zzMLDLGHNvuPM5TETSDs/iEpFo4BOMgPehMebbxpjcYM26Hnh3MIs5jNKCPsL9TES2i8hWrC6hETGVC/gtEAe8FZxS+WC4EwIQkatEpBg4HXhFRN4IVy7BQePDy1zsAp42xuwIVz49icgTwCpguogUi8jt4c4Jq8V5C3B+8DW1Odj6DLcs4L3ge3AdVh/6oE8RHIn01H+llIoQ2kJXSqkIoQVdKaUihBZ0pZSKEFrQlVIqQmhBV0qpCKEFXSmlIoQWdKWUihD/H52ICZ0jDBEgAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "from scipy import stats\n", + "import matplotlib.pyplot as plt\n", + "\n", + "# Indeed, the distribution $X$ is stochastically greater the distribution $y$\n", + "X = stats.norm(loc=+0.2)\n", + "Y = stats.norm(loc=0)\n", + "x = X.rvs(size=801)\n", + "y = Y.rvs(size=399)\n", + "\n", + "grid = np.linspace(-4, 4, 100)\n", + "plt.plot(grid, X.pdf(grid), 'C0')\n", + "plt.plot(grid, Y.pdf(grid), 'C1')\n", + "plt.hist(x, density=True, color='C0', bins=30, alpha=0.5)\n", + "plt.hist(y, density=True, color='C1', bins=30, alpha=0.5)\n", + "plt.title('Distribution PDFs and Sample Histograms')\n", + "plt.legend(['x', 'y'])\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "d10f5fe3-43f8-46e9-8347-01b926170eaa", + "metadata": {}, + "source": [ + "Our first difficulty is determining the correct value of `alternative` to pass into `ks_2samp`. From its [documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html), we see that the alternatives are expressed not in terms of the values of the samples or the location of the underlying distributions, but in terms of _cumulative density functions_ of the underlying distributions. \n", + "\n", + "> - `two-sided`: The null hypothesis is that the two distributions are identical, $F(x)=G(x)$ for all $x$; the alternative is that they are not identical.\n", + "> - `less`: The null hypothesis is that $F(x) >= G(x)$ for all $x$; the alternative is that $F(x) < G(x)$ for at least one $x$.\n", + "> - `greater`: The null hypothesis is that $F(x) <= G(x)$ for all $x$; the alternative is that $F(x) > G(x)$ for at least one $x$.\n", + "\n", + "Note that if a distribution $X$ tends to be greater than $Y$, we find that the cumulative distribution function of $X$ lies _below_ the cumulative distribution function of $Y$." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "5ebe1d2f-f067-46f1-a415-986b19b933be", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwa0lEQVR4nO3dd3xUZdbA8d/JpJGEmoQaekeaCojYwIp9dVcXGxZc1lVsq6vuWnfdte+ua2VZe0VRfAVFsNBUpPdODCSETkggIX3mvH/cAZNMIAEmuZPJ+fqZD3Pn3LlzGGcOzzz3uc8jqooxxpi6L8LtBIwxxgSHFXRjjAkTVtCNMSZMWEE3xpgwYQXdGGPChBV0Y4wJE1bQTa0RkbEi8nCQjtVORPJExOPfnikiNwfj2P7jfSUi1wfreMbUBivoJihEZJOIFIhIrojkiMgcEblFRA5+xlT1FlV9vJrHOvtw+6hqhqomqKo3CLk/JiLvVTj++ar69rEe+xCvN0hEpvjfpz0iMl9EbvTHhoqIz/+PVZ6IZIrIxyIysMIxVET2l9kvpyZyNXWLFXQTTBerakOgPfAUcD/werBfREQig33M2iIiJwPTgVlAFyAR+ANwfpndtqpqAtAQGAysBb4XkbMqHK6f/x+1BFVtUuPJm5BnBd0EnaruVdVJwG+B60WkN4CIvCUif/ffTxKRL8q0Ur8XkQgReRdoB0z2tzzvE5EO/hbpKBHJAKaXeaxsce/sb+3uFZHPRaSZ/7WGikhm2RwP/AoQkeHAX4Df+l9vmT9+sAvHn9dDIpIuIjtF5B0RaeyPHcjjehHJEJHdIvLgYd6eZ4G3VfVpVd2tjkWqemUl76OqaqaqPgK8Bjxd1Xt/qPe1queZ8GD/o02NUdX5QCZwWiXhe/yxZKAFTlFVVb0OyMBp7Seo6jNlnnMG0BM47xAvORK4CWgNlAIvVCPHqcATwEf+1+tXyW43+G/DgE5AAvBShX1OBboDZwGPiEjPigcRkTjgZOCTqvKqxETgBBGJr2K/St/Xo3g9UwdZQTc1bSvQrJLHS4BWQHtVLVHV77XqiYUeU9X9qlpwiPi7qrpSVfcDDwNXHjhpeoyuAf6lqmmqmgf8GRhR4dfBX1W1QFWXAcuAyv5haIrzndt2FDlsBQRoUuaxxf6WeI6IHPjH62jeVxMmrKCbmtYG2FPJ488CqcDXIpImIg9U41ibjyCeDkQBSdXK8vBa+49X9tiROC3gA7aXuZ+P04qvKBvw4RTcI9UGp6WdU+axE1S1if92h/+xo3lfTZiwgm5qjH9kRhvgh4oxVc1V1XtUtRNwMfDHMif9DtWirKql2bbM/XY4rdXdwH4grkxeHpwuieoedyvOid6yxy4FdlTxvHJUNR/4Cfj1kTzP7zJgsf/Xx+Fe43DvqwlzVtBN0IlIIxG5CBgPvKeqKyrZ5yIR6SIiAuwDvP4bOIWy01G89LUi0svfV/034BP/sMb1QKyIXCgiUcBDQEyZ5+0AOhzm5OGHwN0i0lFEEvilz730KHK8D7hBRP4kIokAItJPRMZX3FEcbUTkUeBmnP7ww6rifTVhzgq6CabJIpKL0/XxIPAv4MZD7NsV+BbIw2m1vqKqM/2xJ4GH/H3D9x7B678LvIXT/REL3AHOqBvgVpyRIltwWuxlR71M8P+ZJSKLKznuG/5jzwY2AoXA7UeQ10GqOgc4039LE5E9wDhgSpndWotIHs57swDoAwxV1a+r8RKHe19NmBM7X2KMMeHBWujGGBMmrKAbY0yYsIJujDFhwgq6McaECdcmOUpKStIOHTq49fLGGFMnLVq0aLeqJlcWc62gd+jQgYULF7r18sYYUyeJSPqhYtblYowxYcIKujHGhAkr6MYYEyZCauWXkpISMjMzKSwsdDuVQ4qNjSUlJYWoqCi3UzHGmHJCqqBnZmbSsGFDOnTogDO3UGhRVbKyssjMzKRjx45up2OMMeVU2eUiIm/4l91aeYi4iMgLIpIqIstF5ISjTaawsJDExMSQLOYAIkJiYmJI/4IwxtRf1elDfwsYfpj4+TgzvHUFRgOvHktCoVrMDwj1/Iwx9VeVXS6qOltEOhxml0uBd/zLXM0VkSYi0kpVj2aZLWOMqR5V8JZAST6UFEBpIZQWgbcISotRbxHFRcUUFxdSUlJMSXExpaXFeEtL8XpL8JZ68XlL8fmcP9Xnw6c+1OdFfT5UFVUf6vOBHthW53VVcf7z54E6y6Soz0mtXJ6g+CgbSOh2Gr1Pvyzob0kw+tDbUH7pr0z/YwEFXURG47TiadeuXRBe2hhT56lCYQ7s2wZ5O2D/LueWvwcK9uDLz6Zkfzal+XuhcC9Ssh9PyX6ivAVEHGbtDsFZxSTmkHvULp/+8ut+nrcEQrSgV9YHUekk66o6DmcyfwYMGGATsRtTX3hLIXsT7F4HWanO/exNkJMB+7Y6reyKTyGCfSSQ7YtnH3Hs0zj2kcx+bUc+MRRHxKJR8RDVAE90HBHRDYiIiiUy2n+LiiYqOoaoqBgio6Od7ahoPJFRREZG/nLzRBHh8RAVGUWEJwJPRAQRER48kR4iRIiIiDh480gE4hEE53ER575ECCIgRCARTkkUfumide47fy8R4eQaepuDUdAzKb+WYwrOGox1zoIFCxg1ahTz58/H6/UyaNAgPvroI3r37u12asbUHaXFsH05bFkE25bD9mWwax14iw/uUhLdhKzoVmz2tSbV14vUkkbs0Kbs1CZkSWOiG7WgadMkWjWNp02TWFo0jqVFw1jaNowhMSGaxPgYGkR7XPxLhqZgFPRJwBj/mognAXuD0X/+18mrWL113zEnV1av1o149OLjDhkfOHAgl1xyCQ899BAFBQVce+21VsyNqUppEWyeD2kzIH0ObF3i9GcDxCVRlNyb9A7XsLSwJTOymvJjTlP2FcYTIdCleQI92jaie8uGnNQ8gU7JCbRt1oCYSCvWR6PKgi4iHwJDgSQRyQQeBaIAVHUszlqIFwCpQD6HXkOyTnjkkUcYOHAgsbGxvPDCC26nY0xo2r8b1k2BtV/CxtlOl4l4oPXx6IBRpMf3ZlpOWyalKavW5QLQKDaSQR0TGTOkKf3bNuW41o2IjwmpS2HqvOqMcrmqirgCtwUtI7/DtaRr0p49e8jLy6OkpITCwkLi4+NdycOYkFOUC6s/h2XjIf1HZ0RH43bQ/xroPIyf4/ozcU0uXyzfRnpWPhGyjxPbN+W+4d05vWsyPVs1whNhw35rkv3zWMHo0aN5/PHH2bhxI/fffz8vvfSS2ykZ467MhTD/f04xLy2AZp3htHuh50UUNDuOL1ZsY/yMzSxKX4onQhjSOZHbhnbh3ONa0CQu2u3s6xUr6GW88847REZGcvXVV+P1ehkyZAjTp0/nzDPPdDs1Y2qXtxRWTYS5r8LWxRDdEPqNgP5XQ8pAduYV8c6cdN6dO529BSV0So7noQt78qvj25CUECoDBesfK+hljBw5kpEjRwLg8XiYN2+eyxkZU8u8JU6XyvfPOcMKk7rBBc85xTymIVtzCnjp/1byyaJMSrw+zuvVkhtP6cCgjs3sKuoQYAXdGONc3LPyU/jub5CTDq36w4gPodtwiIggK6+IV75Zzbtz01FVrhjQlt+d1omOSXaOKZRYQTemvsuYB9P+AlsWQss+cPUE6HoOiFDi9fHODxt5/pv17C8u5dcnpHDn2V1JaRrndtamElbQjamvCrLh64dgyXuQ0BIufcXpWolwxoDPTcvi0c9XsW5HLmd0S+bhi3rSpXlDl5M2h2MF3Zj6RhVWfQZf3Q/5WXDKnXDG/RDtdJ/sLyrlya/W8N7cDNo0acC4607knF4trI+8DrCCbkx9UpADX/7R6S9v1R+u/QRa9TsYnr9xD/dOWMbm7HxGndqRe8/tbpfY1yFW0I2pLzLmwac3w74tMOwhOPVu8DglwOtTXpy+gf98t4G2TeP4aPTJDOrYzOWEzZGygm5MuFOFOS/Ct49B4xS4aRq0HXgwvDuviLs/Wsr3G3Zz+fFtePxXve2S/DrK/q8ZE86K82HyHbBiAvS6FC55CWIbHQwvz8xh9DuLyM4v5ulf9+HKAW2tr7wOs4JexsMPP0xSUhJ33nknAA8++CAtWrTgjjvucDkzY47C3kz48CrYvgLOfBhOu+eXSbmBqSu3cddHS0lKiGHirUM4rnVjF5M1wRC6Bf2rB5wPYjC17APnP3XI8KhRo7j88su588478fl8jB8/nvnz5wc3B2Nqw47V8N6voTgPrv4Iup13MKSqjJ2VxtNT13J8uyb8b+QAu1w/TIRuQXdBhw4dSExMZMmSJezYsYPjjz+exMREt9My5shs+hHGXwWRDeDGr6DlL3P6+3zK375YzVtzNnFR31Y8d0U/YqNsFEu4CN2CfpiWdE26+eabeeutt9i+fTs33XSTKzkYc9TWfQUfXw9N2sF1E50//Uq9Ph6YuIJPFmUy6tSOPHhBTyJsOtuwEuF2AqHmsssuY+rUqSxYsIDzzjuv6icYEyrWfgkfXQctjnNGspQp5sWlPu4Yv4RPFmVy19ldeehCK+bhKHRb6C6Jjo5m2LBhNGnSBI/HfoqaOmLNZJhwg3Ox0HUTIfaXE5ylXh+3f7iYaat28NCFPbn5tE6upWlqlhX0Cnw+H3PnzmXChAlup2JM9az90inmrY+Haz8tV8y9PuWeCcuYtmoHj17cixtP6ehenqbGWZdLGatXr6ZLly6cddZZdO3a1e10jKnaxu9hwo3+y/jLt8x9PuUvE1fw+dKt3D+8hxXzesBa6GX06tWLtLQ0t9Mwpnq2LXPGmTfrCNdMKHfBEMBTU9fy0cLN3HFWV/4wtLNLSZraFHItdGfN6dAV6vmZeiLrZ2eceWxjp2UeV37elbfnbGLc7DRGntyeu8+2X5v1RUgV9NjYWLKyskK2aKoqWVlZxMbGup2Kqc/y98D7V4DPC9d9Bo3blAt/vWo7j01exTm9WvDoxcfZpfz1SEh1uaSkpJCZmcmuXbvcTuWQYmNjSUlJcTsNU195S2DC9bB3M1w/GZK7lQsv3ZzDHeOX0DelCS+MOB6PDU2sV0KqoEdFRdGxo524MaZSqjDlXtg4G341FtoNLhfeua+Q37+7kKSEGF6/foDNY14PhVRBN8Ycxrz/wqK3nHnM+19VLlRU6uWW9xaxr6CUibcOsblZ6ikr6MbUBelznIWcu18IZz5SLqSqPPr5KhZn5PDy1SfQs1WjQxzEhLuQOilqjKlE7g5nrHnT9nDZqxBR/mv7wfwMxi/YzK1DO3Nh31YuJWlCgbXQjQll3lL4dBQU7g24ChRg1da9/HXyak7rmsQ953Z3KUkTKqygGxPKZvwdNn3vnAQtMw0uQF5RKWM+WELTuCie/21/G9FirKAbE7J+ngE//BtOGBlwElRVefCzFaRn7eeD3w0m0U6CGqwP3ZjQtD8LPrsFkrrB8KcDwhMWZvL50q3cdXY3BneyRViMo1oFXUSGi8g6EUkVkQcqiTcWkckiskxEVonIjcFP1Zh6QhUm3Q4Fe+DXr0N0XLlwetZ+Hpu8ipM7JXLbsC4uJWlCUZUFXUQ8wMvA+UAv4CoR6VVht9uA1araDxgK/FNEooOcqzH1w6I3Yd2XcPZj0KpvuZDXp/zx42V4IoR/XtnP+s1NOdVpoQ8CUlU1TVWLgfHApRX2UaChOJNGJAB7gNKgZmpMfbAnDaY9CJ3PhJP+EBAeO+tnFqVn8/ilvWndpIELCZpQVp2C3gbYXGY70/9YWS8BPYGtwArgTlX1VTyQiIwWkYUisjCU52sxxhU+H3w+BiKi4NKXA8abr9q6l+e/Xc+FfVtxaf/WLiVpQll1Cnplv+kqTod4HrAUaA30B14SkYDL1VR1nKoOUNUBycnJR5iqMWFu/jhI/xGGPwmNyhfs4lIf93y8jKZx0fzjV71tBkVTqeoU9EygbZntFJyWeFk3AhPVkQpsBHoEJ0Vj6oGsn+Hbx6DredD/6oDw2Fk/s3Z7Lk9c1ocmcXZ6ylSuOgV9AdBVRDr6T3SOACZV2CcDOAtARFoA3QFb+seY6vD54PPbwBMNFz8PFVrfG3bk8uL0DVzSrzVn92rhTo6mTqjywiJVLRWRMcA0wAO8oaqrROQWf3ws8DjwloiswOmiuV9Vd9dg3saEj8VvQcZPTr95ha4Wr0+579PlJMRE8ujFFQeXGVNeta4UVdUpwJQKj40tc38rcG5wUzOmHsjdDt88Bh1Og/7XBITfnrOJJRk5/GdEf7sa1FTJrhQ1xk1TH4DSQrjo+YCulq05BTz39TqGdU/mkn42qsVUzQq6MW5Z/zWs+gxOvxeSAq/4fPyL1fhU+dulNqrFVI8VdGPcUJwPU+6BpO5wyl0B4RnrdvLVyu3cfmZX2jaLC3y+MZWw2RaNccMP/4acDLhhCkSWH4ZYWOLl0c9X0Tk5nt+d1smlBE1dZAXdmNq2Jw1+/A/0uRI6nBIQfmVGKhl78vngdycRHWk/ok312afFmNo29c/giYJz/hYQysjKZ+zsNC7t35ohnZNcSM7UZVbQjalN66bC+qkw9AFoFLj+59+/XE1khPCXC3q6kJyp66ygG1NbSgph6v3OidCTbgkI/7BhN1+v3sFtw7rQolGsCwmaus760I2pLfPGQvYmuO4zp8uljFKvj79OXkW7ZnGMOrWjO/mZOs9a6MbUhrxdMPs56Dbcmeu8gvfmprNhZx4PXtiT2CiPCwmacGAF3ZjaMOMfUFoA5/49IJSTX8y/v93AqV2SONcm3zLHwAq6MTVtxypY/DYMvBmSugaEX5yeSm5hCQ9d1NOuCDXHxAq6MTVJ1VlSLqYRnHF/QDgjK593ftrEFSe2pUfLgDVhjDkiVtCNqUmp30LaDKeYxzULCD89bS2RERH88dxuLiRnwo0VdGNqis8L3zwKTTs63S0VLErP5svl2xh9eicbpmiCwoYtGlNTln8MO1fBb94ImK9FVXlyyhqSG8Yw+nSbr8UEh7XQjakJJYXOyJZW/aHXZQHhb1bvYGF6Nnef3Y34GGtXmeCwT5IxNWH+ONi72VlWLqJ8u8nrU56dto5OyfFcOSDFpQRNOLIWujHBVpAN3/8TupwNnc4ICE9cnMmGnXn86dzuRHrsK2iCxz5NxgTbjy9A4V44+7GAUGGJl39/s55+KY0Z3rtl7edmwpoVdGOCKXeHM2dLn99Ayz4B4ffmprN1byH3D+9hFxGZoLOCbkwwff9PKC2CoX8OCOUWlvDyjFRO65rEkC4217kJPivoxgRLTgYsfANOuA4SOweE3/hhE9n5JfzpvO4uJGfqAyvoxgTLzKdBIuD0+wJCOfnFvPZ9Gucd14K+KU1qPzdTL1hBNyYYdq2HZR/AoN9B4zYB4XGz08grLuXuc+wSf1NzrKAbEwyznoLIBnDq3QGh3XlFvPnjJi7u29om4DI1ygq6Mcdqx2pYORFO+j3EB57sfHXmzxSVernr7MCpc40JJivoxhyrWU9BdAIMuT0gtH1vIe/OTefXJ6TQKTnBheRMfWIF3ZhjsX0FrP4cTr610ulxX52Zis+n3HGWtc5NzbOCbsyxmPkUxDSGwbcGhLbtLeDD+Zu5YkAKbZvFuZCcqW+qVdBFZLiIrBORVBF54BD7DBWRpSKySkRmBTdNY0LQ1iWw9gsYMgYaNAkIvzLjZ3yq3Dq0S+3nZuqlKmdbFBEP8DJwDpAJLBCRSaq6usw+TYBXgOGqmiEizWsoX2NCx8ynIbYJnHRLQGhrTgEfLdjMFQPaWuvc1JrqtNAHAamqmqaqxcB44NIK+1wNTFTVDABV3RncNI0JMVuXwvqv4OQxEBs4FPGVmakoym3DAq8YNaamVKegtwE2l9nO9D9WVjegqYjMFJFFIjKysgOJyGgRWSgiC3ft2nV0GRsTCmY9A7GN4aTRAaGyrfOUptY6N7WnOgW9sinhtMJ2JHAicCFwHvCwiARcEqeq41R1gKoOSE5OPuJkjQkJ25bBui9h8G1OUa/g1Zk/A3DbMOs7N7WrOisWZQJty2ynAFsr2We3qu4H9ovIbKAfsD4oWRoTSg62zn8fENq+t5CPFmzmNye2pU2TBi4kZ+qz6rTQFwBdRaSjiEQDI4BJFfb5HDhNRCJFJA44CVgT3FSNCQHbVzgjWwbfWunIlrGzDoxssb5zU/uqbKGraqmIjAGmAR7gDVVdJSK3+ONjVXWNiEwFlgM+4DVVXVmTiRvjilnPQEyjSke27NxXyIfzM7j8hDY2ssW4olqLRKvqFGBKhcfGVth+Fng2eKkZE2J2roE1k+D0P1XaOh83O41Sn1rfuXGNXSlqTHXNfs6Zs6WSq0J35xXx3rx0Lu3fmvaJ8S4kZ4wVdGOqZ/cGWDURBo6qdM6W177fSFGpz1rnxlVW0I2pju//CZ4YODlwRsXs/cW8+9MmLurbms42o6JxkRV0Y6qyZyMs/xgG3AQJgddPvDlnE/uLvYyx1rlxmRV0Y6ryw78hIrLS+c73FZbw5o8bGX5cS7q3bOhCcsb8wgq6MYezNxOWfgAnXAeNWgWE35mzidzCUsacaa1z4z4r6MYczo8vAAqn3BkQ2l9Uyus/bOTMHs3p3SZwCgBjapsVdGMOJXcHLH4b+o2AJu0Cwh/MyyA7v8RGtpiQYQXdmEP56SXwFsOpfwwIFZZ4Gfd9Gqd0SeTE9k1dSM6YQFbQjalM/h5Y8Dr0/jUkBs7L8vHCzezKLWLMMFsr1IQOK+jGVGbuq1CyH067JyBUXOrjv7PSOLF9UwZ3CrzIyBi3WEE3pqLCvTDvv9DjImjeMyD8f0u2sCWngDFndkGksuUCjHGHFXRjKpr/Pyja60zCVUGp18crM1Pp3aYRQ7vZIi0mtFhBN6as4v0w9xXocg607h8Q/nLFNjZl5TNmmLXOTeixgm5MWYvegvysSlvnPp/y8oxUujZP4NxeLWs/N2OqYAXdmANKCmHOi9DhNGh3UkD4mzU7WL8jj9uGdSEiwlrnJvRYQTfmgKXvQ+42OP3egJCq8tL0VNonxnFR38ApAIwJBVbQjQHwlsCPz0ObAdDxjIDwrPW7WLFlL384ozORHvvamNBkn0xjAFZMgJwMp++8wsnOA63zVo1jufyEFJcSNKZqVtCN8XmdBSxa9IFu5wWE523cw8L0bG45ozPRkfaVMaHLPp3GrP4/yEp1+s4rGYr40vRUkhJi+O3AtrWfmzFHwAq6qd98Pmfx56Tu0POSgPDijGx+SN3N6NM7EhvlcSFBY6rPCrqp39Z/BTtXO3O2RAR+HV6ankqTuCiuOam9C8kZc2SsoJv6SxVmPwtNOzizKlawcstepq/dyahTOhIfE1n7+RlzhKygm/or9VvYusSZ79wTWLBfnpFKw9hIrj+lQ+3nZsxRsIJu6idVmPUMNG4L/a4KCK/fkctXK7dzw5AONIqNciFBY46cFXRTP22cBZnznbVCI6MDwi/PSCUu2sONp3R0ITljjo4VdFM/zXoWGraC468LCP28K4/Jy7Zy3cntaRYfWOyNCVVW0E39kz4H0n9wWudRsQHhl2ekEh0Zwe9O6+RCcsYcPSvopv6Z9QzEJ8MJ1weENu3ez+dLt3LtSe1JSohxITljjp4VdFO/bJ4PaTNgyB0QHRcQfmVmKpERwujTrXVu6p5qFXQRGS4i60QkVUQeOMx+A0XEKyK/CV6KxgTRzKcgLgkGjgoIbd6Tz8TFW7hqUDuaNwrsijEm1FVZ0EXEA7wMnA/0Aq4SkV6H2O9pYFqwkzQmKDYvgJ+/gyG3Q3R8QPiVmalEiHDLGZ1dSM6YY1edFvogIFVV01S1GBgPXFrJfrcDnwI7g5ifMcEz6ymIS4SBNweENu/JZ8LCTH47sC0tG1vr3NRN1SnobYDNZbYz/Y8dJCJtgMuAsYc7kIiMFpGFIrJw165dR5qrMUcvc6FzZejJYyAmISD88gyndX7rMGudm7qrOgW9ssUTtcL288D9quo93IFUdZyqDlDVAcnJydVM0ZggmPkUNGgKg34XENq8J59PFmVy1aC2tGrcwIXkjAmO6sw4lAmUnQg6BdhaYZ8BwHhx5pJOAi4QkVJV/b9gJGnMMdm8AFK/gbMegZiGAeGXpqcSESH8YWgXF5IzJniqU9AXAF1FpCOwBRgBXF12B1U9eH20iLwFfGHF3ISMmU84feeDRgeEMrLy+XRxJtcObm9956bOq7LLRVVLgTE4o1fWAB+r6ioRuUVEbqnpBI05Juk/wc/TnatCK2udz9jgb51b37mp+6o1ybOqTgGmVHis0hOgqnrDsadlTJDMfALim8PAwL7zjbv38+niLYw8uT0tbNy5CQN2pagJXxu/h42z4dS7K70q9D/frifaE2GtcxM2rKCb8KQKM56AhJYw4MaA8PoduXy+bCvXD+lA84bWOjfhwQq6CU8/T4eMOc5aoVGBQxGf/3Y98dGR/N7mbDFhxAq6CT+q8N3foHE7ODFwRsWVW/YyZcV2Rp3akaY237kJI1bQTfhZMxm2LYWhD0Bk4BS4//pmPY0bRDHqNFuNyIQXK+gmvPi8MP3vkNQd+o0ICC/YtIfpa3cy+vROtlaoCTvVGrZoTJ2x/CPYvQ6ufAciPOVCqsrTX62lecMYbrK1Qk0Ysha6CR+lRTDzSWjVH3peEhCevnYnC9OzueOsrjSI9gQ+35g6zlroJnwsfANyMuCif4OUn1PO61OembqODolx/HZg20McwJi6zVroJjwU7nXWCu14BnQ+KyA8adkW1u3I5Z5zuxPlsY+9CU/2yTbh4cf/QMEeOOevAa3zolIv//x6Pce1bsSFfVq5lKAxNc8Kuqn79m2Fn16B3r+B1scHhN+Zk05mdgEPnN+DiIjKpvc3JjxYQTd138wnwVcKZz0cEMrJL+bF6Rs4o1syp3W1RVVMeLOCbuq2nWtgyXvOOqFNOwSEX5yeSl5RKX+5oGft52ZMLbOCbuq2aQ9CdEM4/U8BofSs/bzz0yauOLEt3VsGzoVuTLixgm7qrg3fws/fwRn3QXxiQPiZaeuIjIjgj+d2cyE5Y2qfFXRTN3lL4esHoVmnSpeWW7BpD18u38bo0zvZ4hWm3rALi0zdtOhN2LUWfvs+RJafMdHrUx6btIpWjWP5/Rk2Pa6pP6yFbuqeghxnZEuH06DHhQHhTxZtZtXWfTxwfg/ioq3NYuoPK+im7pn5JBRkw3lPBFxEtK+whGenrWNA+6Zc0q+1Swka4w4r6KZu2b4S5o+DATdBq74B4Re/20DW/mIevfg4ROwiIlO/WEE3dYcqTPkTxDaBYQ8GhFN35vLWnE1ccWIKfVIa135+xrjMOhhN3bHiE2ed0Iv/A3HNyoVUlYf/bxUNojzcN7yHSwka4y5roZu6oSgXvn7Imavl+OsCwp8v3cpPaVncN7wHSQmBy84ZUx9YC93UDdP/AXk7YMT7ASsR7S0o4e9frqFf2yZcNaidSwka4z4r6Cb0bVkM8/8LA0dByoCA8L++Xsee/UW8ecNAPDaboqnHrMvFhDZvKUy+A+Kbw1mPBISXbc7h3bnpXDe4vZ0INfWetdBNaJv3KmxfAVe8DbHlC3ZxqY/7P11O84ax3HNed5cSNCZ0WEE3oSsnA2Y8Ad2GQ69LA8JjZ/3M2u25vDZyAI1io1xI0JjQYl0uJjSpwqTbQSLggucCrgjdsCOXl6ancnG/1pzdq4VLSRoTWqpV0EVkuIisE5FUEXmgkvg1IrLcf5sjIv2Cn6qpVxa9CWkz4dzHoUnbciGvT7n/0+XEx3h49OJe7uRnTAiqsqCLiAd4GTgf6AVcJSIVv0UbgTNUtS/wODAu2ImaeiQ7Hb5+GDoNhRNvDAi//kMaizNyeOTiXjbm3JgyqtNCHwSkqmqaqhYD44FyHZqqOkdVs/2bc4GU4KZp6g2fDyaNAQQueTGgq2Xd9lyem7ae845rwa/6t3EnR2NCVHUKehtgc5ntTP9jhzIK+KqygIiMFpGFIrJw165d1c/S1B8LX4eNs/1dLeUvEiou9XHXR0tp1CCSJy7rY5NvGVNBdQp6Zd8arXRHkWE4Bf3+yuKqOk5VB6jqgORkW4HdVLBjtXN5f5dz4MQbAsL/+W49a7bt48nL+5JoXS3GBKjOsMVMoOxZqRRga8WdRKQv8BpwvqpmBSc9U2+UFMKnoyCmIfzq1YCulgWb9vDqzJ+5ckAK59ioFmMqVZ0W+gKgq4h0FJFoYAQwqewOItIOmAhcp6rrg5+mCXvfPgo7VzvFPKH8r7fs/cXc8eES2jaL4+GLbFSLMYdSZQtdVUtFZAwwDfAAb6jqKhG5xR8fCzwCJAKv+Ps1S1U1cNINYyqzfhrMGwsn/QG6nlMupKrcO2EZWXnFTLx1CA3tAiJjDqlaV4qq6hRgSoXHxpa5fzNwc3BTM/VCdjp89nto0QfOfiwg/PoPG/lu7U4eu7gXvdvYXC3GHI5dKWrcU1IIH490hipe+TZExZYLL8nI5umpazm3VwuuH9LBnRyNqUNsLhfjnqkPwLalMOIDSOxcLrQzt5Bb3ltEi0axPPubfjZE0ZhqsIJu3LHkfefy/lPugh4XlgsVl/q49b3F7C0oYeIfTqFxnPWbG1MdVtBN7cuYC1/cBR1PhzMfDgj/dfIqFqZn8+JVx9OrdaPaz8+YOsr60E3tyk6H8ddA47bOHOee8m2K9+am8/68DH5/Ricu7tfapSSNqZusoJvaU7gPPhwBvhK4+iOIa1YuPH3tDh75fCXDuidz33k9XErSmLrLulxM7Sgthgk3wK51cO2nkNS1XHhF5l7GfLCEXq0b8dLVJ9jaoMYcBSvopub5fPD5rfDzd84Mip2HlQtnZudz09sLaBoXzRvXDyQ+xj6WxhwN++aYmqUK0/4MKybAWY/CCSPLhXfuK+Ta1+ZRWOLl/ZtPonmj2EMcyBhTFetDNzVr9rPOZf2Db4NT7y4X2rO/mGtfn8fO3CLeunEQ3Vo0dClJY8KDFXRTc2Y9CzP+Af2ugnP/Xm4Gxb0FJYx8Yx7pWfm8dv0ATmzf1MVEjQkP1uViasasZ5xi3ncEXPoyRPzSdsjeX8wNb85n3fZcxo0cwJDOSS4makz4sIJugksVZj4Fs55yWuaXvgwRnoPhnbmFXPfafDZm7efVa05kWPfmLiZrTHixgm6Cx+eFKffCwjeg/zXOiJYyxXxLTgHX/G8uO3OLePOGgZzSxVrmxgSTFXQTHAdWHFr7hTM/y9mPleszX7llLze9tYCCEi/vjjrJ+syNqQFW0M2xy9sJH10Hm+fC8Kdg8B/Khb9etZ07xy+lWXw07446ie4tbTSLMTXBCro5NluXOHOz5O+B37wJvS8/GFJV/vd9Gk9+tZa+KU3438gTad7QxpkbU1OsoJujt+wjmHwHxCfDqGnQqt/B0N6CEu77ZBnTVu3gwj6teO6KfjSI9hzmYMaYY2UF3Ry5olz46n5Y+j60P9VZbSj+lxOcK7fs5bYPFrMlu4CHLuzJqFM72gIVxtQCK+jmyGxZBJ/eDNmb4Iz74fT7Dk6BW+r18d/ZaTz/7XoS42MYP3owAzo0O/zxjDFBYwXdVE9JAcx6Gn58ARq2ghu+hPZDDobTduVxz4RlLMnI4cI+rXj8V71pFh/tYsLG1D9W0E3VNs6GyXfCnjQ4/lrnMv4GzrDDgmIvr85MZeysNOJiPLx41fG2MIUxLrGCbg5tTxp88yismQRNO8LISdDpDMAZwfL16h08/sVqMrML+FX/1vzlgp42W6IxLrKCbgLl7YIf/g3zx4EnGoY9CENuh6gGAMxNy+KZqWtZnJFDtxYJjB89mMGdEl1O2hhjBd38IncHzHnBuXS/pMDpXjnzIWjYElVl7s9ZvDIzle837KZlo1ieuKwPVwxIIcpjk3YaEwqsoBvYvgLm/ddZhMJbDH2uhNPugeRulHh9TFu+lf/NTmNZ5l6SEqL5ywU9GHlyB2KjbFy5MaHECnp9VZTnzLuy+B1I/xEiG0C/ETDkDkjszOY9+Xw4dS0fL8xkd14RHZPi+cdlvfn1CSlWyI0JUVbQ65OSQtg4C1Z9BqsnQcl+aNoBznkcjr+Wnd44pizfxuTlc1iUnk2EwJk9WnDVoLYM7d7cFm42JsRZQQ93+7ZB2gzY8DVs+AaK8yCmMfT5Db6+I1gd2Yvp63Yx/c01LMvMQRV6tGzIn87rzuUntKFV4wZu/w2MMdVkBT2cqEJOBmye78x8uOkH2LXWicUn4z3ucjYln8n3pb2YsymX+e/sISf/R0Sgb0oT7jqrGxf2bUmX5jYbojF1kRX0uqo4H7I2wK71sGMlbF8O25ZD/m4AfFHx7Es6gQ1dh/MTfZm+J5lV8/Mo8SqQSrtmcZzbqwWDOyVyerdkkhJi3P37GGOOWbUKuogMB/4DeIDXVPWpCnHxxy8A8oEbVHVxkHOtP1ShINuZZzx3G+zbAvu2Qk46ZKej2Rth7xYEBcArkexq0Im0yIEsi+/Ad3kdWZzbGl+uM5ywSVwUPVtGM+rUTvRLaUz/dk2sK8WYMFRlQRcRD/AycA6QCSwQkUmqurrMbucDXf23k4BX/X+GL58PfKXgKwFviXPfWwzeYrS0GF9pEaVFBfhKCvAW5eMtzsdXlI+vKA9fUR5alIcW7oOifUjhXjxFOXiKcoguziGmeA8e9Qa85G5pSoY2Z5O3A5t8g0nV1qRqGzZpSyiJoU2TBqQ0a8Bx3eK5MCmeTskJ9GjZkOYNY2y2Q2Pqgeq00AcBqaqaBiAi44FLgbIF/VLgHVVVYK6INBGRVqq6LdgJL5/5CY1nP3pw+0ArFUC0zP2D9/TgPlJmu/x9Jx6BD1EQfAjqbKN4ymx78OHBS0SZ161IcH7KVDW4L1cbkEsDcjWOHBLI0cZkaxuyaMQempAX1Yz86GTyG7TAG9+C+PgEEuOjSYyPoXlCNH0axdKiUQwtG8WSlBBDhI1CMaZeq05BbwNsLrOdSWDru7J92gDlCrqIjAZGA7Rr1+5IcwUgOr4Ju+O6lHtMy5TvsutYli3vv+wjIGW2D94XVJyyfWB/Fc8vj4kHn0SgEoFGRPpjHlQinW1PFL6IKDQiCiKi0MhY1BMDnmg0Kg6JauDcYhtCdAKe2Hhio6OIiYwgNspD02gPraM8xEV7iI+JJCYywlrVxpgjUp2CXllVqdg8rc4+qOo4YBzAgAEDDt3EPYweA8+GgWcfzVONMSasVWcSjkygbZntFGDrUexjjDGmBlWnoC8AuopIRxGJBkYAkyrsMwkYKY7BwN6a6D83xhhzaFV2uahqqYiMAabhnOd7Q1VXicgt/vhYYArOkMVUnGGLN9ZcysYYYypTrXHoqjoFp2iXfWxsmfsK3Bbc1IwxxhwJm8jaGGPChBV0Y4wJE1bQjTEmTFhBN8aYMCGqR3V9z7G/sMguIP0on54E7A5iOsESqnlB6OZmeR0Zy+vIhGNe7VU1ubKAawX9WIjIQlUd4HYeFYVqXhC6uVleR8byOjL1LS/rcjHGmDBhBd0YY8JEXS3o49xO4BBCNS8I3dwsryNjeR2ZepVXnexDN8YYE6iuttCNMcZUYAXdGGPCRJ0v6CJyr4ioiCS5nQuAiDwuIstFZKmIfC0ird3OCUBEnhWRtf7cPhORJm7nBCAiV4jIKhHxiYjrw8tEZLiIrBORVBF5wO18DhCRN0Rkp4isdDuXA0SkrYjMEJE1/v+Hd7qdE4CIxIrIfBFZ5s/rr27nVJaIeERkiYh8Eexj1+mCLiJtcRavznA7lzKeVdW+qtof+AJ4xOV8DvgG6K2qfYH1wJ9dzueAlcDlwGy3EymzIPr5QC/gKhHp5W5WB70FDHc7iQpKgXtUtScwGLgtRN6vIuBMVe0H9AeG+9dpCBV3Amtq4sB1uqAD/wbuo5Ll7tyiqvvKbMYTIrmp6teqWurfnIuzqpTrVHWNqq5zOw+/gwuiq2oxcGBBdNep6mxgj9t5lKWq21R1sf9+Lk6RauNuVs503qqa59+M8t9C4nsoIinAhcBrNXH8OlvQReQSYIuqLnM7l4pE5B8ishm4htBpoZd1E/CV20mEoEMtdm6qICIdgOOBeS6nAhzs1lgK7AS+UdWQyAt4HqcR6quJg1drgQu3iMi3QMtKQg8CfwHOrd2MHIfLS1U/V9UHgQdF5M/AGODRUMjLv8+DOD+V36+NnKqbV4io1mLnpjwRSQA+Be6q8AvVNarqBfr7zxV9JiK9VdXV8w8ichGwU1UXicjQmniNkC7oqnp2ZY+LSB+gI7BMRMDpPlgsIoNUdbtbeVXiA+BLaqmgV5WXiFwPXAScpbV4AcIRvF9us8XOj5CIROEU8/dVdaLb+VSkqjkiMhPn/IPbJ5RPAS4RkQuAWKCRiLynqtcG6wXqZJeLqq5Q1eaq2kFVO+B8EU+ojWJeFRHpWmbzEmCtW7mUJSLDgfuBS1Q13+18QlR1FkQ3fuK0pl4H1qjqv9zO5wARST4wiktEGgBnEwLfQ1X9s6qm+GvWCGB6MIs51NGCHuKeEpGVIrIcp0soJIZyAS8BDYFv/EMqx1b1hNogIpeJSCZwMvCliExzKxf/SeMDC6KvAT5W1VVu5VOWiHwI/AR0F5FMERnldk44Lc7rgDP9n6ml/tan21oBM/zfwQU4fehBHyIYiuzSf2OMCRPWQjfGmDBhBd0YY8KEFXRjjAkTVtCNMSZMWEE3xpgwYQXdGGPChBV0Y4wJE/8Pp3DRw+UPwvsAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "plt.plot(grid, X.cdf(grid), 'C0')\n", + "plt.plot(grid, Y.cdf(grid), 'C1')\n", + "plt.title('Distribution CDFs')\n", + "plt.legend(['x', 'y'])\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "c44e1bfe-8ebe-41d1-b96d-90f970cb8311", + "metadata": {}, + "source": [ + "Therefore, to test the alternative that $X$ is stochastically greater than $Y$, we pass `alternative='less'` into `ks_2samp`." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "97815b21-eb3e-401e-9cff-1da838141a70", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "KstestResult(statistic=0.12484394506866417, pvalue=0.00022195733373729093)\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "C:\\Users\\matth\\AppData\\Local\\Temp\\ipykernel_35644\\3756467023.py:1: RuntimeWarning: ks_2samp: Exact calculation unsuccessful. Switching to method=asymp.\n", + " res1 = stats.ks_2samp(x, y, alternative='less', method='exact')\n" + ] + } + ], + "source": [ + "res1 = stats.ks_2samp(x, y, alternative='less', method='exact')\n", + "print(res1)" + ] + }, + { + "cell_type": "markdown", + "id": "33065824-d0c3-4ad6-9c08-296a160636fd", + "metadata": {}, + "source": [ + "The $p$-value is tiny, confirming what we already know: the data are inconsistent with the null hypothesis, and we have evidence to reject it in favor of the alternative.\n", + "\n", + "The warning states that `ks_2samp` was unable to compute an exact $p$-value, and an asymptotic $p$-value is being returned instead. To determine whether the asymptotic $p$-value is accurate for these sample sizes, we can perform a permutation test." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "649978b2-02ff-42a6-90d1-e04b3051c31c", + "metadata": {}, + "outputs": [], + "source": [ + "def statistic(x, y):\n", + " return stats.ks_2samp(x, y, alternative='less').statistic\n", + "\n", + "# This would be extremely slow!\n", + "# res2 = stats.permutation_test((x, y), statistic, alternative='greater')" + ] + }, + { + "cell_type": "markdown", + "id": "69c52b5c-f7aa-4eaf-b2ad-9e53723aa9c7", + "metadata": {}, + "source": [ + "The calculation above would be extremely slow to run. Unfortunately, `ks_2samp` does not accept an `axis` argument, so we can't speed it up using vectorization without truly implementing the statistic ourselves. However, lack of vectorization is not be the bottleneck here. Note that the call to `ks_2samp` is quite slow with the default parameters *even for 1D inputs*." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "4825ddb9-5bf9-445f-bb82-031d3b330fc8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "374 ms ± 2.57 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" + ] + } + ], + "source": [ + "# No need for the warning; we know the exact calculation is unsuccessful\n", + "import warnings\n", + "warnings.filterwarnings('ignore')\n", + "%timeit stats.ks_2samp(x, y, alternative='less', method='exact')" + ] + }, + { + "cell_type": "markdown", + "id": "f728c880-bbde-4003-87a6-c4090925088d", + "metadata": {}, + "source": [ + "By default, `permutation_test` needs to call `ks_2samp` 9999 times, which would take about an hour. We can speed this up dramatically by noting `permutation_test` only uses `ks_2samp` to compute the test statistic, so the `pvalue` attribute of the `ks_2samp` result object is not used at all. We can use `ks_2samp` to compute essentially the same value of the test statistic, but much faster, by specifying `method='asymp'`." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "abfe9b45-4e7b-4284-bd1b-2620eced89ff", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "167 µs ± 947 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n" + ] + } + ], + "source": [ + "# method='asymp' and method='exact' result in the same statistic value\n", + "res1 = stats.ks_2samp(x, y, alternative='less', method='asymp')\n", + "res2 = stats.ks_2samp(x, y, alternative='less', method='exact')\n", + "np.testing.assert_allclose(res1.statistic, res2.statistic, atol=1e-15)\n", + "\n", + "# but method='asymp' is much faster\n", + "%timeit stats.ks_2samp(x, y, alternative='less', method='asymp')" + ] + }, + { + "cell_type": "markdown", + "id": "bafdc39b-6132-4121-b7b8-6853603b129a", + "metadata": {}, + "source": [ + "Now we can run a randomized `permutation_test` in reasonable time. " + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "c0c64b0b-31c5-4586-b696-36f03642040b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.0002219573337372917 0.9999\n" + ] + } + ], + "source": [ + "def statistic(x, y):\n", + " return stats.ks_2samp(x, y, alternative='less', method='asymp').statistic\n", + "\n", + "res3 = stats.permutation_test((x, y), statistic, alternative='less')\n", + "print(res1.pvalue, res3.pvalue)" + ] + }, + { + "cell_type": "markdown", + "id": "c9cb3241-fea3-4b3d-90af-a60a7852ffbd", + "metadata": {}, + "source": [ + "This was much faster, but something is still wrong. Either the approximate $p$-value is wildly inaccurate, or we have set up our test incorrectly. The latter turns out to be the case: the value of `alternative` passed into `ks_2samp` changes *the definition of the test statistic*, but a *greater* p-value is always considered more extreme. Therefore, even if we wish to perform a test equivalent to `ks_2samp` with `alternative='less'`, we actually need to pass `alternative='greater'` into `permutation_test`!" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "3fc268c6-db67-4bee-9c3e-b50ea2896e98", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.0002219573337372917 0.0005\n" + ] + } + ], + "source": [ + "# greater values of the statistic returned by `ks_2samp` are more extreme\n", + "res4= stats.permutation_test((x, y), statistic, alternative='greater')\n", + "print(res1.pvalue, res4.pvalue)" + ] + }, + { + "cell_type": "markdown", + "id": "9b06fd03-ca61-498c-9542-5dabc43bcddd", + "metadata": {}, + "source": [ + "At last, `permutation_test` is invoked correctly. Indeed, the asymptotic $p$-value produced by `ks_2samp` appears to be reliable for these sample sizes." + ] + }, + { + "cell_type": "markdown", + "id": "a8013eef-9aba-475d-b83a-ffbbbe311e8d", + "metadata": {}, + "source": [ + "### Other Tests\n", + "As we can see, `permutation_test` with `permutation_type='independent'` is a versatile tool for comparing independent samples. Provided only data and a statistic, it can produce the null distribution and replicate the $p$-value of many such tests in SciPy, and it may be more accurate than these existing implementations, especially for small samples and when there are ties:\n", + "\n", + "- [`ttest_ind`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html)\n", + "- [`cramervonmises_2samp`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.cramervonmises_2samp.html)\n", + "- [`ks_2samp`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html)\n", + "- [`epps_singleton_2samp`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.epps_singleton_2samp.html)\n", + "- [`mannwhitneyu`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html)\n", + "- [`kruskal`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kruskal.html)\n", + "- [`friedmanchisquare`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.friedmanchisquare.html)\n", + "- [`brunnermunzel`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.brunnermunzel.html)\n", + "- [`ansari`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ansari.html)\n", + "- [`bartlett`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bartlett.html)\n", + "- [`levene`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.levene.html)\n", + "- [`anderson_ksamp`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.anderson_ksamp.html)\n", + "- [`fligner`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.fligner.html)\n", + "- [`median_test`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.median_test.html)\n", + "- [`mood`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mood.html)\n", + "\n", + "In addition, `permutation_test` with `permutation_type='independent'` can be used to perform tests not yet implemented in SciPy.\n", + "\n", + "However, there are other types of permutation tests that do not assume that the samples are entirely independent. We continue the study of `permutation_test` with [Paired-Sample Tests](https://nbviewer.org/github/mdhaber/scipy/blob/resampling_tutorial/doc/source/tutorial/stats/notebooks/resampling_tutorial_2b.ipynb)." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.5" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_2b.ipynb b/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_2b.ipynb new file mode 100644 index 0000000..9cd2c07 --- /dev/null +++ b/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_2b.ipynb @@ -0,0 +1,544 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0da5ab55-df7a-48d9-a9e8-a84473eb2a72", + "metadata": { + "tags": [] + }, + "source": [ + "### Paired-Sample Tests\n", + "\n", + "In [The Design of Experiments](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2458144/https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2458144/), Fisher describes an experiment conducted by Charles Darwin to measure the effect of cross-fertilization on plant growth. In Darwin's experiment, pairs of plants were grown from the same batch of seed in the same pot under the same conditions, except that one was self-fertilized and the other was cross-fertilized. \n", + "\n", + "> The evident object of these precautions is to increase the sensitiveness of the experiment, by making such differences in growth rate as were to be observed as little as possible dependent from environmental circumstancces, and as much as possible, therefore, from intrinsic differences due to their mode of origin.\n", + "\n", + "The `x` and `y` arrays below record the height of the cross-fertilized and self-fertilized plants, respectively." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "77e2b91f-6d6c-43c0-b6f1-0be309885c9d", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "x = np.array([23.5, 12, 21, 22, 19.125, 21.5, 22.125, 20.375, 18.25, 21.625, 23.25, 21, 22.125, 23, 12]) # (in) cross-fertilized\n", + "y = np.array([17.375, 20.375, 20, 20, 18.375, 18.625, 18.625, 15.25, 16.5, 18, 16.25, 18, 12.75, 15.5, 18]) # (in) self-fertilized\n", + "assert len(x) == len(y) # elements at corresponding positions form a pair" + ] + }, + { + "cell_type": "markdown", + "id": "24b37c06-fb34-493e-ab4b-c49a14185c69", + "metadata": {}, + "source": [ + "The null hypothesis was that the method of fertilization would have no effect, and Darwin's alternative hypothesis was that cross-fertilized plants would be taller. Fisher recommends the t-test, which is implemented for paired (\"related\") samples by `stats.ttest_rel`." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "005bb7e7-8751-4298-a13e-c796e149df9e", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Ttest_relResult(statistic=2.1479874613311205, pvalue=0.024851472010900447)" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from scipy import stats\n", + "# alternative hypothesis 'greater': the mean of the distribution underlying \n", + "# `x` is greater than the mean of the distribution underlying `y` \n", + "res_t = stats.ttest_rel(x, y, alternative='greater')\n", + "res_t" + ] + }, + { + "cell_type": "markdown", + "id": "809a24f6-4345-47f5-b7b8-9c0b64b7384f", + "metadata": {}, + "source": [ + "However, the t-test is derived under the assumption that samples were drawn from a normal population. He reports that \n", + "\n", + "> There has, however, in recent years, been a tendency for theoretical statisticians, not closely in touch with the requirements of experimental data, to stress the element of normality, in the hypothesis tested, as though it were a serious limitation to the test applied.\n", + "\n", + "(The same is true today!) And so he proposes a different test that does not rely on the assumption of normality:\n", + "\n", + "> On the hypothesis that the two series of seeds are random sampled from identical populations, and that their sites have been assigned to members of each pair independently at random, the 15 differences [in height between pairs] would each have occured with equal frequency with a positive or with a negative sign... \n", + "Since *ex hypothesi* each of these $2^{15}$ combinations will occur by chance with equal frequency, a knowledge of how many of them are equal to or greater than the the value actually observed affords a direct arithmetical test of the significance of this value.\n", + "\n", + "He continues to perform the following test." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "5c328b74-1774-4008-a8fc-d922fa0d94b9", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Text(0, 0.5, 'Observed Frequency')" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjeklEQVR4nO3df5xddX3n8debIKCtbLSMNgZwAhvE4FLAEaiKP1pdE7BGfYCG2oJom8aCW9e2a1C3ujy6u6z4o1KRiBIBqwT8Uc0WLCKt0roNMEFEAkRDiBJIIZUVUFhi8L1/nO/ozc2dmTOTe2bunXk/H4/7mHvO98f93Dv33s8953vO98g2ERER3bDXdAcQEREzR5JKRER0TZJKRER0TZJKRER0TZJKRER0zd7THcB0OuCAAzw4ODjdYURE9JX169f/m+2BTmWzOqkMDg4yPDw83WFERPQVST8YrSy7vyIiomsaTSqSFkvaKGmTpJUdyiXp/FJ+q6RjyvqDJP2jpDskbZD0Jy1tni7pWknfL3+f1lJ2dulro6RXNfncIiJid40lFUlzgAuAJcAi4FRJi9qqLQEWltty4MKyfifwp7afCxwPnNnSdiVwne2FwHVlmVK+DDgCWAx8vMQQERFTpMktlWOBTbY3294BrAGWttVZClzmyjpgrqR5trfZvhnA9iPAHcD8ljaXlvuXAq9tWb/G9uO27wY2lRgiImKKNJlU5gP3tCxv5ZeJoXYdSYPA0cANZdUzbW8DKH+fMYHHQ9JyScOShrdv3z6R5xMREeNoMqmow7r22SvHrCPpV4EvAu+w/XAXHg/bF9kesj00MNDxiLiIiJikJpPKVuCgluUDgfvq1pH0JKqE8lnbX2qpc7+keaXOPOCBCTxeREQ0qMmkchOwUNICSftQDaKvbauzFjitHAV2PPCQ7W2SBFwM3GH7wx3anF7unw58pWX9Mkn7SlpANfh/Y/efVkREjKaxkx9t75R0FnANMAdYbXuDpBWlfBVwNXAi1aD6o8AZpfmLgN8HvivplrLu3bavBs4FrpT0VuCHwCmlvw2SrgRupzp67EzbTzT1/CIiYneazRfpGhoacs6oj14yuPIqtpx70m73I3qJpPW2hzqV5Yz6iIjomiSViGkwuPKqrtaL6BVJKhER0TVJKhER0TVJKhER0TVJKhFTqM4YyeDKq2rXi+g1SSoRU6BTophMUujUR5JL9JIklYiI6JoklYgeMNbWRrZEop/M6mvUR0y3bp6vkjPwoxdkSyViio2WICayRZKtl+hVSSoREdE1SSoRDZmOrYlswcR0S1KJmAGSTKJXJKlEdNFUfLkngUQvS1KJiIiuySHFEQ3KVkXMNo1uqUhaLGmjpE2SVnYol6TzS/mtko5pKVst6QFJt7W1uULSLeW2ZeRyw5IGJT3WUraqyecW0aqXk0cvxxYzT2NbKpLmABcArwS2AjdJWmv79pZqS4CF5XYccGH5C3AJ8DHgstZ+bb+x5TE+BDzUUnyX7aO6+kQiahr58p7uL/GRx8+JkDEdmtxSORbYZHuz7R3AGmBpW52lwGWurAPmSpoHYPt64MHROpck4A3A5Y1EHxERE9ZkUpkP3NOyvLWsm2id0ZwA3G/7+y3rFkj6tqRvSjqhUyNJyyUNSxrevn17zYeKiIg6mkwq6rDOk6gzmlPZdStlG3Cw7aOBdwKfk7T/bp3bF9kesj00MDBQ86Ei+s9074aL2anJpLIVOKhl+UDgvknU2Y2kvYHXA1eMrLP9uO0flfvrgbuAwyYVecQE9PKXd6+M88Ts0WRSuQlYKGmBpH2AZcDatjprgdPKUWDHAw/Z3laj71cAd9reOrJC0kA5OABJh1AN/m/uxhOJmCmSXKJpjR39ZXunpLOAa4A5wGrbGyStKOWrgKuBE4FNwKPAGSPtJV0OvAw4QNJW4H22Ly7Fy9h9gP4lwDmSdgJPACtsjzrQHxER3Se77hDGzDM0NOTh4eHpDiP6WD/+8s+hxrGnJK23PdSpLNO0RExSPyYU6N+4oz8kqURMQr6YIzpLUomYoJmQUGbCc4jelKQSERFdk6QSERFdk6QSUUN2F0XUk6QSMcslYUY3JalERETX5MqPETXNtF/0M+35RG/IlkrEOPLlG1FfkkpERHRNkkpERHRNkkpERHRNkkpERHRNkkpERHRNkkrEKGbTUV+57HB0S5JKRER0TZJKxBhm0y/32fRcozmNJhVJiyVtlLRJ0soO5ZJ0fim/VdIxLWWrJT0g6ba2Nu+XdK+kW8rtxJays0tfGyW9qsnnFhERu2ssqUiaA1wALAEWAadKWtRWbQmwsNyWAxe2lF0CLB6l+4/YPqrcri6PtwhYBhxR2n28xBAxYbP5V/vgyqtm9fOPPdPklsqxwCbbm23vANYAS9vqLAUuc2UdMFfSPADb1wMPTuDxlgJrbD9u+25gU4khIiKmyLhJRdLzJtn3fOCeluWtZd1E63RyVtldtlrS0ybSl6TlkoYlDW/fvr3GQ8Vskl/pEXumzpbKKkk3SvpjSXMn0Lc6rPMk6rS7EDgUOArYBnxoIn3Zvsj2kO2hgYGBcR4qZpMkk93lNYmJGjep2H4x8CbgIGBY0uckvbJG31tLmxEHAvdNok57PPfbfsL2z4FP8stdXBPuKyIiuqvWmIrt7wPvBd4FvBQ4X9Kdkl4/RrObgIWSFkjah2oQfW1bnbXAaeUosOOBh2xvGyuWkTGX4nXAyNFha4FlkvaVtIBq8P/GOs8vIr/IR5fXJiZi3It0SToSOAM4CbgW+B3bN0t6FvAvwJc6tbO9U9JZwDXAHGC17Q2SVpTyVcDVwIlUg+qPlscZedzLgZcBB0jaCrzP9sXAByQdRbVrawvwR6W/DZKuBG4HdgJn2n5iQq9GRETsEdljD2FIup5qN9MXbD/WVvb7tj/TYHyNGhoa8vDw8HSHET0gv8bHtuXck6Y7hOghktbbHupUVudywicCj4386pe0F7Cf7Uf7OaFERET31RlT+Trw5Jblp5R1ERERu6iTVPaz/ZORhXL/Kc2FFBER/apOUvlp25xczwceG6N+RMxAGXeKOuqMqbwD+LykkXM+5gFvbCyiiCmUL8p68jpFXXVOfrwJOBx4G/DHwHNtr286sIjoPUkuMZ46WyoALwAGS/2jJWH7ssaiioiIvlTn5MfPUM21dQswcjKhgSSViIjYRZ0tlSFgkcc7SzKiz2RXTkT31Tn66zbg15sOJCIi+l+dLZUDgNsl3Qg8PrLS9msaiyoiIvpSnaTy/qaDiIiImWHcpGL7m5KeDSy0/XVJT6GadTgiImIXdS4n/IfAF4BPlFXzgS83GFNE17UOymeAfs/lNYzR1BmoPxN4EfAw/OKCXc9oMqiIiOhPdZLK47Z3jCxI2pvxryMfERGzUJ2k8k1J7waeXK5N/3ngfzcbVkSzsvtm8vLaxVjqJJWVwHbgu1SX7r2a6nr145K0WNJGSZskrexQLknnl/Jb22ZDXi3pAUm3tbU5T9Kdpf7fSppb1g9KekzSLeW2qk6MMXsMrrwqX4hdlNcyOqkzoeTPbX/S9im2Ty73x939JWkOcAGwBFgEnCppUVu1JcDCclsOXNhSdgmwuEPX1wLPs30k8D3g7Jayu2wfVW4rxosxIvZMEku0qzP31910GEOxfcg4TY8FNtneXPpZAywFbm+psxS4rCSpdZLmSppne5vt6yUNdnjcr7UsrgNOHu85RETE1Kg799eI/YBTgKfXaDcfuKdleStwXI0684FtNfoHeAtwRcvyAknfpjpS7b22/6m9gaTlVFtFHHzwwTUfJiIi6qiz++tHLbd7bf8V8Fs1+lan7iZRp3Pn0nuAncBny6ptwMG2jwbeCXxO0v67dW5fZHvI9tDAwECdh4o+l100zctrHCPq7P46pmVxL6otl6fW6HsrcFDL8oHAfZOo0ymm04FXA789Mr5j+3HK3GS210u6CzgMGK4Ra0REdEGd3V8farm/E9gCvKFGu5uAhZIWAPcCy4DfbauzFjirjLccBzxke8xdX5IWA+8CXmr70Zb1A8CDtp+QdAjV4P/mGnHGDJZf0FNncOVVbDn3pOkOI6ZZnbm/Xj6Zjm3vlHQWcA3VXGGrbW+QtKKUr6I6PPlEYBPwKHDGSHtJlwMvAw6QtBV4n+2LgY8B+wLXSgJYV470eglwjqSdVBcTW2H7wcnEHhERk6Pxjg6W9M6xym1/uKsRTaGhoSEPD2fv2EyVrZTpka2VmU/SettDncrqHv31AqpdVQC/A1zPrkdtRURE1L5I1zG2HwGQ9H7g87b/oMnAIiKi/9SZpuVgYEfL8g5gsJFoIiKir9VJKp8BbpT0fknvA24ALms2rIjoVxnLmt3qHP313yV9FTihrDrD9rebDSsiIvpRnS0VgKcAD9v+KLC1nHsS0bPya3l65fWfvepcTvh9VCcbjswG/CTgb5oMKiIi+lOdLZXXAa8Bfgpg+z7qTdMSERGzTJ2ksqPMr2UASb/SbEgREdGv6iSVKyV9Apgr6Q+BrwOfbDasiIjoR2Me/aVqcq0rgMOprlHyHOAvbF87BbFFTEoGiSOmz5hJxbYlfdn286ku4xsRETGqOru/1kl6QeORROyhbKH0nvxPZp86c3+9HFghaQvVEWCi2og5ssnAIiYjX2K9I/+L2WnUpCLpYNs/BJZMYTwREdHHxtr99WUA2z8APmz7B623KYkuYhz5NRzRW8ZKKmq5f0jTgURERP8bK6l4lPsREREdjZVUfkPSw5IeAY4s9x+W9Iikh+t0LmmxpI2SNkla2aFcks4v5bdKOqalbLWkByTd1tbm6ZKulfT98vdpLWVnl742SnpVnRijf43s+hpceVV2g/Ww/J9ml1GTiu05tve3/VTbe5f7I8v7j9expDnABVQD/YuAUyUtaqu2BFhYbsuBC1vKLgEWd+h6JXCd7YXAdWWZ0vcy4IjS7uMlhoiImCJ1p76fjGOBTbY3294BrAGWttVZClzmyjqqqWDmAdi+HniwQ79LgUvL/UuB17asX2P7cdt3A5tKDBERMUWaTCrzgXtalreWdROt0+6ZtrcBlL/PmEhfkpZLGpY0vH379nGfRERE1NdkUlGHde0D/nXqdPPxsH2R7SHbQwMDA5N8qIiI6KTJpLIVOKhl+UDgvknUaXf/yC6y8veBPegrIqZABuhnj1GTyshRXqPdavR9E7BQ0gJJ+1ANoq9tq7MWOK0cBXY88NDIrq0xrAVOL/dPB77Ssn6ZpH3L5Y4XAjfWiDMiIrpk1GlabD8VQNI5wL8Cn6HaxfQmalz50fZOSWcB1wBzgNW2N0haUcpXAVcDJ1INqj8KnDHSXtLlwMuAAyRtBd5n+2LgXKprvLwV+CFwSulvg6QrgduBncCZtp+o/1JEv8iv3v428v/bcu5J0xxJNEHVRR3HqCDdYPu48db1o6GhIQ8PD093GDFBSSozQ5JK/5K03vZQp7I6YypPSHqTpDmS9pL0JiBbABGxR/LjYGaqk1R+F3gDcH+5nVLWRUy5fBFF9LZxr6diewu7n7QYERGxm3G3VCQdJum6kTm4JB0p6b3NhxYREf2mzu6vTwJnAz8DsH0r1eHBEVMqu75mnvxPZ546SeUpttvP99jZRDAREdHf6iSVf5N0KGXKE0knA+OdoBgREbPQuAP1wJnARcDhku4F7qY6ATIiImIXdZLKD2y/QtKvAHvZfqTpoCIioj/V2f11t6SLgOOBnzQcT8QuMpA7e+R/PTPUSSrPAb5OtRvsbkkfk/TiZsOKiIh+NG5Ssf2Y7Sttvx44Gtgf+GbjkUVERN+pdT0VSS+V9HHgZmA/qmlbIhqV3SER/afOGfV3A+8A/gl4nu032P5i04FFtEqCmbnyv51Zxjz6S9Ic4NO2z5mieCIioo+NuaVSLnL18imKJSIi+lyd81T+j6SPAVcAPx1ZafvmxqKKaJHdIzNf/sczR52B+hcCRwDnAB8qtw/W6VzSYkkbJW2StLJDuSSdX8pvlXTMeG0lXSHplnLbIumWsn5Q0mMtZavqxBi9J18wkfdA/6pzPZVJ7f4q4zEXAK8EtgI3SVpr+/aWakuAheV2HHAhcNxYbW2/seUxPgQ81NLfXbaPmky8ERGx5+oc/fVMSRdL+mpZXiTprTX6PhbYZHuz7R3AGna/2NdS4DJX1gFzJc2r01aSqA5tvrxGLBERMQXq7P66BLgGeFZZ/h7VIcbjmQ/c07K8tayrU6dO2xOA+21/v2XdAknflvRNSSd0CkrScknDkoa3b99e42nEdMjuj9kp//f+VyepHGD7SuDnALZ3Ak/UaKcO61yzTp22p7LrVso24GDbRwPvBD4naf/dOrEvsj1ke2hgYGDU4GN65EslRuS90J/qJJWfSvo1fnk9lePZdRxjNFuBg1qWDwTuq1lnzLaS9gZeT3VEGgC2H7f9o3J/PXAXcFiNOKNH5EskYNf3Qd4T/adOUnknsBY4VNK3gMuAt9dodxOwUNICSftQXYJ4bVudtcBp5Siw44GHbG+r0fYVwJ22t46skDRQBviRdAjV4P/mGnFGRESX1Dn662ZJL6WarVjARts/q9Fup6SzqMZj5gCrbW+QtKKUrwKuBk4ENgGPAmeM1bal+2XsPkD/EuAcSSO751bYfnC8OCMionvGTSqSTgH+viSE9wLHSPrLOic/2r6aKnG0rlvVct9UU+rXattS9uYO674IZE6yiIhpVGf313+1/Ui5hsqrgEupzieJ6JrsO4+YGeoklZEjvU4CLrT9FWCf5kKKiIh+VSep3CvpE1QnGl4tad+a7SIiYpapkxzeQDVgvtj2j4GnA3/eZFAREdGf6lxO+FFgC7BE0tuBeba/1nRgERGQ8bZ+U2fur7+gGpz/NeAA4NPlKLCIiCmRxNI/6lxP5VTgaNv/D0DSuVTXqv/LJgOLmW/ki2LLuSdNcyTRD/J+6Q91xlS2APu1LO9LNQVKRETELkbdUpH011TzfT0ObJB0bVl+JfDPUxNezAbZtRExc4y1+2u4/F0P/G3L+m80Fk1ERPS1UXd/2b7U9qVUMwGvp0oyV7Ssj5iQzD4b3ZD3UW8bNalI2lvSB6imob8U+BvgHkkfkPSkqQowIiL6x1gD9edRnei4wPbzy8WvDgXmAh+cgtgiIqLPjJVUXg38oe1HRlbYfhh4G9V09RGTkl0WETPXWEnFZWr69pVPsPulfSMiIsZMKrdLOq19paTfA+5sLqSIiOhXYx1SfCbwJUlvoTr6y8ALgCcDr5uC2GIGyq6v6Ia8j3rXqEnF9r3AcZJ+CziC6lLCX7V93VQFFxER/aXOLMX/YPuvbZ8/0YQiabGkjZI2SVrZoVySzi/lt0o6Zry2kt4v6V5Jt5TbiS1lZ5f6GyW9aiKxRkTEnqszoeSkSJoDXEA1rctW4CZJa23f3lJtCbCw3I6jukzxcTXafsT2Loc1S1oELKPaqnoW8HVJh5UDCyIiYgo0eQXHY4FNtjfb3gGsAZa21VkKXObKOmCupHk127ZbCqyx/bjtu4FNpZ+IiJgiTSaV+cA9Lctby7o6dcZre1bZXbZa0tMm8HhIWi5pWNLw9u3bJ/J8Yg9kYDWaMvLeynusNzSZVNRhXfv5LaPVGavthVRn9h8FbAM+NIHHw/ZFtodsDw0MDHRoEt2UD3rE7NLYmArVlsJBLcsHAvfVrLPPaG1t3z+yUtIngb+bwOPFFEpCiZh9mtxSuQlYKGmBpH2oBtHXttVZC5xWjgI7HnjI9rax2pYxlxGvA25r6WuZpH0lLaAa/L+xqScXERG7a2xLxfZOSWcB1wBzgNW2N0haUcpXAVdTzSO2CXgUOGOstqXrD0g6imrX1hbgj0qbDZKuBG4HdgJn5siv3pAtlmha3mO9Qx2m95o1hoaGPDw8PH7FmJDBlVex5dyT8kGPaTHy3su17Jsjab3toU5lTe7+ilksCSWmS9570ytJJSIiuiZJJSIiuiZJJSIiuiZJJboq+7OjVwyuvCrvx2mQpBJdkQ9w9Kq8L6dWkkpERHRNkkrssfwSjF6X9+jUSVKJiIiuSVKJiIiuSVKJiIiuSVKJCWs90iv7qqNf5D07NZJUIiKia5JUIiKia5JUYtKyGyH6Td6zzUtSiYiIrklSiYhZJ1sszWnscsIAkhYDH6W6JPCnbJ/bVq5SfiLV5YTfbPvmsdpKOg/4HWAHcBdwhu0fSxoE7gA2lu7X2V7R5PObTfIhjJmm9T2dq0R2T2NbKpLmABcAS4BFwKmSFrVVWwIsLLflwIU12l4LPM/2kcD3gLNb+rvL9lHlloQSEbXkR1P3NLn761hgk+3NtncAa4ClbXWWApe5sg6YK2neWG1tf832ztJ+HXBgg88hIiImoMmkMh+4p2V5a1lXp06dtgBvAb7asrxA0rclfVPSCZ2CkrRc0rCk4e3bt9d7JhERUUuTSUUd1rlmnXHbSnoPsBP4bFm1DTjY9tHAO4HPSdp/t07si2wP2R4aGBgY5ylErpMSs0Xe593RZFLZChzUsnwgcF/NOmO2lXQ68GrgTbYNYPtx2z8q99dTDeIf1pVnMotkKouY7fLe3zNNJpWbgIWSFkjaB1gGrG2rsxY4TZXjgYdsbxurbTkq7F3Aa2w/OtKRpIEywI+kQ6gG/zc3+PwiIqJNY4cU294p6SzgGqrDglfb3iBpRSlfBVxNdTjxJqpDis8Yq23p+mPAvsC11RHJvzh0+CXAOZJ2Ak8AK2w/2NTzmw3yiy1mm7zn91yj56nYvpoqcbSuW9Vy38CZdduW9f9+lPpfBL64J/HOdvlARcSeyhn1sZskl5jtMrY4eUkqs1yO7ororPVzkc9JfUkqERHRNUkqAWQzP2I0+WxMTJLKLJUPSkQ0IUklIqKmjK2ML0klIiK6Jklllmg/kiUiJi+fodElqcxQdZJIPhgRe679czTbP1dJKjNY3uwRzcpnbHcqk/zOSkNDQx4eHp7uMBqRN3fE9JnplyeWtN72UKeybKlERETXJKlERDRgtu4tSFKZQTIJXkRvmM2fxSSVPtP6Zu10ItZsfBNH9JOZnnCSVCIiomty9FcfHf01U3/ZRMxm/XikWI7+6lFj7bpq3bWVZBIxc431We/H3duNJhVJiyVtlLRJ0soO5ZJ0fim/VdIx47WV9HRJ10r6fvn7tJays0v9jZJe1eRz21N13xxJLBGzw0Q/6736ndBYUpE0B7gAWAIsAk6VtKit2hJgYbktBy6s0XYlcJ3thcB1ZZlSvgw4AlgMfLz0ExERU6SxMRVJvwm83/aryvLZALb/Z0udTwDfsH15Wd4IvAwYHK3tSB3b2yTNK+2f096/pGtKH/8yWozdHFMZXHnVLvtGe/VXRETMLu1jNiPfTXsyljPWmMrek+51fPOBe1qWtwLH1agzf5y2z7S9DaAklme09LWuQ1+7kLScaqsI4CclSXWF/tekmh0A/Fu3Ypgi/RZzv8UL/Rdz4m3epGIe7Xtpkt9XI549WkGTSUUd1rVvFo1Wp07byTweti8CLhqnrykjaXi0jN+r+i3mfosX+i/mxNu8fom5yYH6rcBBLcsHAvfVrDNW2/vLbi/K3wcm8HgREdGgJpPKTcBCSQsk7UM1iL62rc5a4LRyFNjxwENl19ZYbdcCp5f7pwNfaVm/TNK+khZQDf7f2NSTi4iI3TW2+8v2TklnAdcAc4DVtjdIWlHKVwFXAycCm4BHgTPGalu6Phe4UtJbgR8Cp5Q2GyRdCdwO7ATOtP1EU8+vi3pmV9wE9FvM/RYv9F/Mibd5fRHzrD6jPiIiuitn1EdERNckqURERNckqUwjSW8vU8pskPSBlvU9O92MpD+TZEkHtKzryXglnSfpzjIF0N9KmttS1qsxjzm10XSTdJCkf5R0R3nf/klZP+r0Sb1C0hxJ35b0d2W5Z2OWNFfSF8r79w5Jv9nL8bZKUpkmkl4OLAWOtH0E8MGyvmenm5F0EPBKqgMkRtb1bLzAtcDzbB8JfA84G3o35ppTG023ncCf2n4ucDxwZomx4/RJPeZPgDtalns55o8Cf2/7cOA3qOLu5Xh/IUll+rwNONf24wC2R863WQqssf247bupjow7dppibPcR4L+w60mlPRuv7a/Z3lkW11GduwS9G/OxwCbbm23vANZQxdozbG+zfXO5/wjVl918qjgvLdUuBV47LQGOQtKBwEnAp1pW92TMkvYHXgJcDGB7h+0f06PxtktSmT6HASdIukHSNyW9oKwfbeqaaSXpNcC9tr/TVtST8XbwFuCr5X6vxtyrcXUkaRA4GriBtumTgGeM0XQ6/BXVD6Kft6zr1ZgPAbYDny676z4l6Vfo3Xh30eQ0LbOepK8Dv96h6D1Ur/3TqHYhvIDq3JtDmNwUNV0xTrzvBv5jp2Yd1k3ZcepjxWz7K6XOe6h223x2pFmH+r1wbH2vxrUbSb8KfBF4h+2HpU6h9wZJrwYesL1e0sumOZw69gaOAd5u+wZJH6VHd3V1kqTSINuvGK1M0tuAL7k6UehGST+nmjBu2qabGS1eSf8BWAB8p3x5HAjcLOlYpnl6nLFeYwBJpwOvBn7bvzwpq1en9OnVuHYh6UlUCeWztr9UVt8vaV7L7OEPjN7DlHsR8BpJJwL7AftL+ht6N+atwFbbN5TlL1AllV6NdxfZ/TV9vgz8FoCkw4B9qGYg7bnpZmx/1/YzbA/aHqR60x9j+1/pwXhHSFoMvAt4je1HW4p6NeY6UxtNK1W/Ki4G7rD94Zai0aZPmna2z7Z9YHnvLgP+wfbv0aMxl8/VPZKeU1b9NtVMIT0Zb7tsqUyf1cBqSbcBO4DTyy/pvppupsenx/kYsC9wbdnCWmd7Ra/GPM70RL3iRcDvA9+VdEtZ925GmT6px/VyzG8HPlt+XGymmsJqL3o33l/INC0REdE12f0VERFdk6QSERFdk6QSERFdk6QSERFdk6QSERFdk6QSM4Kkb7TPNizpHZI+Pk6boYbjurzMkvyf29a/drKTRUo6qpzIV7f+XEl/PNF6kp4l6Qvdqh+zQ5JKzBSXU53Y1mpZWT8tJP068ELbR9r+SFvxa6lmIp6Mo6guw13XXGDcpNJez/Z9tk/uYv2YBZJUYqb4AvBqSfvCLyY7fBbwz5IulDRcrv/x3zo1lvSTlvsnS7qk3B+Q9EVJN5Xbizq03U/SpyV9t0wA+PJS9DXgGZJukXRCS/0XAq8Bzitlh5bb30taL+mfJB1e6p4i6TZJ35F0fTkZ7hzgjaXtG9tiOULSjaXsVkkLqU7yO7SsO0/Sr0q6TtLNJeaRmZDb6w2Wk3Pr9ttaf46kD5b+b5X09tr/yehvtnPLbUbcgKuApeX+SuC8cv/p5e8c4BtU17Ch3B8q93/S0s/JwCXl/ueAF5f7B1NNT9L+uH8KfLrcP5zqbOf9gEHgtlFivQQ4uWX5OmBhuX8c1VQiAN8F5pf7c8vfNwMfG6XfvwbeVO7vAzy5PQ6qmTT2L/cPoJr6Xx3q/WK5Zr+t9d9GNT/Y3q3/g9xm/i3TtMRMMrIL7Cvl71vK+jdIWk71ZTqParfTrTX7fAWwSL+chXd/SU91dS2RES+m+tLF9p2SfkB1aYOH6zyAqhl/Xwh8vuVx9i1/vwVcUqaV+VKH5u3+BXiPquuHfMn297X7DMIC/oekl1BNBT8feGYX+m31CmCVy/VsbD9YI/aYAZJUYib5MvBhSccAT7Z9c5kw8s+AF9j+v2W31n4d2rbOV9Ravhfwm7YfG+Nx93Te972AH9s+areg7BWSjqO6wNQtknar01b/c5JuKPWvkfQHVHNHtXoTMAA83/bPJG2h82sy0X5biR6dtj+alTGVmDFs/4Rql9ZqfjlAvz/wU+AhSc+kulRvJ/dLeq6kvYDXtaz/GnDWyMIoX+rXU31Rj8w4fTCwcZxwHwGeWuJ+GLhb0imlD0n6jXL/UNs32P4LqlmsD2pt207VNXk22z6falbbIzvU/3dU1xf5WRn/eXZ7TJPst9XXgBWS9i7tnz7O6xEzRJJKzDSXU13Tew2AqytVfhvYQJVsvjVKu5XA3wH/AGxrWf+fgKEy2Hw7sKJD248DcyR9F7gCeLPLZaLHsAb48zKwfyhVUnqrpO+UWEcGz88rg923USWv7wD/SLVLbreBeuCNwG2qZhA+HLjM9o+Ab5UB//OoLlY2JGm4PO6d5bVqrzfRflt9imps6dbynH53nNcjZojMUhwREV2TLZWIiOiaJJWIiOiaJJWIiOiaJJWIiOiaJJWIiOiaJJWIiOiaJJWIiOia/w/6uzjBlVsrLQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "from itertools import product\n", + "import matplotlib.pyplot as plt\n", + "\n", + "def statistic(d):\n", + " # d is the differences in height between paired plants\n", + " # the statistic is the sum of these differences\n", + " return np.sum(d)\n", + "\n", + "# compute the statistic for all possible combinations of signs on the elements of `d`\n", + "def null_distribution(d):\n", + " signs = (-1, 1) \n", + " n = len(d) # number of observations per sample\n", + " null_distribution = []\n", + " for dsigns in product(*[signs]*n):\n", + " stat = statistic(d * dsigns)\n", + " null_distribution.append(stat)\n", + " return null_distribution\n", + "\n", + "d = x - y\n", + "null_dist = null_distribution(d)\n", + "assert len(null_dist) == 2**15\n", + "bins = np.unique(null_dist).tolist()\n", + "bins.append(np.max(null_dist)+1)\n", + "plt.hist(null_dist, density=True, bins=bins)\n", + "plt.xlabel(\"Value of test statistic\")\n", + "plt.ylabel(\"Observed Frequency\")" + ] + }, + { + "cell_type": "markdown", + "id": "fa460986-ab98-448f-878d-35abe9e7c82a", + "metadata": {}, + "source": [ + "> In just 863 cases out of 32,768 the total deviation will have a positive value as great as or greater than that observed.\n", + "\n", + "The $p$-value is simply the ratio of the two." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "ee2ae5b0-8ed4-4f20-be2d-904365f7ff6a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "t-test p-value: 0.024851472010900447\n", + "permutation test p-value: 0.026336669921875\n" + ] + } + ], + "source": [ + "assert np.sum(null_dist >= statistic(d)) == 863\n", + "pvalue = np.sum(null_dist >= statistic(d)) / len(null_dist)\n", + "print(f\"t-test p-value: {res_t.pvalue}\")\n", + "print(f\"permutation test p-value: {pvalue}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e105edca-1c7e-4223-ab29-ed5db55fcf4f", + "metadata": {}, + "source": [ + "The two tests agree remarkably well in this case, suggesting the applicability of the t-test even when the original data are not strictly normally distributed." + ] + }, + { + "cell_type": "markdown", + "id": "bbcb4fdb-7191-4bc6-82c0-0fdfac445b4d", + "metadata": {}, + "source": [ + "`permutation_test` minimizes the code required to perform the same procedure. The most essential difference here compared to the independent \n", + "sample tests above is that we pass `permutation_type='samples'` instead of `permutation_type='independent'`. Note also that we can simply pass `np.sum` as the statistic because it satisfies the required interface, even with `vectorized=True`." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "dd6d90a0-e443-4ec2-a6c5-04eddccea877", + "metadata": {}, + "outputs": [], + "source": [ + "# Note that the first argument of `permutation_test` is a sequence containing all the samples.\n", + "# We only have one sample here, but it still needs to be in a sequence.\n", + "res = stats.permutation_test((d,), np.sum, alternative='greater', permutation_type='samples', vectorized=True, n_resamples=np.inf)\n", + "assert res.pvalue == pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "3ef5fc8c-00a0-4f70-b385-76f189b1f66b", + "metadata": {}, + "source": [ + "The value of the `permutation_type`, \"samples\", comes from the fact that permuting the signs of the differences is equivalent to permuting the *sample* to which each paired observation is assigned. That is, we can obtain the same $p$-value by performing a two-sample test in which observations are exchanged between the two samples in all possible ways *without* breaking up pairs. This is what `permutation_test` does with `permutation_type='samples'` when there are two or more samples." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "c27182a9-ba9d-4b7f-8fb4-66b49eee11bb", + "metadata": {}, + "outputs": [], + "source": [ + "def statistic(x, y, axis=0):\n", + " # observations will be permuted between samples `x` and `y`,\n", + " # changing the sign of the corresponding element of `d` \n", + " d = x - y\n", + " return np.sum(d, axis=axis)\n", + "\n", + "res = stats.permutation_test((x, y), statistic, alternative='greater', permutation_type='samples', n_resamples=np.inf)\n", + "assert res.pvalue == pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "fe585e67-6955-43a5-a509-0b3e1743d22b", + "metadata": {}, + "source": [ + "Other common hypothesis tests can be performed as paired-sample permutation tests. We continue with several comparisons against `scipy.stats.wilcoxon` to further illustrate the usage of `permutation_test` with `permutation_type='samples'`.\n", + "\n", + "#### Two-sample Test" + ] + }, + { + "cell_type": "markdown", + "id": "d6b915f5-6182-4060-806c-60aa242071ae", + "metadata": {}, + "source": [ + "[`wilcoxon`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html) is to `ttest_rel` as `mannwhitneyu` is to `ttest_ind`; it is a nonparametric permutation test of the null hypothesis that each observation in a pair is drawn from the same distribution. Because it is a paired-sample test, it computes the $p$-value by reassigning observations between the samples in all possible ways while maintaining the pairs." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "4b4c4b12-0e1d-4a41-a202-3682cae871b1", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "WilcoxonResult(statistic=10.0, pvalue=0.765625)" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rng = np.random.default_rng()\n", + "x = rng.random(7)\n", + "y = rng.random(7)\n", + "# Each element in `x` is paired with the corresponding element of `y` \n", + "stats.wilcoxon(x, y, alternative='greater')" + ] + }, + { + "cell_type": "markdown", + "id": "d542cee8-1f20-440f-be69-9812f593099a", + "metadata": {}, + "source": [ + "Using `wilcoxon` to compute only the statistic, we can use `permutation_test` to calculate precisely the same $p$-value." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "65a8170d-26be-4c64-9be3-d89e49699cc8", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "C:\\Users\\matth\\Desktop\\scipy\\scipy\\stats\\_morestats.py:3380: UserWarning: Sample size too small for normal approximation.\n", + " warnings.warn(\"Sample size too small for normal approximation.\")\n" + ] + }, + { + "data": { + "text/plain": [ + "0.765625" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def statistic(x, y):\n", + " return stats.wilcoxon(x, y, alternative='greater', method='approx').statistic\n", + "\n", + "res = stats.permutation_test((x, y), statistic, alternative='greater', permutation_type='samples')\n", + "res.pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "8bd72655-a67b-4493-aa7b-54a4cee5ef98", + "metadata": {}, + "source": [ + "(The warning can be safely ignored because it is referring to the $p$-value returned by the test, whereas we are using only the statistic value.)\n", + "\n", + "Again, the advantage of `permutation_test` is its ability to handles ties in the data; `wilcoxon` does not." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "3f9a43ec-861c-418d-b736-646075f65785", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "C:\\Users\\matth\\Desktop\\scipy\\scipy\\stats\\_morestats.py:3366: UserWarning: Exact p-value calculation does not work if there are zeros. Switching to normal approximation.\n", + " warnings.warn(\"Exact p-value calculation does not work if there are \"\n" + ] + }, + { + "data": { + "text/plain": [ + "WilcoxonResult(statistic=4.5, pvalue=0.8539232992870668)" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "x = rng.integers(0, 5, size=7)\n", + "y = rng.integers(0, 5, size=7)\n", + "stats.wilcoxon(x, y, method='exact')" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "226a7dd7-7dc9-4ef1-9161-b2a59bf324fd", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(4.5, 1.0)" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "res = stats.permutation_test((x, y), statistic, permutation_type='samples')\n", + "res.statistic, res.pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "f10349d8-9232-482b-92ed-74e59023dd08", + "metadata": {}, + "source": [ + "#### One-sample Test\n", + "\n", + "The `wilcoxon` statistic does not depend on the specific values in `x` and `y`, only on the values of `x - y`. Instead of passing `x` and `y` into `wilcoxon` separately, `wilcoxon` can accept a single sample - the differences between paired observations." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "c77d4c4c-efec-429b-9af1-b900308f05a5", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "WilcoxonResult(statistic=25.0, pvalue=0.0390625)" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "x = np.array([209, 200, 177, 169, 159, 169, 187])\n", + "y = np.array([151, 168, 147, 164, 166, 163, 176])\n", + "stats.wilcoxon(x - y, alternative='greater', method='exact')" + ] + }, + { + "cell_type": "markdown", + "id": "7d4d542a-a922-47ca-aebc-9328209a1da3", + "metadata": {}, + "source": [ + "As described above, this is relatively common in paired-sample tests (e.g. see also `ttest_rel`), so `permutation_test` also supports single samples as input and forms the null distribution by permuting the *signs* of each observation." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "9667f070-749c-45c2-8e51-1e284fda6a1c", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.0390625" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def statistic(d):\n", + " # return just the Mann-Whitney U statistic\n", + " return stats.wilcoxon(d, alternative='greater').statistic\n", + "\n", + "res = stats.permutation_test((x-y,), statistic, alternative='greater', permutation_type='samples')\n", + "res.pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "ecb0eb66-4c5c-4dd5-8403-a8c1a46f0ebd", + "metadata": {}, + "source": [ + "#### Gotchas\n", + "\n", + "Suppose that we wish to perform the Wilcoxon test with a two-sided alternative. We might expect that `alternative='two-sided'` should be specified everywhere the `alternative` parameter is accepted." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "353b33cc-2bfc-43ea-b870-47fb600aa879", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.078125 0.15625\n" + ] + } + ], + "source": [ + "alternative = 'two-sided'\n", + "res1 = stats.wilcoxon(x - y, alternative=alternative)\n", + "\n", + "def statistic(d):\n", + " return stats.wilcoxon(d, alternative=alternative).statistic\n", + "\n", + "res2 = stats.permutation_test((x-y,), statistic, alternative=alternative, permutation_type='samples')\n", + "\n", + "print(res1.pvalue, res2.pvalue)" + ] + }, + { + "cell_type": "markdown", + "id": "cefd2711-9f37-46fb-97c1-073362fa7041", + "metadata": {}, + "source": [ + "This guess was not correct; the `pvalue` returned by `permutation_test` is greater by a factor of two. Note that the documentation of `wilcoxon` states:\n", + "> If `alternative` is “two-sided”, [the `statistic` is] the sum of the ranks of the differences above or below zero, whichever is smaller.\n", + "\n", + "The sign information that should be carried by the statistic (\"above or below zero\") is not preserved by `statistic`. This can be corrected by passing `alternative='greater'` or `alternative='less'` in the call to `wilcoxon` so that that the statistic is always:\n", + "> the sum of the ranks of the differences above zero." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "2aa4d597-f85e-4e31-a5f3-d38b8dd6fef4", + "metadata": {}, + "outputs": [], + "source": [ + "def statistic(d):\n", + " # `alternative='less'` or `alternative='greater'` will work\n", + " return stats.wilcoxon(d, alternative='less').statistic\n", + "\n", + "res2 = stats.permutation_test((x-y,), statistic, alternative=alternative, permutation_type='samples')\n", + "\n", + "np.testing.assert_allclose(res2.pvalue, res1.pvalue, atol=1e-15)" + ] + }, + { + "cell_type": "markdown", + "id": "31159ba3-500b-468f-90cd-00647704cd1b", + "metadata": {}, + "source": [ + "### Other Tests\n", + "`permutation_test` with `permutation_type='samples'` is a versatile tool for comparing paired samples. Provided only data and a statistic, it can produce the null distribution and replicate the $p$-value of similar tests tests in SciPy, and it may be more accurate than these existing implementations, especially for small samples and when there are ties:\n", + "\n", + "- [`ttest_rel`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_rel.html)\n", + "- [`wilcoxon`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html)\n", + "- [`page_trend_test`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.page_trend_test.html)\n", + "\n", + "In addition, `permutation_test` with `permutation_type='samples'` can be used to perform tests not yet implemented in SciPy.\n", + "\n", + "However, there are yet other types of permutation tests that assume neither that samples are independent nor paired. We conclude the study of `permutation_test` with [Correlated-Sample Tests](https://nbviewer.org/github/mdhaber/scipy/blob/resampling_tutorial/doc/source/tutorial/stats/notebooks/resampling_tutorial_2c.ipynb)." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.5" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_2c.ipynb b/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_2c.ipynb new file mode 100644 index 0000000..de68a7c --- /dev/null +++ b/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_2c.ipynb @@ -0,0 +1,532 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "693a57e1-90c1-4947-8bb4-55043cafd19f", + "metadata": {}, + "source": [ + "### Correlated-Sample Tests" + ] + }, + { + "cell_type": "markdown", + "id": "ffeda6a3-f272-4d58-aa8b-a85be4c17f04", + "metadata": {}, + "source": [ + "Hollander and Wolfe's [Nonparametric Statistic methods](https://onlinelibrary.wiley.com/doi/book/10.1002/9781119196037) considers data from [[6](https://www.jci.org/articles/view/106443)], which studied the relationship between free proline (an amino acid) and total collagen (a protein often found in connective tissue) in diseased human livers.\n", + "\n", + "The `x` and `y` arrays below record the measurements of the two compounds." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "9f6a6603-3dec-4d28-b101-28c550d6e326", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "x = np.array([7.1, 7.1, 7.2, 8.3, 9.4, 10.5, 11.4]) # total collagen (mg/g dry weight of liver)\n", + "y = np.array([2.8, 2.9, 2.8, 2.6, 3.5, 4.6, 5.0]) # free proline (μ mole/g dry weight of liver)" + ] + }, + { + "cell_type": "markdown", + "id": "341c6722-3240-4baa-854f-6cab6d397557", + "metadata": {}, + "source": [ + "The text shows the results of analysis using Spearman's correlation coeefficient, a statistic sensitive to linear association between the ranks of the samples. Specifically, the null hypothesis that there is no association between total collagen and free protein is tested against the alternative that there is a positive, linear association between the two." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "26d45212-2e68-4177-ad0d-56c6560fd0f3", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "SpearmanrResult(correlation=0.7000000000000001, pvalue=0.03995834515444954)" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from scipy import stats\n", + "res_asymptotic = stats.spearmanr(x, y, alternative='greater')\n", + "res_asymptotic" + ] + }, + { + "cell_type": "markdown", + "id": "6a472b08-7ec8-4aeb-8588-8dbec497c84f", + "metadata": {}, + "source": [ + "As usual, the $p$-value of SciPy's `stats.spearmanr` approximates the probability of obtaining such an extreme value of the statistic under the null hypothesis. The small $p$-value corresponding with the positive correlation coefficient provides \"marginal evidence\" that the total collagen and free proline are positively correlated.\n", + "\n", + "However, the $p$-value of `stats.spearmanr` is based on an asymptotic approximation, which may not be accurate for such a small sample. Even if `spearmanr` did implement an exact $p$-value calculation, it is unlikely that it would support data with ties due to the limitations of common algorithms. Therefore, we consider how a permutation test oculd be used to compute an exact $p$-value.\n", + "\n", + "Under the null hypothesis, all the proline measurements are independent samples from the same distribution; they are uncorrelated with the measurements of collagen, and the observed pairings have no significance. Therefore, the null distribution is formed by computing the statistic for *all possible pairings* of proline and collagen measurements *without permuting samples*. Because `spearmanr` treats elements `x[i]` and `y[i]` as paired, we can accomplish this by computing the statistic for all possible orderings of only one of the two arrays (e.g. `y`)." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "bfce021a-2bf1-4797-affd-b90baa419591", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Text(0, 0.5, 'Observed Frequency')" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbfElEQVR4nO3df5gdVZ3n8feHAAIKIqRBTMBENoLRJwg04CCOgDom8QeyCwhmdVAwmxlxh50dH+I4i47MzqOLo47DjxgxRFkFRRAZDQMig8zAIGkQAgk/DBCgCSMBHQLIAgnf/aOqsbi593bd7nvq9u36vJ7nPl0/TlV9u+7t++1z6tQpRQRmZlZfW/U6ADMz6y0nAjOzmnMiMDOrOScCM7OacyIwM6u5rXsdQKemTp0aM2bM6HUYZmZ95eabb34sIgaareu7RDBjxgyGhoZ6HYaZWV+R9ECrdW4aMjOrOScCM7OacyIwM6s5JwIzs5pzIjAzqzknAjOzmnMiMDOrOScCM7OacyIwM6u5vruz2KzfzVj8kxen133hPT2MxCzjGoGZWc05EZiZ1ZwTgZlZzTkRmJnVnBOBmVnNORGYmdWcE4GZWc0lSwSSlkl6VNIdo5Q7SNJmScekisXMzFpLWSNYDsxtV0DSFOCLwJUJ4zAzszaSJYKIuA74zSjFPglcAjyaKg4zM2uvZ9cIJE0DjgaWlCi7UNKQpKENGzakD87MrEZ6ebH4q8BpEbF5tIIRsTQiBiNicGBgIH1kZmY10stB5waBiyQBTAXmS9oUEZf1MCYzs9rpWSKIiJkj05KWAz92EjAzq16yRCDpQuBwYKqkYeCzwDYAETHqdQEzM6tGskQQESd0UPbEVHGYmVl7vrPYzKzmnAjMzGrOicDMrOacCMzMas6JwMys5pwIzMxqzonAzKzmnAjMzGqul2MNmXXNjMU/eXF63RfeM2H3aTYRuUZgZlZzTgRmZjXnRGBmVnNOBGZmNedEYGZWc+41ZD3lnjlmvecagZlZzTkRmJnVnBOBmVnNORGYmdVcskQgaZmkRyXd0WL9Akmr8tcNkvZLFYuZmbWWskawHJjbZv39wNsjYg5wBrA0YSxmZtZCsu6jEXGdpBlt1t9QmL0RmJ4qFjMza22iXCM4Cbii1UpJCyUNSRrasGFDhWGZmU1+PU8Eko4gSwSntSoTEUsjYjAiBgcGBqoLzsysBnp6Z7GkOcB5wLyIeLyXsZiZ1VXPagSS9gIuBT4cEff0Kg4zs7pLViOQdCFwODBV0jDwWWAbgIhYApwO7AqcIwlgU0QMporHzMyaS9lr6IRR1p8MnJzq+GZmVo5HHzWzF3k02Hrqea8hMzPrLScCM7OaGzURSHpTFYGYmVlvlKkRLJF0k6Q/lbRz6oDMzKxaoyaCiDgMWADsCQxJ+q6kdyWPzMzMKlGq11BE/ErSXwFDwNeA/ZV1/v/LiLg0ZYBmNjr39rHxKHONYI6krwB3AkcC74uIN+TTX0kcn5mZJVamRnAW8A2y//6fGVkYEevzWoKZmfWxMolgPvBMRGwGkLQVsF1E/C4iLkganZmZJVem19DVwPaF+R3yZWZmNgmUSQTbRcRTIzP59A7pQjIzsyqVSQRPSzpgZEbSgcAzbcqbmVkfKXON4FTgYknr8/k9gA8mi8jMzCo1aiKIiJWS9gX2AQTcFRHPJ4/MzMwqUXYY6oOAGXn5/SUREd9OFpWZmVVm1EQg6QJgb+BWYHO+OAAnAjOzSaBMjWAQmB0RkToYMzOrXpleQ3cAr04diJmZ9UaZGsFUYI2km4BnRxZGxPvbbSRpGfBe4NGI2OKZBvmgdX9Pdufy74ATI+KWDmI3M7MuKJMIPjfGfS8nG6eo1bWEecCs/HUIcG7+06xS/TpyZzFus/Eo033055JeC8yKiKsl7QBMKbHddZJmtClyFPDt/NrDjZJ2lrRHRDxSNngzMxu/MsNQfxz4AfD1fNE04LIuHHsa8FBhfjhf1iyGhZKGJA1t2LChC4c2M7MRZS4WfwJ4K7ARsofUALt14dhqsqxpz6SIWBoRgxExODAw0IVDm5nZiDKJ4NmIeG5kRtLWtPjC7tAw2eMvR0wH1rcoa2ZmiZRJBD+X9JfA9vmzii8G/rELx74c+IgybwGe8PUBM7Pqlek1tBg4Cbgd+G/ACuC80TaSdCFwODBV0jDwWWAbgIhYku9nPrCWrPvoRzsP32zi6tfeSFY/ZXoNvUD2qMpvdLLjiDhhlPVBdv3BzMx6qMxYQ/fT5JpARLwuSURmZlapsmMNjdgOOBbYJU04ZmZWtVEvFkfE44XXwxHxVeDI9KGZmVkVyjQNHVCY3YqshrBjsojMbELwxe76KNM09HeF6U3AOuC4JNGYmVnlyvQaOqKKQMzMrDfKNA39ebv1EfHl7oVj1n/chGL9rmyvoYPI7gQGeB9wHS8dMM7MzPpU2QfTHBARTwJI+hxwcUScnDIwMzOrRplEsBfwXGH+OWBGkmisb7g5xGzyKJMILgBukvRDsjuMj6b1U8fMzKzPlOk19L8lXQG8LV/00Yj4ZdqwzMysKmVqBAA7ABsj4nxJA5JmRsT9KQMzs8nDTYkTW5lHVX4WOA34dL5oG+D/pgzKzMyqU+bBNEcD7weeBoiI9XiICTOzSaNMInguf3ZAAEh6edqQzMysSmUSwfclfR3YWdLHgavp8CE1ZmY2cbW9WCxJwPeAfYGNwD7A6RHx0wpiMzOzCrRNBBERki6LiAMBf/mbmU1CZZqGbpR00Fh2LmmupLslrZW0uMn6V0r6R0m3SVotyQ+wNzOrWJn7CI4AFklaR9ZzSGSVhTntNpI0BTgbeBcwDKyUdHlErCkU+wSwJiLeJ2kAuFvSdyLiuSa7tJrqVh/0XvZlLx67TJlW8XVrP2ZFLROBpL0i4kFg3hj3fTCwNiLuy/d3EXAUUEwEAeyYX4t4BfAbsoffmJlZRdo1DV0GEBEPAF+OiAeKrxL7nsZLh6oezpcVnQW8AVgP3A78WUS80LgjSQslDUka2rBhQ4lDm5lZWe0SgQrTrxvDvtVkWTTMvxu4FXgN8GbgLEk7bbFRxNKIGIyIwYGBgTGEYmZmrbRLBNFiuqxhYM/C/HSy//yLPgpcGpm1wP1kXVXNzKwi7RLBfpI2SnoSmJNPb5T0pKSNJfa9EpglaaakbYHj+f1TzkY8CLwDQNLuZPcp3Nf5r2FmZmPV8mJxREwZz44jYpOkU4ArgSnAsohYLWlRvn4JcAawXNLtZE1Jp0XEY+M5rllRmV42ZnVXdhjqMYmIFcCKhmVLCtPrgT9KGYOZmbVX5oYyMzObxJLWCMz6TaumpMblvlHLJhPXCMzMaq7dncVP0qbbaERs0d/fzMz6T7teQzsCSPo88O/ABWQ9exbgJ5RZAv00Rk7ZJiSzflCmaejdEXFORDwZERsj4lzgv6QOzMzMqlEmEWyWtEDSFElbSVoAbE4dmJmZVaNMr6EPAX+fvwK4Pl9mZolMhqG3rX+MmggiYh3Z8NFmZjYJjZoIJL0eOBfYPSLeJGkO8P6I+Jvk0dmYTJQHsPg/0O6YyBegfX/F5FDmGsE3gE8DzwNExCqyAeTMzGwSKJMIdoiImxqW+SliZmaTRJmLxY9J2pv85jJJxwCPJI3KLLGJ0tzSqzjchGdFZRLBJ4ClwL6SHiZ7eMyCpFGZmVllyiSCByLinZJeDmwVEU+mDsrMzKpTJhHcL+mfgO8B1ySOx2zcJkqzj1m/KHOxeB/garImovslnSXpsLRhmZlZVUZNBBHxTER8PyL+M7A/sBPw8+SRmZlZJUo9j0DS2yWdA9wCbAcclzQqMzOrzKiJQNL9wKnAvwBviojjIuKSMjuXNFfS3ZLWSlrcoszhkm6VtFqSaxpmZhVre7FY0hTg/Ij4fKc7zrc9G3gXMAyslHR5RKwplNkZOAeYGxEPStqt0+OYmdn4tE0EEbFZ0hFAx4kAOBhYGxH3AUi6iGzwujWFMh8CLo2IB/PjPTqG41hCKXrglNlnmQe/+EaoicfvT38qc43ghryn0NskHTDyKrHdNOChwvxwvqzo9cCrJF0r6WZJH2m2I0kLJQ1JGtqwYUOJQ5uZWVll7iM4NP9ZrBUEcOQo26nJssZnIG8NHAi8A9ge+DdJN0bEPS/ZKGIp2d3NDA4OtnyOspmZda7M8wiOGOO+h4E9C/PTgfVNyjwWEU8DT0u6DtgPuAczM6tEmV5Du0v6pqQr8vnZkk4qse+VwCxJMyVtSzZ09eUNZX4EvE3S1pJ2AA4B7uzsVzAzs/Eo0zS0HDgf+Ew+fw/ZcBPfbLdRRGySdApwJTAFWBYRqyUtytcviYg78+ErVgEvAOdFxB1j+k2sdjyURHV8EXhyK5MIpkbE9yV9Gl78gi/18PqIWAGsaFi2pGH+TODMkvGamVmXlek19LSkXfn98wjeAjyRNCozM6tMmRrBn5O17e8t6XpgADgmaVRmZhVxs1e5XkO3SHo72SikAu6OiOeTR2ZmZpUo02voWGD7iFgNfAD4XskbyszMrA+UaRr6XxFxcf4MgncDXwLOJevqaRXr12qse/j0XtXvQerj9evfwkRU5mLxSA+h9wDnRsSPgG3ThWRmZlUqkwgelvR1smcQrJD0spLbmZlZHyjTNHQcMBf4UkT8h6Q9gE+lDcvM6szNPtUq86jK3wHrgHmSPgnsERFXpQ7MzMyqUabX0OnAt4BdganA+ZL+KnVgZmZWjTJNQycA+0fE/wOQ9AWyZxf/TcrALC1XvW2sxtsbyJ+9iafMRd91ZA+sH/Ey4N4k0ZiZWeVa1ggk/QPZ+ELPAqsl/TSffxfwr9WEZ2ZmqbVrGhrKf94M/LCw/Npk0ZiZJeImqdZaJoKI+BaApO2A/0RWG7h35FqBmZlNDu2ahrYG/hb4GPAA2fWE6ZLOBz7jgee6L/V/LGUu8nkoCJto/J98eu0uFp8J7ALMjIgDI2J/YG9gZ7LxhszMbBJolwjeC3w8Ip4cWRARG4E/AeanDszMzKrR7mJxREQ0WbhZ0hbLzcw65WafiaFdjWCNpI80LpT0X4G7yuxc0lxJd0taK2lxm3IHSdosyU8+MzOrWLsawSeASyV9jKwLaQAHAdsDR4+2Y0lTgLPJ7jsYBlZKujwi1jQp90XgyjH9BmZmNi7tuo8+DBwi6UjgjWSPqbwiIn5Wct8HA2sj4j4ASRcBRwFrGsp9EriELMnYJOIeSGb9ocwzi68BrhnDvqcBDxXmh2l4qpmkaWS1iyNpkwgkLQQWAuy1115jCMXMzFpJ+YAZNVnWeJH5q8BpEbG5SdnfbxSxNCIGI2JwYGCgW/GZmRnlRh8dq2Fgz8L8dGB9Q5lB4CJJkA1xPV/Spoi4LGFcZtan3NyYRspEsBKYJWkm8DBwPPChYoGImDkyLWk58GMnATOzaiVLBBGxSdIpZL2BpgDLImK1pEX5+iWpjm1mZuWlrBEQESuAFQ3LmiaAiDgxZSzWWjcfNGL9ZzK/f5P5d+umlBeLzcysDzgRmJnVXNKmITOzqrk5qHOuEZiZ1ZxrBD1QZsRFj8poVl6KWkCd/gZdIzAzqzknAjOzmnMiMDOrOScCM7OacyIwM6s59xqa5Nyn2vqFP6u94xqBmVnNORGYmdWcm4bMzDowGW80c43AzKzmnAjMzGrOTUMd6FWVsExvCve4MLOxco3AzKzmnAjMzGouaSKQNFfS3ZLWSlrcZP0CSavy1w2S9ksZj5mZbSlZIpA0BTgbmAfMBk6QNLuh2P3A2yNiDnAGsDRVPGZm1lzKi8UHA2sj4j4ASRcBRwFrRgpExA2F8jcC0xPGU4lOLyj7Iq9Z/5os9xSkbBqaBjxUmB/Ol7VyEnBFsxWSFkoakjS0YcOGLoZoZmYpE4GaLIumBaUjyBLBac3WR8TSiBiMiMGBgYEuhmhmZimbhoaBPQvz04H1jYUkzQHOA+ZFxOMJ4zEzsyZS1ghWArMkzZS0LXA8cHmxgKS9gEuBD0fEPQljMTOzFpLVCCJik6RTgCuBKcCyiFgtaVG+fglwOrArcI4kgE0RMZgqJjMz21LSISYiYgWwomHZksL0ycDJKWPoJfcIMpuYqv7bnOi9i3xnsZlZzTkRmJnVnEcfNTPrgn5uCnaNwMys5pwIzMxqzk1DufFc1e9VlbCfq6JmE9FE+ZuqupeRawRmZjXnRGBmVnNuGjIzq9BEvLnMNQIzs5pzjWCMJspFJTNLb7L/vbtGYGZWc04EZmY150RgZlZzTgRmZjXnRGBmVnO16jU0liv/k723gJn1TpnvlyruO3CNwMys5pwIzMxqLmkikDRX0t2S1kpa3GS9JH0tX79K0gEp4zEzsy0lSwSSpgBnA/OA2cAJkmY3FJsHzMpfC4FzU8VjZmbNpawRHAysjYj7IuI54CLgqIYyRwHfjsyNwM6S9kgYk5mZNUjZa2ga8FBhfhg4pESZacAjxUKSFpLVGACeknR3h7FMBR4rW1hf7HDv49NRbBWaqHGBYxsrxzY2Eya2Jt9NncT22lYrUiYCNVkWYyhDRCwFlo45EGkoIgbHun1KEzW2iRoXOLaxcmxjU4fYUjYNDQN7FuanA+vHUMbMzBJKmQhWArMkzZS0LXA8cHlDmcuBj+S9h94CPBERjzTuyMzM0knWNBQRmySdAlwJTAGWRcRqSYvy9UuAFcB8YC3wO+CjicIZc7NSBSZqbBM1LnBsY+XYxmbSx6aILZrkzcysRnxnsZlZzTkRmJnV3KRIBJKOlbRa0guSWnalajXkhaRdJP1U0q/yn6/qYmyj7lvSPpJuLbw2Sjo1X/c5SQ8X1s2vMra83DpJt+fHH+p0+1SxSdpT0j9LujN///+ssK7r5208Q6aMtm0FsS3IY1ol6QZJ+xXWNX1/K4ztcElPFN6r08tuW0FsnyrEdYekzZJ2ydclO2+Slkl6VNIdLdZ397MWEX3/At4A7ANcCwy2KDMFuBd4HbAtcBswO1/3f4DF+fRi4ItdjK2jfedx/jvw2nz+c8BfJDpvpWID1gFTx/u7dTs2YA/ggHx6R+Cewnva1fPW7vNTKDMfuILs/pi3AL8ou20FsR0KvCqfnjcSW7v3t8LYDgd+PJZtU8fWUP59wDUVnbc/BA4A7mixvquftUlRI4iIOyNitLuN2w15cRTwrXz6W8AHuhhep/t+B3BvRDzQxRhaGe/v3dPzFhGPRMQt+fSTwJ1kd6anMJ4hU8psmzS2iLghIn6bz95Ids9OFcbzu/f8vDU4Abiwi8dvKSKuA37TpkhXP2uTIhGU1Go4C4DdI79/If+5WxeP2+m+j2fLD9spefVvWTebXzqILYCrJN2sbLiPTrdPGRsAkmYA+wO/KCzu5nlr9/kZrUyZbVPHVnQS2X+TI1q9v1XG9geSbpN0haQ3drht6tiQtAMwF7iksDjleRtNVz9rffOEMklXA69usuozEfGjMrtosqwrfWfbxdbhfrYF3g98urD4XOAMsljPAP4O+FjFsb01ItZL2g34qaS78v9YxqWL5+0VZH+gp0bExnzxuM5bs8M0WVZ2yJRkn71RjrtlQekIskRwWGFxkve3g9huIWsKfSq/lnMZ2YjEE+a8kTULXR8Rxf/SU5630XT1s9Y3iSAi3jnOXbQbzuLXkvaIiEfy6tWj3YpNUif7ngfcEhG/Luz7xWlJ3wB+XHVsEbE+//mopB+SVT+vYwKcN0nbkCWB70TEpYV9j+u8NTGeIVO2LbFt6tiQNAc4D5gXEY+PLG/z/lYSWyF5ExErJJ0jaWqZbVPHVrBFTT3xeRtNVz9rdWoaajfkxeXAH+fTfwyUqWGU1cm+t2iD1EuH5T4aaNqLIFVskl4uaceRaeCPCjH09LxJEvBN4M6I+HLDum6ft/EMmVJm26SxSdoLuBT4cETcU1je7v2tKrZX5+8lkg4m+156vMy2qWPLY3ol8HYKn8EKzttouvtZS3HFu+oX2R/6MPAs8Gvgynz5a4AVhXLzyXqW3EvWpDSyfFfgZ8Cv8p+7dDG2pvtuEtsOZB/+VzZsfwFwO7Aqf0P3qDI2st4Ht+Wv1RPpvJE1b0R+bm7NX/NTnbdmnx9gEbAonxbZw5juzY892G7bLv8NjBbbecBvC+dpaLT3t8LYTsmPfRvZhexDJ8p5y+dPBC5q2C7peSP7h/AR4Hmy77aTUn7WPMSEmVnN1alpyMzMmnAiMDOrOScCM7OacyIwM6s5JwIzs5pzIrCekXStpHc3LDtV0jmjbJP0QeKSLsyHpvgfDcs/IGn2GPf5ZnUwAqqknSX9aaflJL1G0g+6Vd7qwYnAeulCshteipqNtVQZSa8m68c+JyK+0rD6A8CYEgHwZrL+3WXtDIyaCBrLRcT6iDimi+WtBpwIrJd+ALxX0svgxYHjXgP8q6RzJQ0pe87AXzfbWNJTheljJC3PpwckXSJpZf56a5Ntt5N0vrLx5H+Zj8EDcBWwm7Ix5t9WKH8o2ThQZ+br9s5f/6Rs0LF/kbRvXvZYZWPX3ybpuvwOz88DH8y3/WBDLG+UdFO+bpWkWcAXgL3zZWdKeoWkn0m6JY95ZETJxnIzlI9hX3K/xfJTJH0p3/8qSZ8s/U5af+v2nXp++dXJC/gJcFQ+vRg4M58euZN4CtlzJubk89eS30UJPFXYzzHA8nz6u8Bh+fReZENQNB73fwLn59P7Ag8C2wEzaD0G/HLgmML8z4BZ+fQh5GPVk93pOS2f3jn/eSJwVov9/gOwIJ/eFti+MQ6yccF2yqenAmvJ7i5tLPfifMn9Fsv/Cdm4TVsX3wO/Jv+rbwads0lrpHnoR/nPkRFCj1M2tO/WZA+gmU02XEQZ7wRm58PXAOwkacfInlkw4jCyL0oi4i5JDwCvBzZSgrIRTw8FLi4c52X5z+uB5ZK+Tza+z2j+DfiMpOnApRHxq8I+Xzwk8LeS/hB4gWxo4d27sN+idwJLImITQLx0pE2bxJwIrNcuA76s7FF720fELZJmAn8BHBQRv82bfLZrsm1xfJTi+q2AP4iIZ9oct+03YglbAf8REW/eIqiIRZIOAd4D3CppizIN5b8r6Rd5+SslnQzc11BsATAAHBgRz0taR/Nz0ul+i0R3h3m2PuFrBNZTEfEUWXPPMn5/kXgn4GngCUm7kw3P3cyvJb1B0lZkAw+OuIpsIDMg67HTZNvryL5ckfR6siak0Z5y9yTZIzGJbOjk+yUdm+9Dyp8DLGnviPhFRJwOPEY2LPCL2zaS9Drgvoj4GtkAeXOalH8l8GieBI4AXtsY0xj3W3QVsEjS1vn2u4xyPmyScCKwieBCYD+yx+oREbcBvyQb1XEZWVNLM4vJnjNwDdlIjSP+OzCYX/BcQzZqY6NzgCmSbge+B5wYEc+OEudFwKfyi8t7kyWSkySNjEA5cgH3zPyC6x1kCec24J/Jmqu2uFgMfBC4Q9KtZNcrvh3Z8wKuzy86nwl8J/+dhvLj3pWfq8Zyne636DyyayWr8t/pQ6OcD5skPPqomVnNuUZgZlZzTgRmZjXnRGBmVnNOBGZmNedEYGZWc04EZmY150RgZlZz/x9bRm393vhxIwAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "from itertools import permutations\n", + "import matplotlib.pyplot as plt\n", + "\n", + "def statistic(x, y):\n", + " return stats.spearmanr(x, y, alternative='greater').correlation\n", + "\n", + "def null_distribution(x, y):\n", + " # the order of `x` can remain fixed\n", + " # By generating all possible orderings of `y` (alone),\n", + " # we explore all possible pairings between observations\n", + " # in `x` and `y`\n", + " null_distribution = []\n", + " for yperm in permutations(y):\n", + " stat = statistic(x, yperm)\n", + " null_distribution.append(stat)\n", + " return null_distribution\n", + "\n", + "null_dist = null_distribution(x, y)\n", + "plt.hist(null_dist, density=True, bins=100)\n", + "plt.xlabel(\"Value of test statistic\")\n", + "plt.ylabel(\"Observed Frequency\")" + ] + }, + { + "cell_type": "markdown", + "id": "e5709630-c43b-49d7-a760-ef5c46b55443", + "metadata": {}, + "source": [ + "The $p$-value is the percentage of values in the null distribution that equal or exceed the observed value of the statistic." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "d0e86da4-b5d0-4a5b-9cbe-1f1192bce745", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Asymptotic p-value: 0.03995834515444954\n", + "Exact p-value: 0.04563492063492063\n" + ] + } + ], + "source": [ + "pvalue = np.sum(null_dist >= statistic(x, y)) / len(null_dist)\n", + "print(f\"Asymptotic p-value: {res_asymptotic.pvalue}\")\n", + "print(f\"Exact p-value: {pvalue}\")" + ] + }, + { + "cell_type": "markdown", + "id": "f272d090-d956-426d-86f4-16018675db28", + "metadata": {}, + "source": [ + "The asymptotic p-value is reasonably accurate in this case, but it smaller. This suggests that the asymptotic approximation is not conservative; that is, it is more likely to lead to _Type I error_ - being taken as evidence against the null hypothesis even when the null hypothesis is actually true.\n", + "\n", + "`permutation_test` can perform the same test using `permutation_type='pairings'` (so named because it forms the null distribution by permuting the pairings of the observations without permuting observations between samples)." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "26863c88-a0c5-4bb6-b20b-c9265c8702b3", + "metadata": {}, + "outputs": [], + "source": [ + "def statistic(y):\n", + " return stats.spearmanr(x, y).correlation\n", + "res = stats.permutation_test((y,), statistic, alternative='greater', permutation_type='pairings', n_resamples=np.infty)\n", + "assert res.pvalue == pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "1e176203-d762-43a9-8a13-95830b8bcdaf", + "metadata": {}, + "source": [ + "Many other correlation tests can be performed as permutation tests. We continue with another example to help avoid common pitfalls in the usage of `permutation_test` with `permutation_type='pairings'`." + ] + }, + { + "cell_type": "markdown", + "id": "713434e9-ad0e-4de0-a2ab-c1c9f15fa3fe", + "metadata": {}, + "source": [ + "#### Gotchas\n", + "\n", + "Another example of a correlation test in SciPy is `scipy.stats.kendalltau`." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "c06f4848-7dc1-4d38-9c10-c3b75d5bbedd", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "KendalltauResult(correlation=-0.39999999999999997, pvalue=0.48333333333333334)" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Each element in `x` is paired with the corresponding element of `y`\n", + "rng = np.random.default_rng()\n", + "x = rng.random(5)\n", + "y = rng.random(5) \n", + "stats.kendalltau(x, y, alternative='two-sided')" + ] + }, + { + "cell_type": "markdown", + "id": "b1d8e8f8-6350-4e8a-9e04-cb9aa9be6b34", + "metadata": {}, + "source": [ + "Like `mannwhitneyu` and `wilcoxon`, `kendalltau` computes its p-value using permutations. Using `kendalltau` to compute only the statistic, we could compute the same $p$-value with `permutation_test`." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "b6039a81-5a7f-4b17-9372-91644d1f522b", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.4816" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def statistic(x, y):\n", + " return stats.kendalltau(x, y).correlation\n", + "res = stats.permutation_test((x, y,), statistic, alternative='two-sided', permutation_type='pairings')\n", + "res.pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "14f3d86c-a40c-4b52-b0f5-4044756e5b3f", + "metadata": {}, + "source": [ + "What happened here? In all cases before, `permutation_test` produced the exact p-value, but here we have only a four-digit approximation.\n", + "\n", + "Note that the null distribution contains only 9999 elements, the default for a randomized test." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "83f73b8b-1112-4f69-812e-6354fcf348c0", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "9999" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "len(res.null_distribution)" + ] + }, + { + "cell_type": "markdown", + "id": "7680292d-e64c-4827-aaaa-21a4b42a58fe", + "metadata": {}, + "source": [ + "If we were to allow for unlimited permutations, `permutation_test` would eventually compute the exact answer. " + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "6ea2531c-b691-4c03-b644-426311f1d466", + "metadata": {}, + "outputs": [], + "source": [ + "res = stats.permutation_test((x, y), statistic, alternative='two-sided', permutation_type='pairings', n_resamples=np.inf)" + ] + }, + { + "cell_type": "markdown", + "id": "21aa5416-f1f3-4304-a8e9-12d800aaab49", + "metadata": {}, + "source": [ + "Then we can compute the $p$-value as the percentage of elements in the null distribution as extreme as the observed value of the test statistic. In this case, either large or small values are considered more extreme because `alternative='two-sided'`. " + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "47c98f9d-5412-4eee-ba26-711088cdd83f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.48333333333333334\n" + ] + } + ], + "source": [ + "pvalue = np.sum(np.abs(res.null_distribution) >= np.abs(statistic(x, y)) ) / len(res.null_distribution)\n", + "print(pvalue)" + ] + }, + { + "cell_type": "markdown", + "id": "fd18314e-6613-4f56-8637-1ead152e627c", + "metadata": {}, + "source": [ + "Note that this definition only makes sense for distributions that are symmetric about a known median. To produce the same value for symmetric distributions but generalize to asymmetric distributions, `permutation_test` actually computes the p-value by doubling the minimum of the `'greater'` and `'less'` p-values." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "1cf372d4-209e-4861-9580-7368c1968cdb", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.48333333333333334\n" + ] + } + ], + "source": [ + "pvalue_greater = np.sum(res.null_distribution >= statistic(x, y) ) / len(res.null_distribution)\n", + "pvalue_less = np.sum(res.null_distribution <= statistic(x, y) ) / len(res.null_distribution)\n", + "pvalue = 2 * min(pvalue_greater, pvalue_less)\n", + "print(pvalue)" + ] + }, + { + "cell_type": "markdown", + "id": "a912f79d-1fae-42e6-9c2c-744b36c1e285", + "metadata": {}, + "source": [ + "But let's step back a minute - theoretically, there are only $5!=120$ possible pairings of the observations between the two samples, so why did it take so many resamples to compute an exact answer? \n", + "\n", + "`permutation_test` permutes the orders of _all_ provided samples, so we computed all possible permutations of both `x` and `y`. That's $5! \\cdot 5!=14,400$ permutations:" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "06f986f4-0662-44b7-9509-9a58de803544", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "14400" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "len(res.null_distribution)" + ] + }, + { + "cell_type": "markdown", + "id": "e20badf7-1766-4181-80c0-ae71d465ec70", + "metadata": {}, + "source": [ + "This is $5!$ times the amount of work it needed to do, since only the pairings between observations in `x` and `y` affect the statistic, not the order of the pairs within the arrays. We improve efficiency by leaving `x` out of the call to `permutation_test` and simply include it as part of the `statistic` itself." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "9a0f9fe9-2eb3-49d0-bf23-555e4048bdb7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.48333333333333334\n" + ] + } + ], + "source": [ + "def statistic(y):\n", + " return stats.kendalltau(x, y, alternative='two-sided').correlation\n", + "\n", + "res = stats.permutation_test((y,), statistic, alternative='two-sided', permutation_type='pairings')\n", + "assert len(res.null_distribution) == 120\n", + "print(res.pvalue)" + ] + }, + { + "cell_type": "markdown", + "id": "86228230-34c8-4b15-8d32-02329ede4d90", + "metadata": {}, + "source": [ + "Again, `permutation_test` is particularly useful when there are ties because, according to the [`kendalltau` documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html):\n", + "\n", + "> ‘exact’: computes the exact p-value, but can only be used if no ties are present. " + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "9fa298f8-5e90-44bb-9c4a-57f88a3432aa", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "ValueError: Ties found, exact method cannot be used.\n" + ] + } + ], + "source": [ + "# with more observations than distinct values,\n", + "# there will be ties within and between samples\n", + "x = rng.integers(0, 5, size=7) \n", + "y = rng.integers(0, 5, size=7)\n", + "try:\n", + " stats.kendalltau(x, y, method='exact', alternative='two-sided')\n", + "except ValueError as e:\n", + " print(f\"{type(e).__name__}: {e}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b8d88171-4457-4140-b34a-239edfdf6d70", + "metadata": {}, + "source": [ + "`permutation_test` has no such restriction. Since we are using `kendalltau` only to compute the correlation statistic, we can pass option `method='asymptotic'` to avoid the computational expense of computing exact p-values." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "899c532f-b6b3-4819-b58d-445c724624be", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.5714285714285714" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def statistic(y):\n", + " return stats.kendalltau(x, y, alternative='two-sided', method='asymptotic').correlation\n", + "\n", + "res = stats.permutation_test((y,), statistic, alternative='two-sided', permutation_type='pairings')\n", + "res.pvalue" + ] + }, + { + "cell_type": "markdown", + "id": "427a5fa2-096a-419e-9f68-29338f6ef4c5", + "metadata": {}, + "source": [ + "### Other Tests\n", + "`permutation_test` with `permutation_type='pairings'` is a versatile tool for assessing association between samples. Provided only data and a statistic, it can produce the null distribution and replicate the $p$-value of similar tests tests in SciPy, and it may be more accurate than these existing implementations, especially for small samples and when there are ties:\n", + "\n", + "- [`pearsonr`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html)\n", + "- [`spearmanr`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html)\n", + "- [`kendalltau`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html)\n", + "- [`somersd`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.somersd.html)\n", + "- [`linregress`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.linregress.html)\n", + "\n", + "In addition, `permutation_test` with `permutation_type='pairings'` can be used to perform tests not yet implemented in SciPy.\n", + "\n", + "But there is much more to statistics than $p$-values! We conclude with a discussion of one of the most versatile techniques of all: [the bootstrap](https://nbviewer.org/github/mdhaber/scipy/blob/resampling_tutorial/doc/source/tutorial/stats/notebooks/resampling_tutorial_3.ipynb)." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.5" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_3.ipynb b/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_3.ipynb new file mode 100644 index 0000000..4d7b58d --- /dev/null +++ b/ipython/ResamplingAndMonteCarloMethods/resampling_tutorial_3.ipynb @@ -0,0 +1,799 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "5ddad2d4-6b8e-417b-b2f1-818ec162c523", + "metadata": {}, + "source": [ + "## The Bootstrap\n", + "\"Bootstrapping\" refers to computational techniques for making inferences about a statistic beyond point estimates by treating the samples as though they were they were the populations of interest. Regarding the origin of the term, [An Introduction to the Bootstrap](https://cindy.informatik.uni-bremen.de/cosy/teaching/CM_2011/Eval3/pe_efron_93.pdf) states:\n", + "\n", + "> The use of the term bootstrap derives from the phrase *to\n", + "pull oneself up by one's bootstrap*, widely thought to be based on\n", + "one of the eighteenth century Adventures of Baron Munchausen,\n", + "by Rudolph Erich Raspe. (The Baron had fallen to the bottom of\n", + "a deep lake. Just when it looked like all was lost, he thought to\n", + "pick himself up by his own bootstraps.)\n", + "\n", + "Let us return to the experiment that we considered at the beginning of the discussion of permutation tests. Again, a new medical treatment is intended to prolong life after a form of surgery. Sixteen mice are randomly assigned to either a treatment group or control group. All mice receive the surgery, but only the treatment group will receive the new treatment. The survival time of each mouse after surgery is recorded below.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "7581fc1a-795e-41b1-9de5-4167100d91d5", + "metadata": {}, + "outputs": [], + "source": [ + "# survival times measured in days\n", + "import numpy as np\n", + "x = np.array([94, 197, 16, 38, 99, 141, 23]) # treatment group\n", + "y = np.array([52, 104, 146, 10, 51, 30, 40, 27, 46]) # control group" + ] + }, + { + "cell_type": "markdown", + "id": "ca5fe064-d6b2-493b-8609-4a8d3b7be900", + "metadata": {}, + "source": [ + "The permutation test allowed us to study whether or not the treatment had any effect on the survival times. In many studies, we are interested not only in whether there is an effect; we are also interested in the _magnitude_ of the effect. It would be misleading to report only the difference in mean survival times, especially since the permutation test and t-test showed that there was a ~$14\\%$ chance of observing such an extreme difference in means due to chance alone. In addition to reporting our statistic (the difference in means), we should also report some measurement of our uncertainty.\n", + "\n", + "One way of quantifying our uncertainty is the _standard error_ of our statistic. Suppose we were to perform the same experiment (with new mice) repeatedly. Because the mice are random samples from some greater population and there will be some random error in the effect of the treatment, we would not observe the same value of the statistic every time; rather, the values of the statistic would form a distribution. The standard error is the standard deviation of this distribution.\n", + "\n", + "How do we calculate the standard error if we do not know the underlying distribution from which the mice survival times are sampled? The typical approach, which we will not discuss in detail, assumes that the underlying distributions are normal; from this assumption and some math, statisticians have derived a formula to estimate the standard error of the statistic. This approach is limited in applicability, however, as it may not produce a good estimate if the original distributions are non-normal; moreover, standard error formulas are only available for a few statistics. Instead, we take a different approach, beginning with the mild assumption that the observed samples are representative of the distributions from which they were taken. We estimate the standard error by repeatedly *resampling from the observed data* (with replacement), calculating the statistic of the resample each time, and computing the standard deviation of the resulting distribution. This makes sense: to estimate the standard error, we would happily re-sample from the distribution itself it if were available to us. It's not, so we do the next best thing, which is re-sampling from the data we already have." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "e30a2a3c-dae6-4c29-aa9f-d585a4b39213", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Observed Statistic Value: 30.63492063492064\n", + "Standard Error: 27.336478035417056\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAd7ElEQVR4nO3de7gdVZ3m8e9LkJsNHTFBMQmdgBGMDGoMl0ZtL0ibBCTaAwqtD4i0mSjYareXII7d+vTMkxFHGlokRkTAC4g2QkZCA2Iro2MgIUK4awgRDkSJ2nIRmxh854+qA5vNPntXnZw6Zyd5P8+zn1O1aq21fzs52b9Uraq1ZJuIiIiqthvrACIiYsuSxBEREbUkcURERC1JHBERUUsSR0RE1LL9WAcwGiZMmOCpU6eOdRgREVuUG2+88Ve2J7aXbxOJY+rUqaxcuXKsw4iI2KJI+nmn8lyqioiIWpI4IiKiliSOiIioJYkjIiJqSeKIiIhakjgiIqKWJI6IiKgliSMiImpJ4oiIiFq2iSfHI5o0deEVw267btERIxhJxOjIGUdERNSSxBEREbUkcURERC1JHBERUUsSR0RE1JLEERERtSRxRERELUkcERFRS6OJQ9JsSXdJWiNpYYfjknRWeXy1pJktx86T9KCkW4fo+0OSLGlCk58hIiKerrHEIWkccDYwB5gBHCdpRlu1OcD08jUfOKfl2PnA7CH6ngIcDtw7slFHREQvTZ5xHASssb3W9kbgYmBeW515wIUuLAfGS9oTwPZ1wG+G6PsM4COAmwk9IiKG0mTimATc17I/UJbVrfM0ko4C7rd9c4968yWtlLRyw4YN1aOOiIiumkwc6lDWfoZQpc5TlaVdgNOAT/R6c9tLbM+yPWvixIm9qkdEREVNJo4BYErL/mTggWHUabUPMA24WdK6sv4qSc/f7GgjIqKSJhPHCmC6pGmSdgCOBZa21VkKHF/eXXUI8JDt9UN1aPsW23vYnmp7KkXimWn7Fw19hoiIaNNY4rC9CTgFuAq4A7jE9m2SFkhaUFZbBqwF1gBfBN472F7SRcCPgX0lDUg6qalYIyKiukYXcrK9jCI5tJYtbtk2cPIQbY+r0P/UzQwxIiJqypPjERFRSxJHRETUksQRERG1JHFEREQtjQ6OR0R3UxdesVnt1y06YoQiiaguiSNiC7Y5iSdJJ4Yrl6oiIqKWJI6IiKgliSMiImpJ4oiIiFqSOCIiopYkjoiIqCWJIyIiakniiIiIWvIAYASb/wR3xLYkZxwREVFLEkdERNSSxBEREbUkcURERC2NDo5Lmg2cCYwDzrW9qO24yuNzgceAd9peVR47DzgSeND2/i1tTgfeBGwE7gZOtP3bJj9HbBkywB0xOho745A0DjgbmAPMAI6TNKOt2hxgevmaD5zTcux8YHaHrq8B9rd9APBT4NSRjTwiIrpp8lLVQcAa22ttbwQuBua11ZkHXOjCcmC8pD0BbF8H/Ka9U9tX295U7i4HJjf2CSIi4hmaTByTgPta9gfKsrp1unkXcGWnA5LmS1opaeWGDRtqdBkREd00mTjUoczDqNO5c+k0YBPwtU7HbS+xPcv2rIkTJ1bpMiIiKmhycHwAmNKyPxl4YBh1nkHSCRQD54fZrpRoIiJiZDR5xrECmC5pmqQdgGOBpW11lgLHq3AI8JDt9d06Le/U+ihwlO3Hmgg8IiKG1ljiKAewTwGuAu4ALrF9m6QFkhaU1ZYBa4E1wBeB9w62l3QR8GNgX0kDkk4qD30O2BW4RtJNkhY39RkiIuKZGn2Ow/YyiuTQWra4ZdvAyUO0PW6I8heOZIwREVFPnhyPiIhakjgiIqKWnolD0v696kRExLajyhnHYkk3SHqvpPFNBxQREf2tZ+Kw/Srg7RTPW6yU9HVJhzceWURE9KVKYxy2fwZ8nOL5idcAZ0m6U9JfNRlcRET0nypjHAdIOoPiWYzXA2+y/eJy+4yG44uIiD5T5TmOz1E8nPcx278fLLT9gKSPNxZZRET0pSqJYy7we9tPAEjaDtjJ9mO2v9JodBER0XeqjHF8F9i5ZX+XsiwiIrZBVc44drL96OCO7Ucl7dJgTLGNytKvEVuGKmccv5M0c3BH0iuA33epHxERW7EqZxwfAL4paXCdjD2BtzUWUURE9LWeicP2Ckn7AftSrNh3p+0/NB5ZRET0parTqh8ITC3rv1wSti9sLKqIiOhbPROHpK8A+wA3AU+UxQaSOCIitkFVzjhmATOytndEREC1u6puBZ7fdCAREbFlqHLGMQG4XdINwOODhbaPaiyqiIjoW1USxz8Ot3NJs4EzgXHAubYXtR1XeXwu8BjwTturymPnAUcCD9rev6XN7sA3KAbr1wFvtf0fw40xIiLqqbIexw8ovqCfVW6vAFb1aidpHHA2MAeYARwnaUZbtTnA9PI1Hzin5dj5wOwOXS8ErrU9Hbi23I+IiFFSZVr1dwPfAr5QFk0CLqvQ90HAGttrbW8ELgbmtdWZB1zownJgvKQ9AWxfB/ymQ7/zgAvK7QuAN1eIJSIiRkiVwfGTgVcCD8OTizrtUaHdJOC+lv2BsqxunXbPs72+jGX9ULFImi9ppaSVGzZsqBBuRERUUSVxPF6eMQAgaXuK5zh6UYey9nZV6gyL7SW2Z9meNXHixJHoMiIiqJY4fiDpY8DO5Vrj3wT+T4V2AxTrlA+aDDwwjDrtfjl4Oav8+WCFWCIiYoRUSRwLgQ3ALcB/A5ZRrD/eywpguqRpknYAjgWWttVZChyvwiHAQ4OXobpYCpxQbp8AXF4hloiIGCFVJjn8I8XSsV+s07HtTZJOAa6iuB33PNu3SVpQHl9MkYTmAmsobsc9cbC9pIuA1wITJA0A/2D7S8Ai4BJJJwH3AsfUiSsiIjZPlbmq7qHDuIPtvXu1tb2MIjm0li1u2TbF4HuntscNUf5r4LBe7x0REc2oOlfVoJ0o/oe/ezPhREREv6vyAOCvW1732/5n4PXNhxYREf2oyqWqmS2721GcgezaWEQREdHXqlyq+t8t25so54dqJJqIiOh7Ve6qet1oBBIREVuGKpeq/q7bcdufHblwIiKi31W9q+pAnnp4703AdTx9jqmIiNhGVF3IaabtRwAk/SPwTdt/02RgERHRn6pMObIXsLFlfyPFIkoREbENqnLG8RXgBknfpniC/C3AhY1GFRERfavKXVX/Q9KVwKvLohNt/6TZsCIiol9VuVQFsAvwsO0zgQFJ0xqMKSIi+liVpWP/AfgocGpZ9Czgq00GFRER/avKGcdbgKOA3wHYfoBMORIRsc2qkjg2ltOfG0DSs5sNKSIi+lmVxHGJpC8A4yW9G/guNRd1ioiIrUfXu6okCfgGsB/wMLAv8Anb14xCbBER0Ye6Jg7blnSZ7VcASRbR09SFV4x1CBHRsCqXqpZLOrDxSCIiYotQJXG8jiJ53C1ptaRbJK2u0rmk2ZLukrRG0sIOxyXprPL46tZFo4ZqK+llkpZLuknSSkkHVYklIiJGxpCXqiTtZfteYM5wOpY0DjgbOBwYAFZIWmr79pZqc4Dp5etg4Bzg4B5tPw180vaVkuaW+68dTowREVFftzOOywBs/xz4rO2ft74q9H0QsMb2WtsbgYuBeW115gEXurCc4s6tPXu0NbBbuf2nwAMVYomIiBHSbXBcLdt7D6PvSTx9zY4BirOKXnUm9Wj7AeAqSZ+hSHyHdnpzSfOB+QB77bXXMMKPiIhOup1xeIjtqtShrL2foep0a/se4IO2pwAfBL7U6c1tL7E9y/asiRMnVgw5IiJ66XbG8VJJD1N8ie9cblPu2/ZuQzcFirOEKS37k3nmZaWh6uzQpe0JwPvL7W8C5/aIIyIiRtCQZxy2x9nezfautrcvtwf3eyUNgBXAdEnTJO0AHMtTy88OWgocX95ddQjwkO31Pdo+ALym3H498LPKnzYiIjZblYWchsX2JkmnAFcB44DzbN8maUF5fDGwDJgLrAEeA07s1rbs+t3AmZK2B/6TchwjIiJGR2OJA8D2Mork0Fq2uGXbwMlV25blPwReMbKRRkREVVUXcoqIiACSOCIioqZuT44/QpfbcCsOkEdExFZmyMRhe1cASZ8CfgF8heJW3LeTFQAjIrZZVS5VvdH2520/Yvth2+cA/7XpwCIioj9VuavqCUlvp5gvysBxwBONRhURjductVPWLTpiBCOJLU2VM46/Bt4K/LJ8HVOWRUTENqjnGYftdTxzVtuIiNhG9TzjkPQiSddKurXcP0DSx5sPLSIi+lGVS1VfBE4F/gBgezXF3FEREbENqpI4drF9Q1vZpiaCiYiI/lclcfxK0j6UDwNKOhpY32hUERHRt6rcjnsysATYT9L9wD0UDwFGRMQ2qEri+LntN0h6NrCd7UeaDioiIvpXlUtV90haAhwCPNpwPBER0eeqJI59ge9SXLK6R9LnJL2q2bAiIqJf9Uwctn9v+xLbfwW8HNgN+EHjkUVERF+qtB6HpNdI+jywCtiJYgqSiIjYBvUcHJd0D3ATcAnwYdu/q9q5pNnAmRTrhp9re1HbcZXH51KsOf5O26t6tZX0PuAUiudJrrD9kaoxRcTmywSJ27auiUPSOODLtj9Vt+Oy7dnA4cAAsELSUtu3t1SbA0wvXwcD5wAHd2sr6XUUc2cdYPtxSXvUjS0iIoava+Kw/UT5RV07cQAHAWtsrwWQdDHFF35r4pgHXGjbwHJJ4yXtCUzt0vY9wCLbj5cxPjiM2KKLzfnfZERs/aqMcfy/8k6qV0uaOfiq0G4ScF/L/kBZVqVOt7YvAl4t6XpJP5B0YKc3lzRf0kpJKzds2FAh3IiIqKLKA4CHlj9bzzoMvL5HO3Uoa1/DfKg63dpuDzyH4rmSA4FLJO1dnrU8VdleQvHEO7NmzRpy7fSIiKinynocrxtm3wPAlJb9ycADFevs0KXtAHBpmShukPRHYAKQ04qIiFFQZT2O50n6kqQry/0Zkk6q0PcKYLqkaZJ2oJiKfWlbnaXA8SocAjxke32PtpdRnu1IehFFkvlVhXgiImIEVBnjOB+4CnhBuf9T4AO9GtneRHHL7FXAHcAltm+TtEDSgrLaMmAtsIZi3Y/3dmtbtjkP2LtcWOpi4IT2y1QREdGcKmMcE2xfIulUKL7UJT1RpXPbyyiSQ2vZ4pZtU0xlUqltWb4ReEeV94+IiJFX5Yzjd5Key1PrcRwCPNRoVBER0beqnHH8HcX4wj6SfgRMBI5uNKqIiOhbVe6qWiXpNRSz5Aq4y/YfGo8sIiL6UpW7qo4Bdi4Hp98MfKPiA4AREbEVqjLG8d9tP1KuwfFG4AKKOaUiImIbVCVxDN5BdQRwju3LKZ6diIiIbVCVxHG/pC9QrMGxTNKOFdtFRMRWqEoCeCvFg3izbf8W2B34cJNBRURE/6qydOxjwDpgTrmA0p62r246sIiI6E9V7qr6BMWA+HMpJhP8sqSPNx1YRET0pyoPAB4HvNz2fwJIWkSx9vg/NRlYRET0pypjHOuAnVr2dwTubiSaiIjoe0OecUj6F4r5qR4HbpN0Tbl/OPDD0QkvIiL6TbdLVSvLnzcC324p/35j0URERN8bMnHYvgBA0k7ACynONu4eHOuIiIht05BjHJK2l/RpiqVaLwC+Ctwn6dOSnjVaAUZERH/pNjh+OsXDftNsv8L2y4F9gPHAZ0YhtoiI6EPdEseRwLttPzJYYPth4D3A3KYDi4iI/tQtcbjTWt62n6BcDTAiIrY93RLH7ZKOby+U9A7gziqdS5ot6S5JayQt7HBcks4qj69uXeejQtsPSbKkCVViiYiIkdHtdtyTgUslvYvillwDBwI7A2/p1bGkccDZFM99DAArJC21fXtLtTnA9PJ1MMU6Hwf3aitpSnns3hqfNSIiRkC323Hvp/gSfz3wEoplY6+0fW3Fvg8C1theCyDpYmAe0Jo45gEXlpfElksaL2lPYGqPtmcAHwEurxhLRESMkCprjn8P+N4w+p4E3NeyP0BxVtGrzqRubSUdBdxv+2ZJQ765pPnAfIC99tprGOFHREQnVSY5HK5O3+rtg+pD1elYLmkX4DTgL3u9ue0lwBKAWbNmbXOD+VMXXjHWIUR0tDm/m+sWHTGCkcRwNbmS3wAwpWV/MvBAxTpDle8DTANulrSuLF8l6fkjGnlERAypycSxApguaZqkHYBjgaVtdZYCx5d3Vx0CPGR7/VBtbd9iew/bU21PpUgwM23/osHPERERLRq7VGV7k6RTKJadHQecZ/s2SQvK44uBZRQPE64BHgNO7Na2qVgjIqK6Jsc4sL2MIjm0li1u2TbFbb+V2naoM3Xzo4yIiDqavFQVERFboSSOiIioJYkjIiJqSeKIiIhakjgiIqKWJI6IiKgliSMiImpp9DmO2DyZbyoi+lHOOCIiopYkjoiIqCWJIyIiakniiIiIWpI4IiKiliSOiIioJYkjIiJqSeKIiIhakjgiIqKWJI6IiKgliSMiImppNHFImi3pLklrJC3scFySziqPr5Y0s1dbSadLurOs/21J45v8DBER8XSNJQ5J44CzgTnADOA4STPaqs0Bppev+cA5FdpeA+xv+wDgp8CpTX2GiIh4pibPOA4C1thea3sjcDEwr63OPOBCF5YD4yXt2a2t7attbyrbLwcmN/gZIiKiTZOJYxJwX8v+QFlWpU6VtgDvAq7s9OaS5ktaKWnlhg0baoYeERFDaXI9DnUoc8U6PdtKOg3YBHyt05vbXgIsAZg1a1b7+0bENmhz1rhZt+iIEYxky9Zk4hgAprTsTwYeqFhnh25tJZ0AHAkcZjtJISJiFDV5qWoFMF3SNEk7AMcCS9vqLAWOL++uOgR4yPb6bm0lzQY+Chxl+7EG44+IiA4aO+OwvUnSKcBVwDjgPNu3SVpQHl8MLAPmAmuAx4ATu7Utu/4csCNwjSSA5bYXNPU5IiLi6Rpdc9z2Mork0Fq2uGXbwMlV25blLxzhMCMiooZGE0dExEjanMHtGDmZciQiImpJ4oiIiFqSOCIiopYkjoiIqCWD4w3KQF5EbI1yxhEREbUkcURERC1JHBERUUsSR0RE1JLEERERtSRxRERELbkdNyKigiwC9ZQkjh7yLEZExNPlUlVERNSSxBEREbUkcURERC1JHBERUUsSR0RE1NJo4pA0W9JdktZIWtjhuCSdVR5fLWlmr7aSdpd0jaSflT+f0+RniIiIp2vsdlxJ44CzgcOBAWCFpKW2b2+pNgeYXr4OBs4BDu7RdiFwre1FZUJZCHy0qc8REbG5xvK2/iaeIWnyjOMgYI3ttbY3AhcD89rqzAMudGE5MF7Snj3azgMuKLcvAN7c4GeIiIg2TT4AOAm4r2V/gOKsoledST3aPs/2egDb6yXt0enNJc0H5pe7j0q6azgfoosJwK9GuM/N1Y8xQX/GlZiqSUzV9WNcE/S/NiumP+tU2GTiUIcyV6xTpW1XtpcAS+q0qUPSStuzmup/OPoxJujPuBJTNYmpun6Mq6mYmrxUNQBMadmfDDxQsU63tr8sL2dR/nxwBGOOiIgemkwcK4DpkqZJ2gE4FljaVmcpcHx5d9UhwEPlZahubZcCJ5TbJwCXN/gZIiKiTWOXqmxvknQKcBUwDjjP9m2SFpTHFwPLgLnAGuAx4MRubcuuFwGXSDoJuBc4pqnP0ENjl8E2Qz/GBP0ZV2KqJjFV149xNRKT7FpDBxERsY3Lk+MREVFLEkdERNSSxDFMkj4kyZImtJSdWk6RcpekN45iLKdLurOctuXbksaPdUzle3edcmaUYpgi6d8l3SHpNknvL8vHfOoaSeMk/UTSd/oopvGSvlX+Pt0h6c/HOi5JHyz/7m6VdJGknUY7JknnSXpQ0q0tZUPGMBr/7oaIaVS+C5I4hkHSFIrpUO5tKZtBcffXS4DZwOfLqVNGwzXA/rYPAH4KnDrWMbVMGzMHmAEcV8Yz2jYBf2/7xcAhwMllHINT10wHri33R9v7gTta9vshpjOBf7O9H/DSMr4xi0vSJOBvgVm296e4WebYMYjpfIp/Q606xjCK/+46xTQq3wVJHMNzBvARnv5Q4jzgYtuP276H4k6xg0YjGNtX295U7i6neO5lTGOi2pQzjbO93vaqcvsRii/CSYzx1DWSJgNHAOe2FI91TLsBfwF8CcD2Rtu/Heu4KO7+3FnS9sAuFM90jWpMtq8DftNWPFQMo/LvrlNMo/VdkMRRk6SjgPtt39x2aKjpU0bbu4Ary+2xjKlf/jyeJGkq8HLgetqmrgE6Tl3ToH+m+M/HH1vKxjqmvYENwJfLS2jnSnr2WMZl+37gMxRn9+spnvW6eixjajFUDP3yu9/Yd0GTU45ssSR9F3h+h0OnAR8D/rJTsw5lI3avc7eYbF9e1jmN4tLM10Yjph7G8r2fQdKfAP8KfMD2w1Kn8EYtliOBB23fKOm1YxbIM20PzATeZ/t6SWcyNpfLnlSOG8wDpgG/Bb4p6R1jGVMFY/673/R3QRJHB7bf0Klc0n+h+AW+ufzimQysknQQ1aZYGfGYWmI7ATgSOMxPPZzTaEw9jOV7P42kZ1Ekja/ZvrQs/qWkPcuJMkd76ppXAkdJmgvsBOwm6atjHBMUf2cDtq8v979FkTjGMq43APfY3gAg6VLg0DGOadBQMYzp7/5ofBfkUlUNtm+xvYftqbanUvxlzLT9C4qpUI6VtKOkaRRrjNwwGnFJmk2xJslRth9rOTRmMVFtypnGqcjwXwLusP3ZlkNjNnWN7VNtTy5/h44Fvmf7HWMZUxnXL4D7JO1bFh0G3D7Gcd0LHCJpl/Lv8jCKcap+mHpoqBi2/u8C23kN8wWsAya07J8G3A3cBcwZxTjWUFy/vKl8LR7rmMr3nktxZ8fdFJfUxuLv6FUUp+SrW/585gLPpbgT5mflz93HKL7XAt8pt8c8JuBlwMryz+sy4DljHRfwSeBO4FbgK8COox0TcBHFGMsfKP7DeFK3GEbj390QMY3Kd0GmHImIiFpyqSoiImpJ4oiIiFqSOCIiopYkjoiIqCWJIyIiakniiC2KpO+3z+wp6QOSPt+jzayG47qonJH0g23lbx7u5I6SXlY+JFi1/nhJ761bT9ILJH1rpOrH1i+JI7Y0F1E8NNfq2LJ8TEh6PnCo7QNsn9F2+M0UswMPx8sonjmpajzQM3G017P9gO2jR7B+bOWSOGJL8y3gSEk7wpMTF74A+KGkcyStVLF2wyc7NZb0aMv20ZLOL7cnSvpXSSvK1ys7tN1J0pcl3VJOAvi68tDVwB6SbpL06pb6hwJHAaeXx/YpX/8m6UZJ/1fSfmXdY1SsN3GzpOvKp+0/BbytbPu2tlheIumG8thqSdOBRcA+Zdnpkv5E0rWSVpUxD85O3F5vqso1HSr221p/nKTPlP2vlvS+yn+TseUa7SdT88prc1/AFcC8cnshcHq5vXv5cxzwfeCAcv/7FOs5ADza0s/RwPnl9teBV5Xbe1FMUdL+vn8PfLnc3o9iOoydgKnArUPEej5wdMv+tcD0cvtgiulGAG4BJpXb48uf7wQ+N0S//wK8vdzeAdi5PQ6Kueh2K7cnUDxVrA71ntyv2G9r/fdQzAO2fevfQV5b9yuTHMaWaPBy1eXlz3eV5W+VNJ/iC3NPiktEqyv2+QZghp6aNXc3Sbu6WMNj0KsovlixfaeknwMvAh6u8gYqZug9lGKG18HiHcufPwLOl3QJcGmH5u1+DJymYl2PS23/TM+c8VfA/5T0FxTTt08CnjcC/bZ6A8W0FpsAbLevWRFboSSO2BJdBnxW0kxgZ9uryonbPgQcaPs/yktQO3Vo2zrHTuvx7YA/t/37Lu+7uXOxbwf81vbLnhGUvUDSwRSLO90k6Rl12up/XdL1Zf2rJP0NsLat2tuBicArbP9B0jo6/5nU7beVGMPp8mNsZIwjtji2H6W4/HQeTw2K7wb8DnhI0vMolqzt5JeSXixpO+AtLeVXA6cM7gzxxX0dxZcxkl5EcUnrrh7hPgLsWsb9MHCPpGPKPiTppeX2Pravt/0J4FcUU2A/2badpL2BtbbPopj59IAO9f+UYt2PP5TjMX/WHtMw+211NbBAxep8SNq9x59HbAWSOGJLdRHFmtgXA7hYkfEnwG0UCeVHQ7RbCHwH+B7FzKKD/haYVQ7w3g4s6ND288A4SbcA3wDeafvxHnFeDHy4HEzfhyLxnCTp5jLWwQHr08sB5lspEtTNwL9TXD57xuA48DbgVkk3UYy3XGj718CPykH20ykW8ZklaWX5vneWf1bt9er22+pcirGe1eVn+usefx6xFcjsuBERUUvOOCIiopYkjoiIqCWJIyIiakniiIiIWpI4IiKiliSOiIioJYkjIiJq+f8XyCrp0n5dgQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "\n", + "rng = np.random.default_rng()\n", + "\n", + "def statistic(x, y, axis=0):\n", + " return np.mean(x, axis=axis) - np.mean(y, axis=axis)\n", + "\n", + "def bootstrap_distribution(x, y):\n", + " nx, ny = len(x), len(y)\n", + " N = 1000\n", + " bootstrap_distribution = []\n", + " for i in range(N):\n", + " # random indices to resample from x and y\n", + " ix = rng.integers(0, nx, size=nx)\n", + " iy = rng.integers(0, ny, size=ny)\n", + " xi = x[ix]\n", + " yi = y[iy]\n", + " stat = statistic(xi, yi)\n", + " bootstrap_distribution.append(stat)\n", + " return bootstrap_distribution\n", + "\n", + "boot_dist = bootstrap_distribution(x, y)\n", + "\n", + "plt.hist(boot_dist, density=True, bins=20)\n", + "plt.xlabel(\"Value of test statistic\")\n", + "plt.ylabel(\"Observed Frequency\")\n", + "\n", + "observed_statistic = statistic(x, y)\n", + "standard_error = np.std(boot_dist, ddof=1)\n", + "print(f\"Observed Statistic Value: {observed_statistic}\")\n", + "print(f\"Standard Error: {standard_error}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b7802b49-0a53-4f5f-95b1-a0710ab98a81", + "metadata": {}, + "source": [ + "This is precisely what `bootstrap` does." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "530d48d9-8944-43e3-848b-37831e3630f7", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "26.78533925848876" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from scipy import stats\n", + "# `n_resamples=1000` indicates that the statistic will be calculated for\n", + "# each of 1000 resamples.\n", + "# The meaning of `method='percentile'` will be discussed below\n", + "res = stats.bootstrap((x, y), statistic, n_resamples=1000, method='percentile')\n", + "assert res.standard_error == np.std(res.bootstrap_distribution, ddof=1)\n", + "res.standard_error" + ] + }, + { + "cell_type": "markdown", + "id": "bce2cf1c-4a82-4438-afb7-1832690132f2", + "metadata": {}, + "source": [ + "The two standard errors estimates differ slightly because the bootstrap algorithm is inherently stochastic, but that is OK. The best we can hope for is an approximation, and these two approximations agree with one another quite well.\n", + "\n", + "An even better way of quantifying the uncertainty, especially when the distribution of the statisic is non-normal, is to produce a *confidence interval* on the statistic. Suppose we perform the the experiment repeatedly and produce a \"95% confidence interval\" $l_i$ and $u_i$ from the data in each experiment $i$; this means that the true value of the statistic (the difference in the *population* means) will be between $l_i$ and $u_i$ in 95% of the replications $i$." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "590cc9d0-0b34-4faf-829b-8302b4230553", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "ConfidenceInterval(low=-18.494047619047613, high=82.84563492063491)" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "res.confidence_interval # 95% confidence interval by default" + ] + }, + { + "cell_type": "markdown", + "id": "935b014a-ec43-4c5f-a480-c804edea3e6a", + "metadata": {}, + "source": [ + "By choosing `method='percentile'` above, we indicated that bootstrap should estimate this confidence interval as the central 95% of the bootstrap distribution - that is, the boundaries of our interval will be the 2.5 and 97.5 percentiles of the bootstrap distribution." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "86857327-0c35-4f5b-9529-9b1c65edd133", + "metadata": {}, + "outputs": [], + "source": [ + "ci_percentile = stats.scoreatpercentile(res.bootstrap_distribution, [2.5, 97.5])\n", + "np.testing.assert_allclose(res.confidence_interval, ci_percentile) # confidence interval is the central 95% of the bootstrap distribution " + ] + }, + { + "cell_type": "markdown", + "id": "86f64385-b4a6-4015-8222-3adc85572384", + "metadata": {}, + "source": [ + "Again, this means that if we were to perform the mice experiment repeatedly and each time use `bootstrap` to compute such a confidence interval from the data, we would expect the confidence interval to contain the true value of the difference in mean survival times 95% of the time. Note also that our confidence interval contains 0. This is closely related to our conclusion from the hypothesis tests above: our data is not inconsistent with the null hypothesis that the treatment has no effect." + ] + }, + { + "cell_type": "markdown", + "id": "9e000136-9159-4361-9da8-b2d0f61d30ae", + "metadata": {}, + "source": [ + "### Single-Sample, Scalar-Valued Statistics (and Confidence Intervals)\n", + "This definition of a confidence interval can be difficult to interpret correctly, so we illustrate with a simpler example. Suppose there is an election with only two candidates, `0` and `1`, and all voters will vote for either one or the other (never both, and never for neither). We wish to estimate the percentage of voters who will vote for candidate `1` by performing an experiment before the election: we will ask a random sample of 1000 voters who the will vote for on election day. The results are stored in the array `sample`." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "2a72dd5a-7de6-4d7b-8d6d-e618f7c9f3fa", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "249 for candidate 0, 751 for candidate 1\n" + ] + } + ], + "source": [ + "# Rather than entering `sample` directly, let's generate one to work with.\n", + "# To simulate the results of such an experiment, suppose that the true \n", + "# (but unknown) percentage of voters who will vote for candidate `1` is 75%. \n", + "# If we sample voters at random from the population before the election and \n", + "# ask them who they will vote for, the responses will follow a Bernoulli \n", + "# distribution with shape parameter `p=0.75`.\n", + "p = 0.75\n", + "dist = stats.bernoulli(p=p)\n", + "sample = dist.rvs(size=1000)\n", + "vote_for_0 = np.sum(sample == 0)\n", + "vote_for_1 = np.sum(sample == 1)\n", + "print(f\"{vote_for_0} for candidate 0, {vote_for_1} for candidate 1\")" + ] + }, + { + "cell_type": "markdown", + "id": "c9f9532d-46ee-4d59-ab3b-af452790fab1", + "metadata": {}, + "source": [ + "The statistic we wish to estimate is the percentage of voters who will vote for candidate 1, so we can produce a *point estimate* of the statistic from the sample as:" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "89b46532-a3b2-448f-b47a-22b92529074c", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.751" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def statistic(sample, axis=0):\n", + " return np.sum(sample, axis=axis) / sample.shape[axis]\n", + "statistic(sample)" + ] + }, + { + "cell_type": "markdown", + "id": "b30681e0-52d0-41ac-8e90-11ecee5bf235", + "metadata": {}, + "source": [ + "`bootstrap` can produce a confidence interval around the point estimate." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "ee6d8613-22ca-4a59-bf66-c86ed1bdb726", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "ConfidenceInterval(low=0.726, high=0.772)" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# As with `permutation_test`, the first argument of `bootstrap` needs to be a *sequence* of samples\n", + "data = (sample,)\n", + "# Passing `confidence_level=0.9` produces a 90% confidence interval\n", + "res = stats.bootstrap(data, statistic, confidence_level=0.9)\n", + "res.confidence_interval" + ] + }, + { + "cell_type": "markdown", + "id": "593bb317-2f96-40a3-8f7f-1499015ea2b2", + "metadata": {}, + "source": [ + "Suppose we perform the same experiment $100$ times, each time collecting new data from the same population, but computing the confidence interval in the same way." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "b09ee5a2-1adb-4430-8118-ad16f2ed2549", + "metadata": {}, + "outputs": [], + "source": [ + "# lower and upper limits of confidence intervals produced by `bootstrap`\n", + "n_replications = 100 # 100 replications of the same experiment\n", + "n_observations = 1000 # 1000 observations per sample\n", + "\n", + "# Draw 100 new samples from the same population of voters, each with 1000 observations\n", + "sample = dist.rvs(size=(100, 1000)) \n", + "\n", + "# bootstrap the 90% confidence interval for all 100 samples (10 at a time)\n", + "res = stats.bootstrap((sample,), statistic, confidence_level=0.9, axis=1, batch=10)\n", + "li, ui = res.confidence_interval\n", + " \n", + "# This was equivalent to (but faster than) the following \n", + "# li = np.empty((n_replications,))\n", + "# ui = np.empty((n_replications,))\n", + "# for i in range(n_replications):\n", + "# sample = dist.rvs(size=n_observations) # collect a new sample from the same population of voters\n", + "# res = stats.bootstrap((sample,), statistic, confidence_level=0.9, vectorized=False)\n", + "# li[i], ui[i] = res.confidence_interval" + ] + }, + { + "cell_type": "markdown", + "id": "55cbf673-1d90-4436-9755-12c79cd4c054", + "metadata": {}, + "source": [ + "We expect that the confidence interval will contains the true value of the statistic ($p=0.75$) approximately 90% of the time." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "9856f8c6-4f4e-43e6-98e7-476202f27bcb", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "87\n" + ] + } + ], + "source": [ + "contained = (li < p) & (p < ui)\n", + "print(np.sum(contained))" + ] + }, + { + "cell_type": "markdown", + "id": "c172f0e3-40cf-49ec-841e-0a3a7ab7bf64", + "metadata": {}, + "source": [ + "### Paired-Sample, Vector-Valued Statistics\n", + "\n", + "[An Introduction to the Bootstrap](https://books.google.com/books?id=MWC1DwAAQBAJ&printsec=frontcover) considers a small data set collected when studying a medical device for continuously delivering an anti-inflammatory hormone to test subjects. The arrays `x` and `y` record the number of hours the device was worn and the amount of hormone remaining in the device, respectively." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "07480ef9-594e-4eb6-9661-25cced033299", + "metadata": {}, + "outputs": [], + "source": [ + "x = np.array([99, 152, 293, 155, 196, 53, 184, 171, 52, 376, 385, 402, 29, 76, 296, 151, 177, 209, 119, 188, 115, 88, 58, 49, 150, 107, 125]) # hours worn\n", + "y = np.array([25.8, 20.5, 14.3, 23.2, 20.6, 31.1, 20.9, 20.9, 30.4, 16.3, 11.6, 11.8, 32.5, 32.0, 18.0, 24.1, 26.5, 25.8, 28.8, 22.0, 29.7, 28.9, 32.8, 32.5, 25.4, 31.7, 28.5]) # amount remaining (units unspecified)" + ] + }, + { + "cell_type": "markdown", + "id": "e1e416f8-7f0e-46f8-9584-388fd67602fe", + "metadata": {}, + "source": [ + "A standard linear regression is performed in SciPy as follows." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "f64d81c2-d847-425a-bee0-dc53e8efed05", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjpklEQVR4nO3de3zU1Z3/8deHEAMCxRDAHxUKRi2gCCFEjOKCiFiveK21RcW2SlWqVX9bxVovXW2Lq6J1VbZIFa3UpRVFyuLWleJddBMvCEVW5FJRChhAQVFJcvaPM3GSMJPMJDPz/c7M+/l4zGPmnHy/8/34BT+cnO+5mHMOERHJPh2CDkBERNpGCVxEJEspgYuIZCklcBGRLKUELiKSpTpm8mI9e/Z0AwYMyOQlRUSyXnV19UfOuV7N6zOawAcMGEBVVVUmLykikvXMbH2senWhiIhkKSVwEZEspQQuIpKlMtoHLiLhtHv3bjZs2MDnn38edCh5rVOnTvTt25fCwsKEjlcCFxE2bNhAt27dGDBgAGYWdDh5yTlHTU0NGzZsYP/990/oHHWhiAiff/45JSUlSt4BMjNKSkqS+i1ICRyoXr+Ne5espnr9tqBDEQmMknfwkv0zyPsulOr125g4aylf1tazV8cOzLmwkhH9i4MOS0SkVXnfAl+6poYva+upd7C7tp6la2qCDklEWnDiiSeyffv2Fo+54YYbeOaZZ9r0/c8++ywnn3xyq8cdffTRrU5MvOuuu/jss8/aFEcisiKBt7eLo6XzK0tL2KtjBwoMCjt2oLK0pL3hikgaOOeor69n0aJF7LPPPi0e+y//8i8ce+yxmQmsBXmfwKvXb2PGtfcw5ZiD+M3PZyadxBu6SO54ehUTZy3d4/wR/YuZc2ElVx03UN0nIklI9bOj6dOnM2TIEIYMGcJdd90FwLp16xg8eDCXXnop5eXlvP/++wwYMICPPvoIgJtvvplBgwYxfvx4vvvd73L77bcDcMEFF/DYY48BfgmPG2+8kfLycg499FDeeecdAF577TWOPPJIhg8fzpFHHsmqVatajG/Xrl2cc845DB06lO985zvs2rXrq59dcsklVFRUcMghh3DjjTcCcPfdd/Phhx8yduxYxo4dG/e49gh9H/jSNTUUfPEFAA8/MpUdT98N76+FvfZK+PzmXSTNk/SI/sVK3CJJSPWzo+rqah588EFeffVVnHMcfvjhjBkzhuLiYlatWsWDDz7Ifffd1+Scqqoq5s2bxxtvvEFtbS3l5eWMGDEi5vf37NmT119/nfvuu4/bb7+dWbNmMWjQIJ5//nk6duzIM888w89+9jPmzZsXN8YZM2aw9957s2zZMpYtW0Z5eflXP/vlL39Jjx49qKurY9y4cSxbtozLL7+c6dOns2TJEnr27Bn3uKFDh7b5voW+BV5ZWsJzB4/i7HP/FYBumz+EoiKYMyfh89VFIpJaqX529OKLL3L66afTpUsXunbtyhlnnMELL7wAQP/+/amsrIx5zqmnnkrnzp3p1q0bp5xyStzvP+OMMwAYMWIE69atA+Djjz/m29/+NkOGDOHKK69kxYoVLcb4/PPPc+655wIwdOjQJon3j3/8I+Xl5QwfPpwVK1bwt7/9LeZ3JHpcokKfwBu6OMb84HSq19bAqaf6H5x7LphBKw8z1EUiknqpbhi1tLl6ly5dkj6nuaKiIgAKCgqora0F4Prrr2fs2LEsX76cP//5zwmNv441zG/t2rXcfvvtLF68mGXLlnHSSSfF/K5Ej0tG6BM4+CQ8ZeyBjBjQA+bPh8Z9VcXFcPPNiZ2fhuStMeSSj1LdMBo9ejTz58/ns88+49NPP+WJJ57gn/7pn1o856ijjvoq8e7cuZP//M//TOqaH3/8Mfvttx8As2fPTijGOZHf/JcvX86yZcsA+OSTT+jSpQvdu3dn06ZNPPXUU1+d061bN3bs2NHqcW0V+j7wmL75TXAOrr4abrsNbrjBv9avh298I2NhaAy55LNUPjsqLy/nggsuYOTIkQBceOGFDB8+/KvujlgOO+wwJkyYwLBhw+jfvz8VFRV079494WteffXVTJo0ienTp3PMMce0evwll1zC97//fYYOHUpZWdlXsQ4bNozhw4dzyCGHUFpayqhRo746Z/LkyZxwwgn06dOHJUuWxD2urSyZX0Paq6KiwqV8Q4ctW6B372j5/PPhoYdSe4047l2ymjueXkW9gwKDq44byJSxB6b8OtXrt7F0TQ2VpSX6B0LSYuXKlQwePDjoMJK2c+dOunbtymeffcbo0aOZOXNmk4eL2SjWn4WZVTvnKpofmxVdKC3q1cu3xmfM8OWHH/Z949XVab90Jh6QtjYMUiSfTZ48mbKyMsrLyznzzDOzPnknKzu7UGK5+GKYNAn69IGPP4aKChg5El55BTqk59+phn7AdLaOExkGKZKv/vCHPwQdQqCyvwXeWOfOflRKw8OM116DgoJoOQ3S+YAUNAxSROLLnRZ4YyeeCHV1MGoULF0KDesa1NRAjx7BxpakTLTyRSQ75WYCB99t8sor8Prr0DA7q6QEysrgjTcCDS1ZmikqIrHkVhdKLOXl/iFngzffBDMW/ir5dVVERMIk9xN4A+fgxRe/Kp583Y8YMaCHkrhISNx9990MHjyYiRMnsmDBAqZNmwbA/Pnzm0w5nz17Nh9++GFS371u3TqGDBnSYn1VVRWXX355O/4LMi9/EjjAqFHc+9d3m1SNGNDDj2ARkUDdd999LFq0iDlz5jBhwgSmTp0KpCaBJ6KiooK777475d+bTvmVwPGjOgb9fBFnT7w1Wvnb3/qx4/X1wQUmkscuvvhi1qxZw4QJE7jzzjuZPXs2P/7xj3n55ZdZsGABP/3pTykrK+PWW2+lqqqKiRMnUlZWxq5du6iurmbMmDGMGDGCb33rW2zcuBHwKxwOGzaMI444gnvvvbfVGBpv5HDTTTfxgx/8gKOPPprS0tImif2RRx5h5MiRlJWV8aMf/Yi6urr03JQE5O5DzDiiozoOovqXF/kWeIOCAr+2ytatwQUoErQrrvDPilKprAwia3zH8u///u/813/911dLrzasTXLkkUcyYcIETj75ZM466ywAnnrqKW6//XYqKirYvXs3l112GU8++SS9evVi7ty5XHfddTzwwAN8//vf59/+7d8YM2YMP/3pT5MO+Z133mHJkiXs2LGDgQMHcskll7B69Wrmzp3LSy+9RGFhIZdeeilz5szh/PPPb8NNab+8S+DQbFSHc7B5M+y7ry9v2+Zb41k45FAk36xatYrly5czfvx4AOrq6ujTpw8ff/wx27dvZ8yYMQCcd955SS8eddJJJ1FUVERRURG9e/dm06ZNLF68mOrqag477DDAb/LQu/FSHhmWdwk85roivXv7RN6xox8/Dn7IITQdwSKSD1poKYeNc45DDjmEV155pUn99u3bk97hvbmGJWghugytc45Jkybx61//ul3fnSp51Qfe6roitbX+1ZiZn9EpIoFovCRr8/LAgQPZsmXLVwl89+7drFixgn322Yfu3bvzYmTk2ZwEN4Bpzbhx43jsscfYvHkzAFu3bmX9+vUp+e62yKsEntAuIgUFvtV93nnRusMP94lcRDLunHPO4bbbbmP48OG89957XHDBBVx88cWUlZVRV1fHY489xjXXXMOwYcMoKyvj5ZdfBuDBBx9kypQpHHHEEXTu3DklsRx88MHccsstHHfccQwdOpTx48d/9dA0CK0uJ2tmnYDngSJ8l8tjzrkbzawHMBcYAKwDznbOtTioOi3LySahoQW+u7aewkTX726euO++Gy67LH1BigQgW5eTzUWpXk72C+AY59wwoAw43swqganAYufcQcDiSDnU2rSLiHPw+99Hy5dfrta4iIRCqwnceTsjxcLIywGnAg07JzwEnJaOAFOtTasHnnvung8zzeD441MbnIhIEhLqAzezAjN7E9gM/Ldz7lVgX+fcRoDIe8yxNGY22cyqzKxqy5YtKQo7IM5B452r//IXn8i//DKQcLQfp6RSJnfnktiS/TNIaBihc64OKDOzfYAnzGzPRQXinzsTmAm+Dzyp6MLo4IN9Im/cjdIw3CiD/wNoP05JpU6dOlFTU0NJSUm7h99J2zjnqKmpoVOnTgmfk9Q4cOfcdjN7Fjge2GRmfZxzG82sD751nj+cg507oVu3aJ1ZxjZW1k49kkp9+/Zlw4YNZP1vyVmuU6dO9O3bN+HjW03gZtYL2B1J3p2BY4FbgQXAJGBa5P3JNkWczbp29Yl8yJBo10r//v49za3xhp16GkbUaKceaY/CwkL233//oMOQJCXSAu8DPGRmBfg+8z865xaa2SvAH83sh8DfgW+nMc5wW77cJ+zGe2+awcKFcNJJabmkduoRkVbHgadS0OPAM+L66+GWW5rW6eGQiLRDe8aBSzJuvjn2kMOrrw4mHhHJWUrg6eKcH2bY4LbbfCJXa1xEUkQJPJ2OO27PhN2hAxx4YDDxiEhOUQJPs+r12xj080UcMWV2tPK993xr/JNPUn4tTewRyR9K4GnWMF57Y9eeHDB1YdMfdu+esnVVWl0qV0RyjhJ4mjWM1y4wKOzYgep1W/ecem8Gy5a16zrxlspVq1wkd+XdjjyZFne8tnNw2mnwZGT+07Bh0fo2iDWxR9PtRXKbEngGNNmDs7H58/17424UM/jNb/yytUleo/k/FPcuWa3p9iI5TF0oYeCcH2bY4Cc/aVPfePOlcpt332i6vUhu0UzMsGmeuMeMgWefbfPXxdzEWUSySryZmOpCCRvn4NVXobLSl597zif13buhY/J/XHG7b0Qk66kLJYwOP3zPh5mFhdrKTUSaUAIPM+dg69amdWawdm0w8YhIqCiBh11x8Z6t8dJStcZFRAk8azgH9fVN68zg0UeDiUdEAqcEnk0aVjMcNSpa973vqTUukqeUwLPRiy/GXnP8rLOCiUdEAqEEns2cg3vuiZbnzVNrXCSPKIFnuylTYrfGlchFcp4SeK5wDl57rWmdGWzfHkg4IpJ+SuC55LDD9myNFxerNS6So5TAc5FzsHNn0zozePnlYOIRkbRQAs9VXbrs2RofNUqtcZEcogSe65yL/ZDz+uuDiUdEUkYJPF8413Sc+C23hKo1rq3fRJKn5WTzyZ/+5N+b7wDUuzds2hRMTKCt30TaSC3wfOQczJ0bLW/e7BN5XV0g4cTbkFlEWqYEnq/OPnvPvvGOHQPpVtHWbyJtoy6UfOccvP8+fOMb0Toz+PvfoV+/jIQQa0NmEWmdErj4RO1c09Z3Q0LP0J6p2vpNJHnqQgmJUIzCcA5qa5vWmUUffuaAUNxnkRRRCzwEQjUKo6DAJ/JeveCjj3zd2Wf79wy1xtMlVPdZJAXUAg+BUI7C2LIl9gSgb387mHhSIJT3WaQdlMBDINSjMJyD666Llh97LFQTgJIR6vss0gbmMvhrcUVFhauqqsrY9bJJ9fpt4R+FEStxZ1m3SlbcZ5FmzKzaOVfRvF594CGRDaMwqtdt5Zc3P8Ljv7s8WmkGn34Ke+8dXGBJyIb7LJKoVrtQzKyfmS0xs5VmtsLMfhKpv8nMPjCzNyOvE9MfrgRp6Zoa3uxVyoBrFjb9QZcuWdutIpLNEukDrwX+v3NuMFAJTDGzgyM/u9M5VxZ5LUpblBIKjfuQB/18EW8sW9f0ADNYujSQ2ETyUatdKM65jcDGyOcdZrYS2C/dgUn4NJ8xObx/8Z4TgI44wr9nWd+4SDZKahSKmQ0AhgOvRqp+bGbLzOwBM4vZsWhmk82sysyqtmzZ0r5oJXAj+hczZeyBTfuR4605PnVqZoMTyTMJJ3Az6wrMA65wzn0CzAAOAMrwLfQ7Yp3nnJvpnKtwzlX06tWr/RFLeDkHBxwQLd96q/rGRdIooQRuZoX45D3HOfc4gHNuk3OuzjlXD9wPjExfmJI1Vq+O3RovLAwmHpEclsgoFAN+B6x0zk1vVN+n0WGnA8tTH55kLefgjka/lNXW+kSuvnGRlGl1Io+ZHQW8ALwN1EeqfwZ8F9994oB1wI8iDzzj0kSePJUDE4BEgtTmiTzOuReBWB2ZGjYoiXEOli+HQw+N1pnBBx/A178eXFwiWU5roUjS2rQk65Ahe7a699tPDzlF2kFT6SUp7V6S1Tn48ksoKorWmfk9OhuWrRWRhKgFLklJyZKse+21Z2v8O99Ra1wkSUrgkpSULskabwLQKae0L0iRPKEuFElKWjYgdg7OOAOeeMKXFy7UkEORBCiBS9LSsiTr44/798bdKA2flchFYlIXioSLc9Fk3sDMP/gUkSaUwCV8Tj99z1Z3UZEecoo0owQu4eWc31y5MTNYuzaYeERCRglc2qVNk3qS0bPnnq3x0lK1xkXQQ0xph3ZP6klGw5DDDo3aHJoAJHlOLXBps5RM6klGw9DCM86I1mkCkOQxJXBps5RO6knGvHmxJwDdcENmri8SEq0uJ5tKWk4291Sv35baST3JevhhmDSpaV19vVrlklPiLSerBC65oXnCPvRQWLYsmFhEUixeAlcXiuQG52Ddumj57bd9Uv/008BCEkk3JXDJHf3779k33rWrulMkZymBS+5xDr74ommdGaxcGUw8ImmiBC65qWHN8fHjo3UHH6zWuOQUJXDJbU8/HXvI4TPPBBOPSAopgUvWaNe0fefgN7+JlsePV2tcsp4SuGSFhmn7dzy9iomzlrYtiV9+eezW+K9/nZogRTJMCVyyQkqn7TsHf/tbtPyzn/lEXl/f/kBFMkgJXLJCyqftDx68Z2u8oACOPbZ93yuSQVqNULJCWvbiBJ/EP/kEunf35cWLfWt82zbYZ5/UXEMkTZTAJWukZS9OgK99zSfyU07xGyoDFEeuo/04JcTUhSLS4M9/3rMf3MxPyxcJISVwkcYa1hy/445o3dChGnIooaQELhLLVVfFHnL46KPBxCMSgxK4SEucg2efjZa/9z21xiU0lMAl45KZUZn2TZMTMWZM7Nb4ZZcFE49IhDZ0kIxKZiPkjG6anKgNG6Bfv6Z1X3zhF88SSRNt6CChkMyMykSPzWgrvW9f3xrv3z9aV1QEBx6Y/muLNKNx4NImbd0Ls2FG5e7a+lZnVCZybGCt9HXrfMu7Uydffu89363ywQfw9a+n//oiKIFLG7QnaSYzozKRY2O10jPWzVJU5Fvjl14KM2b4uv328++aACQZoC4USVp7F5Ya0b+YKWMPTCjRtnZsytdIaYv77ov9kPOFFzIfi+QVtcAlacl0g6Rb2tZIaQvnYM4cOPdcXx49OlovkgatjkIxs37Aw8D/A+qBmc6535hZD2AuMABYB5ztnGvxKZJGoeSOtvaB543mY8Xvugt+8pNAQpHsF28USiIJvA/Qxzn3upl1A6qB04ALgK3OuWlmNhUods5d09J3KYFLXnnrLSgra1pXX6+JQJK0Ng8jdM5tdM69Hvm8A1gJ7AecCjwUOewhfFIXkQbDhu3ZfdKhA5x+ejDxSM5J6iGmmQ0AhgOvAvs65zaCT/JA7zjnTDazKjOr2rJlSzvDFclCzsHWrdHy/Pm+Fb5jR2AhSW5IOIGbWVdgHnCFc+6TRM9zzs10zlU45yp69erVlhhFsl9xsU/kDQ82wa9D3kEDwaTtEvrbY2aF+OQ9xzn3eKR6U6R/vKGffHN6QhTJIc89B3V10bJzvjW+alVwMUnWajWBm5kBvwNWOuemN/rRAmBS5PMk4MnUhyeSgzp08In75pujdYMG6eGmJC2RFvgo4DzgGDN7M/I6EZgGjDezd4HxkbKIJOrnP489AWj+/EDCkezT6kQe59yLQLymwbjUhiOSh5yDv/wFjj/elxtGqWgCkLRCT1BEwuBb34rdGr/22mDikaygBC4SJs7BmjXR8rRpPpHX1gYXk4SWErhI2Oy/v0/kxY2WKCgshPLy4GKSUFICFwmrrVvhs8+i5Tfe8K3xzRqxK54SuEiYde7sW+PnnRet23dfDTkUQAlcJDs8/LBfCKsxM3jttWDikVBQAhfJFma+NT5zZrTu8MPVGs9jSuAi2eaii2IPObz//mDikcAogYtkK+eadqFMnhxtpUteUAIXyWaHHRZ7zfHGDz0lZymBi+QC55oOL3zkEd8abzwMsRXV67dx75LVVK9vcWdECRFtaiySK3r18ol8xAh4/XVf16UL7LMPbGs5KVev38bEWUv5sraevTp2YM6FldrrNAuoBS6Sa6qrYffuaHn7dt8aX7s27ilL19TwZW099Q5219azdE1N+uOUdlMCF8lFHTv61vjUqdG60tK4Qw4rS0vYq2MHCgwKO3agsrQkQ4FKe7S6K30qaVd6kYA0T9xPPRVdvjaiev02lq6pobK0RN0nIRNvV3r1gYvkA+fgySfhtNN8+YQTovURI/oXK3FnGXWhiOSLU0+NPQGo8dZuklWUwEXyjXNNN1G+4QafyJuvtSKhpwQuko+++U2fyBv3jRcUqDWeZZTARfJZfT3s2hUt33ADfOMb8OWXwcUkCVMCF8l3nTr51vjs2b78/vtQVORnc0qoKYGLiDdpkm+Rn366L593nu9i2b490LAkPiVwEYkyg8cfh//932hdcbH6xkNKCVxE9nTQQb5b5ZprfLlhpMr69cHGJU0ogYtIfNOmwZYt0fKAAXD++YGFI00pgYtIy3r29K3x3/7Wl3//e98ar64ONq40yLYldZXARSQxkyf79cWLI9PtKypg5Eioqws2rhRpWFL3jqdXMXHW0qxI4krgIpK4zp1h61ZYtMiX/+d//MqHCxcGG1cKZOOSukrgIpK8E07wLe8jj/TlU06Bbt2S2gEobLJxSV0lcBFpmw4d4KWXorv/7NzpdwCaMSPYuNpoRP9i5lxYyVXHDcyaHYm0HriIpMYPfgAPPhgtb9oEvXsHF08OibceuFrgIpIaDzwAf/97tLzvvvDP/xxcPHlACVxEUqdfPz/k8Fe/8uU77vBDDhvP7JSUUQIXkdS79tqma6gMHOgfdGawyzYfKIGLSHp07+4T9h/+4MsLF/oHny+8EGxcOUQJXETS67vf9euLl5b68ujRfq2V3buDjSsHtJrAzewBM9tsZssb1d1kZh+Y2ZuR14npDVNEslphIbz3Hjz3nC+vXg177QVz5wYbV5ZLpAU+Gzg+Rv2dzrmyyGtRasMSkZw0erRfc/ykk3z5nHP8Q85PPgk2rizVagJ3zj0PbM1ALCKSD8x8f/jKldG67t3h1luDiylLtacP/MdmtizSxRJ3ypKZTTazKjOr2tJ4WUoRyW+DBvmHnFde6ctTp/rkvmFDsHFlkbYm8BnAAUAZsBG4I96BzrmZzrkK51xFr1692ng5EclZ06fDP/4RLffrBxddFFw8WaRNCdw5t8k5V+ecqwfuB0amNiwRySv77utb4/fc48uzZvnW+FtvBRtXyLUpgZtZn0bF04Hl8Y4VEUnYlCnw6ad+USyAsjI46ij/4FP2kMgwwkeBV4CBZrbBzH4I/KuZvW1my4CxwJVpjlNE8sXee/uVDRcs8OWXXoKCAvjLX4KNK4S0GqFInqtev42la2qoLC0J3xKqdXVw+OHR7dtKSvxDzk6dgo0rw7QaoYjsIfTbiBUUQFWV3/kHoKbG7wo0a1awcYWEErhIHsuabcQqKvxDzokTffmii/xDzpqQxpshSuAieSzrthF75BFYuzZa7tkTrrsuuHgCpj5wkTwX6j7wlvziF3DTTdHy6tVwwAGBhZNO8frAlcBFJHtt2wY9ekTLZ54Jf/qT717JIXqIKSK5p7jY940/9JAvz5vn1xxfujTYuDJECVxEst/558Pnn8N++/nyEUfAkCFQWxtsXGmmBC4iuaGoyI8RX7zYl1es8OuQP/54sHGlkRK4iOSWY47xU+/HjfPlM8+Ejh397M4cowQuIrnHDJ55BpZHlmmqq4Nu3eDOO4ONK8WUwEUkdx1yiH/IOWWKL191lU/uGzcGG1eKKIGLSO675x748MNo+etfjyb1LKYELiL5oU8f3xqfPt2X77vPt8ZXrAg2rnZQAheR/HLllbBjh3+wCX644bhxPrlnGSVwEck/XbvC7t1+4g/AX//qJwD99a/BxpUkJXARyV9nnOET+ZAhvjxuHPTtC198EWxcCVICF5H81rEjvP02vPKKL3/wgd8womF6fogpgYuIAFRW+glAZ53lyxdc4B9ybgvZJheNKIGLiDQw86sZrl4drevRwy9dG0JK4CIizR1wgB+Vcu21vnzTTT65N95MIgSUwEVE4vnVr+Cjj6Ll0lL43vdCM+RQCVxEpCUlJT5h33+/Lz/6qB9y2LDRcgKq12/j3iWrU75ptBK4iEgiLrwQdu3y+3ACjBwJ5eV+oawWVK/fxsRZS7nj6VVMnLU0pUlcCVxEJEHVm3Zx7x9f4d2H/uQr3njDD0NcsCDuOUvX1PBlbT31DnbX1rN0TU3K4lECFxFJQOOW9CnvdqF6zUdw1FH+h6eeCnvvDZ9+usd5laUl7NWxAwUGhR07UFlakrKYlMBFRBKwR0t63TZ44QV4801/wK5dfor+Pfc0OW9E/2LmXFjJVccNZM6FlYzoX5yymJTARUQSELclPWyYf8j5wx/68mWX+SGHmzZ9de6I/sVMGXtgSpM3gLkMDoepqKhwVVVVGbueiEgqVa/fxtI1NVSWlsROxhs2QL9+0fIVV6RkFyAzq3bOVTSvVwtcRCRBrbak+/b1rfFp03z5rrt8a/ydd9ISjxK4iEiqXXMNfPxxtDx4MPzjHym/jBK4iEg6fO1rvjU+dy6ceCJ06ZLyS3RM+TeKiEjU2Wf7VxqoBS4ikqWUwEVEspQSuIhIllICFxHJUkrgIiJZqtUEbmYPmNlmM1veqK6Hmf23mb0beU/t/FAREWlVIi3w2cDxzeqmAoudcwcBiyNlERHJoFYTuHPueWBrs+pTgYcinx8CTkttWCIi0pq2TuTZ1zm3EcA5t9HMesc70MwmA5MjxZ1mtqqN10yFnsBHrR4VLMWYGoqx/cIeH+RPjP1jVaZ9JqZzbiYwM93XSYSZVcVa0StMFGNqKMb2C3t8oBjbOgplk5n1AYi8b05dSCIikoi2JvAFwKTI50nAk6kJR0REEpXIMMJHgVeAgWa2wcx+CEwDxpvZu8D4SDkbhKIrpxWKMTUUY/uFPT7I8xgzuiOPiIikjmZiiohkKSVwEZEsldMJ3MzWmdnbZvammVVF6gJdBiDZpQnM7FozW21mq8zsWwHFd5OZfRC5j2+a2YlBxRe5Zj8zW2JmK81shZn9JFIfpvsYL8ZQ3Esz62Rmr5nZW5H4fhGpD9M9jBdjKO5hs1gLzOwNM1sYKWfmPjrncvYFrAN6Nqv7V2Bq5PNU4NYMxzQaKAeWtxYTcDDwFlAE7A+8BxQEEN9NwD/HODbj8UWu2wcoj3zuBvxvJJYw3cd4MYbiXgIGdI18LgReBSpDdg/jxRiKe9js2lcBfwAWRsoZuY853QKPI9BlAFxySxOcCvyHc+4L59xaYDUwMoD44sl4fOBn/zrnXo983gGsBPYjXPcxXozxZDRG5+2MFAsjL0e47mG8GOMJ5O+jmfUFTgJmNYsl7fcx1xO4A542s+rIlH5otgwAEHcZgAyKF9N+wPuNjttAy0kgnX5sZssiXSwNvw4GHp+ZDQCG41tnobyPzWKEkNzLyK/9b+In4v23cy509zBOjBCSexhxF3A1UN+oLiP3MdcT+CjnXDlwAjDFzEYHHVCSLEZdEOM+ZwAHAGXARuCOSH2g8ZlZV2AecIVz7pOWDo1Rl5E4Y8QYmnvpnKtzzpUBfYGRZjakhcMDuYdxYgzNPTSzk4HNzrnqRE+JUdfmGHM6gTvnPoy8bwaewP+qEsZlAOLFtAHo1+i4vsCHGY4N59ymyP9I9cD9RH/lCyw+MyvEJ8Y5zrnHI9Whuo+xYgzjvXTObQeexS8bHap7GCvGkN3DUcAEM1sH/AdwjJk9QobuY84mcDPrYmbdGj4DxwHLCecyAPFiWgCcY2ZFZrY/cBDwWqaDa/iLGHE6/j4GFp+ZGfA7YKVzbnqjH4XmPsaLMSz30sx6mdk+kc+dgWOBdwjXPYwZY1juIYBz7lrnXF/n3ADgHOCvzrlzydR9zMQT2iBeQCn+ae9bwArgukh9CX4Tincj7z0yHNej+F/7duP/Nf5hSzEB1+GfVK8CTggovt8DbwPLIn8B+wQVX+SaR+F/7VwGvBl5nRiy+xgvxlDcS2Ao8EYkjuXADZH6MN3DeDGG4h7GiPdooqNQMnIfNZVeRCRL5WwXiohIrlMCFxHJUkrgIiJZSglcRCRLKYGLiGQpJXARkSylBC4ikqX+D/bCIRuaBXJmAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The slope estimate is: -0.0574462986976377\n", + "The intercept estimate is: 34.16752817399911\n", + "The slope standard error is: 0.004464173160311544\n", + "The intercept standard error is: 0.8671972620941928\n" + ] + } + ], + "source": [ + "res_lr = stats.linregress(x, y)\n", + "\n", + "plt.plot(x, y, '.', label='original data')\n", + "plt.plot(x, res_lr.intercept + res_lr.slope*x, 'r', label='fitted line')\n", + "plt.legend()\n", + "plt.show()\n", + "print(f\"The slope estimate is: {res_lr.slope}\")\n", + "print(f\"The intercept estimate is: {res_lr.intercept}\")\n", + "print(f\"The slope standard error is: {res_lr.stderr}\")\n", + "print(f\"The intercept standard error is: {res_lr.intercept_stderr}\")" + ] + }, + { + "cell_type": "markdown", + "id": "3aa96aa6-be6a-4e9c-a55c-f5e4e14fb6c4", + "metadata": {}, + "source": [ + "`linregress` produces point estimates of the slope and intercept as well as standard errors for each statistic, assuming that the residuals between the best fit line and the data are normally distributed. We can test the normality assumption using `stats.shapiro`." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "b5eb0671-a179-47e7-809c-8bc39f56843d", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "ShapiroResult(statistic=0.9171469211578369, pvalue=0.03371993452310562)" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "e = y - res_lr.intercept + res_lr.slope*x\n", + "stats.shapiro(e)" + ] + }, + { + "cell_type": "markdown", + "id": "77de4703-5400-4a4b-a928-0af8878c330a", + "metadata": {}, + "source": [ + "Although the $p$-value is does not conclusively reject the null hypothesis at all reasonable confidence levels, it does suggest that we might want to relax the residual normality assumption. `bootstrap` makes no such assumption about the residuals, and it can go beyond the standard errors, producing bias-corrected confidence intervals. The standard errors produced by `bootstrap` matche those produced by `linregress` fairly well; however `linregress` may overestimate these quantities for this data." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "b393954f-53a8-4e69-baf2-6f75f79020f3", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The slope standard error is: 0.004260082018022721\n", + "The intercept standard error is: 0.7292901429594613\n", + "The confidence interval on the slope is: (-0.06516915578638195, -0.04868503288913481)\n", + "The confidence interval on the intercept is: (32.58844595440181, 35.45478220570269)\n" + ] + } + ], + "source": [ + "def statistic(x, y):\n", + " res = stats.linregress(x, y)\n", + " return res.slope, res.intercept\n", + "\n", + "res = stats.bootstrap((x, y), statistic, vectorized=False, paired=True)\n", + "\n", + "print(f\"The slope standard error is: {res.standard_error[0]}\")\n", + "print(f\"The intercept standard error is: {res.standard_error[1]}\")\n", + "print(f\"The confidence interval on the slope is: {res.confidence_interval.low[0], res.confidence_interval.high[0]}\")\n", + "print(f\"The confidence interval on the intercept is: {res.confidence_interval.low[1], res.confidence_interval.high[1]}\")" + ] + }, + { + "cell_type": "markdown", + "id": "1dceaaf7-ba8e-44d7-afb7-1cc1e74ed491", + "metadata": {}, + "source": [ + "Again, because the statistic has multiple values, a visualization of the bootstrap distribution may be more informative." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "662eee2a-0c36-4f9e-bbaa-8896f328aa8c", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[]" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABP8klEQVR4nO29d3ij13Xn/72oBEiABMHOGU7jSKPqUTVlxbFcIztucouysdyleLPKJi5ry5ZjO3GP2zprrXddlFjFtvSLbEtRXOJVrCguY3nGGhVrpCkcksNhB0ASBEDU+/vj8OC+aOwFJM/ned6HxEV7AWm+7+H3nqK01hAEQRA2H7aNPgFBEARheYiAC4IgbFJEwAVBEDYpIuCCIAibFBFwQRCETYpjPd+sqalJ7969ez3fUhAEYdNz5MiRCa11c/H6ugr47t27cfjw4fV8S0EQhE2PUqq/3LpYKIIgCJsUEXBBEIRNyoICrpSqUUo9qpR6XCn1e6XU386tf1wpdVYpdXTueMXan64gCILALMYDTwJ4kdZ6RinlBPALpdSP5+77stb6C2t3eoIgCEIlFhRwTc1SZuZuOucOaaAiCIKwwSzKA1dK2ZVSRwGMAfiZ1vo3c3fdrJR6Qil1u1IqUOG5NymlDiulDo+Pj6/OWQuCIAiLE3CtdVZrfRDADgBXKqUuBPA1APsAHAQwDOCLFZ77da315Vrry5ubS9IYBUEQhGWypCwUrfUkgIcBXKu1Hp0T9hyAbwC4cvVPz5DNruWrC4IgbD4Wk4XSrJRqmPvdA+AlAJ5RSrVbHnYdgKfW5AwB5HIk4KmUCLkgCAKzmCyUdgDfVkrZQYJ/r9b6QaXUnUqpg6ANzT4Af75WJ6kUCbdSdDubBZxOc1sQBGE7spgslCcAXFJm/YY1OaMyZLNAJgM4HEA6bX4qRUIuCIKwHVnXXijLRSk6UikTdSsF2O20ZrfTIQiCsJ3YFAKeSgHxOIm2y0XRuFLkjdvmXHyxVQRB2G5sCgFPJoHeXiAQAOrryUJxOACtSbizWROBi60iCMJ2YVMIeDgMPP444PMBXV3Ajh2A10ui7XCYLJVs1gi72CqCIGx1NoWAz84CJ08CtbXA+Dhw5gywcyeJeU2NEetcjh7rdNIhtoogCFuZTSHgk5PAM8+QWAcCFJGHQsDgILBvH9DRAbjd5Ifb7eSRp9Mk3lrTutgqgiBsNTaFgI+MAP39JMINDSTezc1AWxuJeUsLsH8/0NpKForNRkc6TRugLhcJufjjgiBsJTaFgJ85Azz5JInyjh1AMEjCPT4ONDUBnZ10u7MT2LWLhDyXM6mGyWShkNtsJPTMkf4IDvWG0LM3iMt2le3JJQiCUHVsCgEPhehnLgcMDNDR0UFCHokAExPA0BAJ+vCwEfKWFvLBWbQTCRJzt5vE3OEAHjsTwZ998xBSmRxcDhvuflePiLggCJuCTSHgz0xE4O8JYXYgiNQQievQEB0dHSTU4TAwOkq3R0fJdtm1i8R8KBnBbwdCuKQjiIM7AwVC/ssTIaQyOeQ0kM7kcKg3JAIuCMKmoOoF/Eh/BNHZ7+L+kW/hEy+9Cb/42ZvyIg4YIW9ro2NyEjh7lqyW4WEATRH8U98hZHI5OB02/O839uCC1gCyWYroL2oNwmW3IZ2l+3v2BjfsswqCICyFqhfwQ70hXDhyHNf0/Q7X9L0bwLtxFX6FQ7iq4HEjI3Tw5ubkJHnnyb0hpN05aFCE/cixEM5vCcBup03O85oDuO2NPXjsLHngl+yU6FsQhM1B1Qt4z94gvnzxH2HKVYuv3/9pAMCv8TwAwCvwr/gxCmcpj4+bzc22NqAmFQQusgEqB6VssIeDOHrUbIYCwLnBAC5sC8BmI+GvqQE8HskfFwShulnSQIeN4LJdAbzC24N7Rt8Od0cYL8DD+ft+hD+GhsKf4a6S501MAE89BZw8FEDNr3rgPXUu2o714NRvAjhyBDh8GDh6lDZIuQgokaCoPBajzdFkcuHzO9IfwW0/P4kj/ZHV+9CCIAiLoOojcAAIZAOYPkTWxiN4ARQ0LsbjeBwHAQB34QbchRvwXnwRX8Z7C547OQlM/jIAlyuA9nZgpp4KgBobgYsvJgFvaaHKzkCABDyXow3O6WkSd5+vfP74kX7JYBEEYeOo+ggcAHp6gD17CteewHOgoLEXp5ACqeuX8D5oKHwOHwDNmTCkUlQM9MQTwO9/Dxw7Bvz7vwMPPEDR+GOPUcQ+M0NinUyaiHxqirJciqcBHeotzWBZCyTKFwShHJsiAm9pAS66CNi9Gzh1ivLAmdPYCzdSaMEofosr0IUz+AA+jw/g87gDN+AduB3Zoo85OkpHQwPQ3g6MjVH64cGDZL0Eg5SC6PNRG1u7nTzxyUnKJ29oIH+8Z28QLocN6czaZbBIlC8IQiU2hYB7PNS4amgIOO88oLubouneXirSAYAxtGIXBuDDNB7Ci3EFDuMtuBNvwZ34Cf4Ir8UPkURNwetOTpq0w9FRk354+eW0HgyStWK3U2TOQh4OU5R+aVcAd7+rZ02rOMtF+SLggiAAm0TAm5pIuL1eiojHxsij3rWLMk6eftrYG1H4cSV+CxeSuA+vxyvxr7gWP8UsPACALvTjDLoKXn96mg5rNWdHB3DVVbTu99N7KQVEo1TB6fFQ06wDTYE1FdT1iPIFQdicKK31wo9aJS6//HJ9+PDhJT9vZgY4dIhaxY6MkIUyNUUbkJOT5FWHw8Czz9LvVmzI4jv4L/gT3Fuw/u59n8M/Jm4sKApi/H5KQWxvpwj8qqtI0Ovr6TanFzqdJOQOh0k9XAukV4sgbG+UUke01peXrG8GAU8k5opykuRRJ5MUhZ89S4IeiZCAJ5MUQZ84QZF6IRq6zJ7tP9S+G38V+1rZ9/X7Sbibm4G9e0nIOzvNOmA6HHo81F/F5Vo7IRcEYXuyqQV8dpZsjUTCeN7hMEXmk5NkfYyMkHhHIiTeoRD55OFw4Wv5e05i6tD+kvf4NXrwPPy67Pv7/dThsK2N/Pcrr6RIPBikdESey8ni7XYbURcEQVgplQR8U3jgjMtFP7NZygQJBs2xezdZK6OjJOJ1dbQeClH0PjFBz50dCKLrvT+CsuXQ/4VX5l/7KhyCBnkjqigFkT3ykRG6kJw8CZxzDm127txJHr3fTxcXbpLl9VI+ud1O9oogCMJqsykEXClK6UunKRrX2gikx0NHYyN51Lt2kb0yPEw//X5KQzx9miL1FICZp3YAANwdYaSGAnnhZioJeTRKx+ioEfLzzqMUx64uumDU1Rkhj8fpvHM5itJFyAVBWE02hYA7nSTYPJQhkzFDGmpqzIAGp5OEPBgkj5qFfGSE7m86N4Kz5xyCtuWgszbE5oSchfpJXIgL8fv8+y4k5Lyh+vTTwIUXAuefTwVH9fWFQu520xpPBRIhFwRhNVhQwJVSNQAeAeCee/w/a60/ppRqBHAPgN0A+gC8SWu9JqWCvFHIE+hTKbrNsy+TSRLx2lozyNjno9L4zk4S8mAQ+GUkBNhzUApQyKFubwhhSxbKRXgKAPB13Igb8c38uhHyHGCJ1mdm6BgZAfr6aG7ngQMk5OecQxed2lo652TSzPRMJMhaYUtIEARhOSy4iamUUgBqtdYzSikngF8A+CsArwMQ1lp/Vil1C4CA1vqD873WcjcxGZ6uk8vR7VSKRNwq5DyVXmv6na2MiQngiaEI7jhzCFmdg9I2tB3rwXRvAH195bJWgL/AbbgNN5esuzGLFNyl627a7Ny7l6yVCy8kIXe76YLidtNFyOuliNxuFyEXBGFhViULRSnlBQn4fwVwB4BrtNbDSql2AA9rrc+d7/krFXAmlyOB5uKdbNYMMOYIPR6nxzgcdN/kJNkeTw5F8NR4CLaJINLDAUQi1H1wdJSyVmZmSt/vRXgID+ElJestGMU4WkrWXS6ycHbtooj84ouBffvIVvF6KRLnYqCGBhJxtoAEQRCKWZGAK6XsAI4A6AZwm9b6g0qpSa11g+UxEa11SZWJUuomADcBQFdX12X9/f3L/xRFcDTOgq618cdZ1GdnSdCVovs4zTAWo5/Hj1M++fi4qfIcGCDBL6YbJ3AC55SsPwdH8QSeU7LucFBp/s6dwAUXUETOQl5ba4S8tpYicquXvxikwEcQtgerFYE3APgBgL8E8IvFCLiV1YrAi2Eht9orbKGk0yTmsRjd5owQ7jDI/b/7+kjIh4bo9sSEEfZi6jGJSZR+1OvwffwQ15Ws22yU5tjRQdH4gQO02dnQYITc5TLWilJ0226v/JmlyZUgbB9WJQ9caz2plHoYwLUARpVS7RYLZWx1TnXpKGXEzmqvcFFNNkuPqa2lDcR4nKJgn4+i9dFREs99+yhr5cwZEvTWVhLy4WESdmYKDVDQsCODDEy4/AO8DgBwKz6JT+PW/HouR423envpovDUU+SRn3suvWdjI50bWz8eD212ApWFXJpcCYKwmCyUZgDpOfH2AHgJgM8BeADAWwF8du7n/Wt5oouFqyJtNhJOpej3+noSSLudRD2TIb+b/WqlSKhraqgHyv79JLinT5OQd3SYAcpMFo58iqE1l/xT+Ag+hY/g+7gOr8f3C87v9Gk6BgeBJ58kIb/gAtr4DAZNuiRnrTQ00POKhVyaXAmCsJgslIsBfBuAHTQA4l6t9d8ppYIA7gXQBWAAwBu11uHKr7R2Fsp8lPPJs1kSyGSSbs/MmCwWrclemZ6maD0Wox7kvb20FomQ0A8Olg54AFBSFAQAY2hGa4U/UPbtowtGdzeJ+f79VNnp9dJfCW43ReR+Pz3eKuTigQvC9mBT90JZDcr55NksRbvsjadSJNiZDN0/PW2yV5JJipz7+2mjMxKh/O+JifKZK+WEHCgtCmL276dIf8cO4LLLyDNvaSHxrqujn14v2T7Awh65IAhbhy3RC2UlWH1ya+TMPnkmQxE355BHo8abTiRIsD0eEtbBQbJSBgeNkIfD9Hv+/eaEOowAApjMr1eq7jxxgo79+8kn7+ykDc+9e6mJFldzxmJGxAGJyAVhO7NtBNxKuciVW8FmsyTmNhuJfixmbIxMhrJSGhooUt6zh8S2r48idY7IrR0Q2zp60Xr9IXzr/o/jpaceza8vJOT79tFmans7pR92d5Ooezz0F0M0ahpoKQU8NRLBDbdLVoogbCe2pYAzLORKGWuFqyMdDhJsu93klsfjZiqQ2019wpubKSofGqLmVizk3BWxpisEZc/hxjd8FDoL3PS9f8Otg/+QPwcWcjsyyMFcWU6domPPHnrtJ54gj3zvXjpYyN1uuqD88nj5rBSJygVh67KtBZzhzBW73ZTr822n0xQFcYGN00m+9MwMCX1rK208dnaScB8/TlkrIyNAaDoInbUBOgeds+Hvch/HR/AVvA734T68IX8OPHi5ARFMoSG/zlkrnZ0U7be3U3n+3r2m38rsLLDPH4TTbkMmm4PTbsOVu4OSKy4IWxwR8CLK+eQ2G4k2e+X8GLebPHJro6rmZjPpnjY8Axg83INJVwgTTwfzI9y+j9dDQeMgHsNjuDT/XlwgdADH8CwO5NfPnqWjuZmyYI4fNxH6uecCLbUBfPqlPTg2EcLV5wRxXnMA3/r1SckVF4QtjAh4BYp98mJ7JZslz5zTEb1eitJDIdpsZCEPhYC+vgBGRgIYaQWGshSlM0dxCRQ0OnAWZ7Ejv/4MzgMAXIsf46e4Nr8+Pk5HUxNZKyzk+/YB3d0BvKQzAL+dvPiL24Jw2W1IZyVXXBC2ItsmjXClVCrX51TERILWuffKxITJJee0wzNnTLn+8DAdxdQggQS8JeufwS34MD5Tst7URFF4ezv93L+fPPnGRspWGYhH8PQEeeBX7g3kL0KCIGwetn0e+Gpi7YJoXeNUxGzWCHkkQhubs7NmY3NwkBpmjYyQoA8PG9/dUH4I82M4iEvxWMl6UxN1P+zooPzxiy+mKUFNTfQXQSBAfyVwto0IuSBsHrZ9HvhqUix82WyhvaI1eea5HHnjwSBVd3q9RmB37CBBP32aIvS+PhJ0I+SqbJn+JThaNgVxYoKOkydpg3N0lMT74EF6z9ZWEnIu1xchF4TNjwj4CmDh474r1iiaZ2E6neSNe71ka0SjlALY0kIFOu3tlDfe1maEfGio8LXKCbn1tlXIp6aAxx6j19m5k0S9oYGqOzs6SNQbGsijr60VIReEzYwI+CpQrsqTo3Ie85bJkI3Cvb9jMWNtxGIkqJEI8OyzFD0PDtKRTlveZwlCHonQcfo0eeK8ufrc55osmaYmupD4fHSh4bF1IuSCsDkQAV9lFrJXADO/s76eBHRqivLKYzGK0iMRap7FFsvgIPnpzFKEPBqlrocnT9ImZzhM73HFFRT1j4/TOTQ1UWUni7jTKUIuCNWOCPgaUVzlabVE/H6TllhTYwYwT0+b3ivNzSS27e3kjff2krUyO2teh4X6UVyBK2A2h8sNYU4kgKefJiHft88Mer70UvLjGxpI0INB+j2TkYhcEKodEfA1hqs8uWcJYKLyujr6vaaGxJI3GaenSeSnpwuj5IkJ6pMyOlo4hPlK/BYAcAduwA24K7/OWSx+TCEK6kebSgHHjlH++L59FO03NFDWys6d9Ht7O71ffT354yLkglCdSBrhBlAuDZE3Qbk1bSJBAj40RCI7PW0icU5DHB4mi6SYd+Eb+AaNIS3gHDxbMtNTKcpa2b2bBPvCC42Qd3YaIfd4SMRFyAVh/ZE88CqEBdxaHMTr8bgpEorFqAgoHKbfh4aoTJ/neA4OksAXcwUexaN4bsn6q/AAHsSrCtZsNvLId++m6P/88ymPPBAwqY+NjeavBclaEYT1QwS8iikeMsHkcpSFMjtLnnQ8bop/kkkTiff3G0G3WitMEBOYQHPJ+sfxMfwtPl6yvmsXiXl9PQn5jh3kyXd0kKC3tFAfGLd79YRcuiYKQmVEwDcJlewVjsq52nNykuyURMLM6uTByQMD5acEFQ9hZh7B8/ECPFKy3tFBfcjr66mVbUcHefIdHWbT0+MxEbljmTsq0jVREOZHKjE3CcXRLF9feTBzLmeiX84dZ5tj1y4SXBbywUESeqbSEOY/xH+WTUHkCwO/T309tbDt7KT34yKhjg7KnmERd5ZeI+blUG/5XuaCIMyPCHiVwkKeyxVmr9hslL3i9ZKN4nBQDnckQmIbDJKP3d1NlsrJkyTk1ilBwNJyybkDYjBIBUHBIGWwdHTQwd0RuVyfJxq53Yv7rD17g3A5bEhnpGuiICwFEfAqx5qGyLCQW+d52my0ybhjB4lsIGDGvvX1GTGfmip8/aUIeShkBJwFnRtodXXRGk8pCgTMxaemZv7PeNmuAO5+V4944IKwRETAq4SFNvGs5fpMLkfC7XKReGezFJH7/SSu0Sjlje/cSRHzvn0k5MePF1orwPKE3OejTdVgkAS8s9O0tm1pIRFvajI58G63+WuimMt2BdZFuGWzVNhKiIBXAUvdxCuX9cHFQfX1JJgeDwks91159lkS2e5uEvLTp0ncQ6HC12GhzsIGm0W0K5XpP/YY2Tlc9s+DmFnIOzroHKxCvlFl+rJZKmw1RMCrgOVu4rEIFtsrSpFPXltLx+wsReWxmJnX2d1NBTwDA1SZWSzkdlBe4/14NV6Nf8mvs5DXtY8gNtwKgLJjnnySov/hYfLCBwcpCu/uNkLOXRA3Sshls1TYaoiAVwEr3cSrZK8ARsRZyD0e2vw8fZqslbNn6efwMAm5ddwbALwGDwAA3ocv4Av4H/n1meE2AEAHzmIYHQDIi3/mGTq6u8lSOXOG0g27u8mTb2sju6e5mbxxm42Ev9xnWG1ks1TYaiyYB66U2gngDgBtAHIAvq61/opS6uMAbgQwPvfQD2utfzTfa0keeGVW25stnhbEcEEQTwkaGCARHxgwvciPH6cIuhh/z0m8cN/9+OHd7y+57yX4GR7CS0rWOzooK6a9nSLzPXvogtHZaTY8XS5Tpr/WQi4euLAZWXYhj1KqHUC71vp3SikfgCMAXgvgTQBmtNZfWOxJiICvP9ZBE8WiHouRjz0zY6be9/VRVefgIP3s7TXPcXVE0Hr9IShbDp2RcfzqW28veb8P4dP4LD5Ust7aSpZNWxv9vns3ReRdXSTiwWBhmT5n30i5viCsYiWmUup+AF8FcDVEwDcVlWZ5JhJ0jI+ThTI0RNbHiRNkrfT3UwoiQCJe0xXC7EAQqaEAvIghhrqS93oYL8AL8XDJen09DV7miHzXLorG+WdDAwm5200izgIuQi5sZ1ZFwJVSuwE8AuBCAO8F8DYA0wAOA3if1jpS5jk3AdQar6ur67L+/v5lnL6wmpSzV3hq0MwMCTiL+Zkz5GlzD5YTJwqfbyg/hBkozFxhamupPJ9Hy+3cSYLe3U1ROXdALNfKVsRc2G6sWMCVUnUA/gPAp7TW31dKtQKYAKABfAJks7xjvteQCLy6qCTkuRwV/AwPUwXnyAj9/vTTZK0MD5O1Yh0uYaU4l5wpJ+ReL1krvMHJnQ+5ZL+lRYRcEFbUC0Up5QRwH4C7tdbfBwCt9ajl/m8AeHCVzlVYJ8qlIfJ6IECphzMzptdKRwdF5b//Pf0+PEzWSjJZ+PylFAXF48BTT9HFYf9+isTb2ijab2sjceciIbebRJyHMPNFRynjmQvCdmJBAVdKKQDfAnBMa/0ly3q71np47uZ1AJ5am1MU1ppymR/sP9fXmxREblzV0kIWy7FjJLI8gDkWK3rdJQh5LkfFRidPkojv3Uvvc/Ys+eVdXeST79pFEbnNZgYxa23EXKJyYTuxmAj8agA3AHhSKXV0bu3DAP5UKXUQZKH0AfjzNTg/YZ1hAbQOY+Yug93dJLTjOoIzsyHsuSKIc88N4NQpEl5OSVyJkGezpndLZycJeWurSW/cs4dEfO9eurCk0yYq5+cDJPCVyvYFYauwoIBrrX8BlDU15835FjY3xfYKC/nhvgg++vNDSGdzcNhseM/BHlzVEsC+fSTiLL7Dw6VTglioo6hDHYzKs5DbkC3YCOXUxs5OyoTZsYNa2J48SR58Zydw7rnUMoAbZ7lc9NM64UiicmGrIpWYwrwU2yu/GwwhnaVy9Gwuh2l3CC+6OICBAcrnPnCAPO2hIRLaUKi0la0PNG3iTrwZb8bd+fUc6I0aEUIEjfl1FvIzZ+jisHMnCfnx4yZSP+88sngAuui4XMYXl6hc2KqIgAuLxm4HrtoXhOvnphz9efuDaGmh3it79lCJfiBABUKPP24KgkZGqGe5lRtwF27AXSVDmMOgEvdLcQSP4dL8+sgIHTx9iCPyU6dIyHfsKBRy3tzkARMSlQtbDRmpJiwZLkd/7p4gLtlJ5ejxOPnRiQSJN6cbTk1RRN7ba3LJiyNy5jIcxmFcUbL+Bvx/uA9vKFlvbqaNzs5OEu+WForOW1pIyJuazKQgwPjkViQqFzYDMhNTWDN4KLPWJOCpFGWtTE1RhHzmDGWtHD9OQj40NL+QVxrC/Dl8ALfgcyXrDQ1m8DIXBHEO+Xnn0brPZyJxp7O8aEtULlQrIuDCmlDcHCqbNeX5mUyhkA8MmL7hp04Zb7uSkNuQRbaMy3cv3og/wb0l6z4fCTg3yerqohTE9nbKWunsJKvH46HHOxzl88clKheqDRFwYdWZb0CCVchTKTrCYYrEBwdpc7O/n3K/udKzuJWtlXLVnWEEEESp+vv9JNrBIAn5vn30s6ODBH7PHhJ7bmdbKSIHJCoXqgOZSi+sOvMNSLDmjyeTJOBOJ9kdTU1mtmZXF0XmzzxTaLcUxxXlcskbESmbSz49TYfPR4VGIyMk4Lt3U8TPG6D795PYO53kj7PFYhVzKRASqhkRcGHZLGZAAg9fdrnocLvpp99Pgx0mJ8mz7uqiTJJnn6VIvb+frJdUqvD1llIUFI3S4fWSkA8Pky++Zw+Jen8/vfeBA3Qudrs5z2Ibxdo3RsRcqBbEQhFWxFIHJGQylK3CG52xGIn41BRZKxMTFCH39lIkPjBAIhyPl3+9pTTOqq2lSLyhgYR81y6yVJqa6Pa+fWSz2GxkrzgcJiovhwi5sF6IhSKsCUudJs8dBe12isZraihCb2ykDcZwmAR2/37a6GxtJVEfHDTWiBUW6m/jLXgL7syvm4g8By4kjsXoqKmhnPTxcXqP3btJyIeGSMi7umjT02ajc7Tb6TnFSDMtYaORCFzYMHijEyBhjcfp9sgIievICG2Ccq+VcJg8bI7Yy/HX+DK+jPeWrLuQRBqFieBOJwl2czM17dqxg+wVvt3VRaX63MqWfXKHo9SjZyQqF9YCyUIRqharzx2NkrWSTlPUHQ7Thmc0SnYKl+ezkBdXdzLX4sf4MV5Rst6EcYTQVLBmt5N14vdTFWl7O9kpTU2mle6BA2TBKFXY1lap8mIuUbmwmoiFIlQtLheJYDpNmSM+H1kl+/aZToThMDDtisDbEEIwGkTziQAmJsx9xUL+E7wcChoHcAzHcH5+nQuEduM0+rEbAEX9Z87Q/Z2d9FoDAyTk3d0m9bGxEbjgAhJ1bpzFvcldrsKNTmlxK6wHIuBCVcCCmE6T+Pn9tO50UuT7aG8Ed/z2ENI56oL42uf2YH84gOPHScDHxigyHx8vfN1ncB4UNAII53usAEAf9gAAevBr/AY9+XUuLmpvp4vI0BBF4vv308VkbIwE/MABsl9SKYrGs1mKuB2O0qhcvHJhrRABF6oKzvpgW8XvJzE8/rsQMjoHDSCrc4jWhPD8iwNobibhPnmSIuexMSoImpgobF4VQSMUNOzIIAOTWnIIVwEArsd3cQ+uz68PD5u0w5kZet1gkDY8d+yg96yvp9u7dlF2DWev8DAMicqFtUYEXKhKXC4S4EyGItcXHAji9kM26kNut+GlB4OoTZAQtrZSlDw+TrnkbW0kuMPDJOrptHndLBxlc8m/hz/F9/Cn+Bv8HT6Jv8mvj43RUV9vhPzkSRLxXbtoM/XkSfLJ9++n87WKNxczARKVC6uPbGIKVQ9H448NRPCbvhAu3xnE+a0BTE2Zzc1olCo+x8fpOH2axHV0lI7JScpoKUe5XPJ78CZcj3tK1hsa6ALh8VDaY1cXifnOnbTW1ARcdBH9brebKk+73ZTtW6NyKxKVC5WQLBRh01NclZlOk2jPzJBlMjxMv6dS5F1PT1NEzh45C3nxyDcmjAACmCxYO4W96Mapksd6vSZzxeulvwJ27SJBr6ujiP3CC+l3Tj/kFES2WCohzbSEYiQLRdj0sC3BESyLotttioHYOqmpIXH3+Sjy7u01RUHj43NZLUVFQY2gVJZ/w0vxUvw/AMA+9JYt04/HyTqpqSEhj8fpvXt7KZecs1nq6iiTpb2dxNvtJoHmqNxuL930lMETwmIRARc2FSx61mjc6aSI1+MhgWxupmj77FmKigESzFTKeOShkMlcKU5BfBl+BgD4PN6P9+OL+fVyQj47S6JdU0MXiJkZKkBqaqJqzuZmulDU1pLQd3ebbJviqLxc1C3j4IT5EAtFWHeW0j9lvsdy7rgVu52i4elpirx5DBtH7gMDdP/ZsyTeLOSjo5WrO9+Jb+KbuLFkvXgIM0AXkYYGEu7aWrJYurpIvLnCs7GRBk1Yy/Q5jXIhe4U/o7C9EA9cqArm6yG+3MdmMoW2A0BCGIuRkMdiJiMllSIhP3uW1rmBFg+d4ArPcrwID+EhvKRk3YsYEvCWrDc3U7Tv95OVwiPfWltJyH0+Kgxyu8lHd7vpeVZ7ZT4kKt8+iAcurCpL7ULIzNdDfLmPffwsncvlO4O4pIvuz+UoAvZ6ydbweilTZWSEfu7ZQxF6bS2J7Pg4RcacV37mTKmQ/zteDAWNc/EMnsF5+fU4agEAbRjGKNry65wR09hI7zE9TZF+QwO9f2srvYffT3ZLMEgRfC5XmEvOQq11oWCLVy6IgAtLZilRdDGL6SG+lMcWn8td7+zBxR1GxAGKdr1ek/o3PU1C7nCQgEejJKI8aGJsjIQ8HCbPvFjIn8UBKOiS2Z0jaAcAXIQn8BQuyq+Hw3Q0NZGYBwJ0Dh4PtQtoa6O/DurrTTaL3U7nyh442yws1MViLl759kQEXFgyS4mii7lsVwB3v6tnUdH7Yh5bfC6/OR3C5bsDBbYKV0k2NVFmistFwj01RWJtt5v+Kw0NJKLj4xQtB4P0uN7e0tmdITRBQcOFJJIw/WafxMUAgJfjR/gJXp5fn5igo7GRjoYG8uOPH6fNzZYWeo/BQXrf7m66yPj9tFkKmB7llYRcovLtxYICrpTaCeAOAG0AcgC+rrX+ilKqEcA9AHYD6APwJq11hd5wwlZiKVF0OZbSQ3yhx1Y6F275ypuc/LvTSQKdSJDn7PORaE5M0HPq6ykK9/vpcaOjphJzepoKhCYmCs8hBfdcZoou2NTkbog343/hNtycX+eI3OczI+YSCYrI29qoMGhykt67qYnsFq/XXHgyGXodj6ewipOtF0ai8q3PgpuYSql2AO1a698ppXwAjgB4LYC3AQhrrT+rlLoFQEBr/cH5Xks2MbcOy/XAN+JcrLnjDDegisfJI+eMlWiUNjpzORLQmRk6WMijUbI7+vtLG2dZKVfdeRv+AjfjtpJ1r5dsFW5fy4VA55xDFxKfjyydHTtI8PlCY7PReS5kr1iRqHxzsmpZKEqp+wF8de64Rms9PCfyD2utz53vuSLgwkZSnK1izSmPRikrJZmkjJVolAQ+l6OqzmTSbIKOj5vqz6EhWivOgmHKCfmvcBWuxq9K1nnABNsrDQ30V8L555OQ+/1m0EQwaDo1ut0k2rzpaRVp7sdSjETlm4tVEXCl1G4AjwC4EMCA1rrBcl9Ea10S/iilbgJwEwB0dXVd1t/fv+STF4TVghtkWWEhn50lvzseJ7vl7Fn6PZMhAR8bM2I/NkZCHouRgI+OmnzzchzFc/AcPFGwloEdTmRKHmu3m/TDpiYSaY+HWtg2N1PE7vdTbnlrKwk5b9LyXxZsr7BI8z/zcqItUXn1s2IBV0rVAfgPAJ/SWn9fKTW5GAG3IhG4UC0U2yo2m/GPrbYKdyBMpeiYmSELJZcz3vnoKAn82bMk6sPDpRcJ5na8HW/HP5WslxvCDFBEzj653092CZfq19SYKUJdXUbEa2rMkIxyxUGVonIR8uplRQKulHICeBDAT7XWX5pbexZioQibGO7PXc5WyWaNrZJKmWERNhuJczhM92cyJOI8FSgaJSGfmqJc8uIGXMz78AV8Af+jZN06hNlKczOJNf+sqaHf2SdnMd+9m366XEbIHQ7jlVtFWms6itvZir1SfSxbwJVSCsC3QRuWf21Z/zyAkGUTs1Fr/YH5XksEXKhGiqNx7tFt9cdnZkgEx8ZMyX06TeLNNgsL+dQUHSzkg4MmDbCY1+CH+CGuK1l3IlUweIJpajKbmizc9fXUwra+3qQd7tpFkbndThYMt7UFjE9enEdeHIFLr/LqYSUC/gcA/hPAk6A0QgD4MIDfALgXQBeAAQBv1FqHy77IHCLgQjVTLOTcw1spEmCOyHM5isanpymCTSbJ/85k6AiFSMjZbuHhzCMjJPbluBRHcAQl/z4RQBiTKHUm2Tpha4V98vPPp3Wvl7JZOjvpMTU1dFgbaPGwCatIV/LKJSrfWKQXiiAsgmyWRKycraI1iXg8ToKezZLfPTtLj49GKQq32WiNbRUW8uFhUyAUjZZ//04MYhA7S9b34SR6sa9kva6OPPKWFhLx+nqKsM85hzY4a2vpCAQoDdHrJdFmMWePnAXdSrl0RInKNwbphSIIi4BFzBqNW6NyjnY5Gu/sNBkqAAkoV2zu3EnCOTVFzwsGScBbWsyotuKe5GexAwoatZjBDHz59VPoBgA8H4/gF3h+fp3z1KenTcMsrtx0u+kcurroXEMhEvDdu8mGsdspardODWJBB8pnsFjnekpUvvGIgAtCGayDFsplrDQ0mAZZ8TgJ5cwMiXcgQEc4TJG530+PD4dJ4INBEtrRURLVoSF6rpUY6qCgYUMWWcs/0//EHwIA3ow7cTfenF+fnqYjGjUVnoEAnVt/P11oOjrITonHzSCKYJBE2+2mKJyrVVnQy5XqK0UH/5UiUfnGIRaKICxAsa3CkSdH69EoRbyJBHng09MUdXPr2nCY7teaxHNy0hQC8bi3UIg2PYuF3Eq5oqBP4CP4KD5Rss6j3rh5Fpfit7dTjnlHh2nu1dREa263Gf9mt9NtFnPOWMmfi2SwrCvigQvCCmABL97k5J9ak3DPzpKlkkqZXuSpFK1zGmIyaYqGEgmTrTI+Tl453654LmWE/Id4Da7DD0vW6+roCAQoKvd4jLDv2GHyzL1eI+S1tcb35w1PHl3H5ftWrFE5I1H56iIeuCCsAGvEDRhbhUXLZiN7hNvVJhIUufp8FFXHYqY3+cQECarbTRF5bW1hIVBjoxHycsMluOhnCO1oxwgA4LW4HxoKZ7ADXTiTfyx75JOT9P7skUej9F6BANk/zc10UZmYoMe0ttLjczm6ADkcppMjZ7IUR+HW2+KVrw8i4IKwBIoLYcpF5YEAecyxGAkXD12eniYxY5FPpUg4Z2aML71jB6Udjo2RP83ZK+ynW+nAMADgAbwKr8KDAICdGKw4u3N2ls6pro7e1+czlk5jI+BsiyDuC+E5kSD2jwfy0bnfT58xnTZiztaKy2X8cGsULl75+iACLgjLwG43IlWu26HHQwf747zG+eQARd7hMAl8XR2tJ5P0uK4uquQcHS2NyIvf69X4FwDAJ3ErbsWn8+vlhDyZNI25amtNB8SQimC0/RB0LIff9trwhqkeXBAP5Od6NjbSBYWrVLm1AGeucGTOFzVrVG69zecuZfurg3jggrBCitMN2TKwjkKbmjLiyc2xYjEj2qGQyfiYmjIRbzRKg5i5AyJvdk5NVe63cgPuwB14a8m6HRnkUKicLhdZP/6ek8ic9yxV8WugO34urmroRmOjmenJPnpDA/214JgL/7ggiEXc7TaRt1VerJvAjETli0M8cEFYI3gTEygtAuL7GxpMWX46TYLHWR6zsxR1x+Mk1H6/EXJ+biRCEXltLUXDExNG1GOxwve7E2/BnXgLno9H8AhekF/ndMRazOTneKZS9DqJJ4NoPMcGZctBaRtSg0Ec7StsX8vzO30+I+Zsr2Qy5nMlk/S5uPeKROVrh0TggrCKFEfj1giTBSoeN9kpmQw9ljc+OQ2RKzptNrqdyZDwhUKU1x0KmYg8HDY54OXoxgmcwDkl650YxBA687ddHRHU7w/BlwgikA3A5zPNsGprabOzvZ02N+vrjZBz3rnDYXxvruzkhlrOubYui4nKZdOzFEkjFIR1pFjIWbyttsrMjNlc5Fzz6WkS+FTKNMVi3zmRMNOCwmES8okJuhhMTFCEHI2Wz1wBqK9KGKXj7w7iMTyOg4WPnYuwg0GyT9xuU7nZ0UGZKnyfy2X6l/t8JvLO5crbK5ylYv1LxRqVW8VbonJCLBRBWEdYdNlCKN545EHKvNGZTpOQc6l+LGbSECMRsiXq6uhxXC3Z1ETCfeYM3ceVoJwmGCmaUBtBIxQ0nEghBXd+/SguAQBcjt/mG2pFIuZgH7y+ns4jHqfMmOZmOofWVpMOyYVDdXV0O5czefEOB31Ga+UnYMSbxZqjdJtNUhEXQgRcENYIa18VFvRstlCYHA4SvNlZk9HCYud2k5DX1JBAjo2ZrA+u7OzsJCEdGwP6+kg4OWuFBzFzbxYmDVfZIcyHcQUA4Bw8m7dcJieN9x0MmtFumQxdMIaGTDTe2mp6rnCFJ2ex2GwmDZELmzh7xToGrtgrt+baA7LpWYxYKIKwDlgtg/lslXjc9BfnYRCJhJnROTVlcsIdDhJRvjik0ySop0+T6MbjJKbRqPHVK/EELsJFeKpg7Qo8mhd1xuMhsQ4GSZy9XmOt1NeTiAcCZvRbba3py1JXZyo7M5lSe6WmpnSjk7FG5dbiqe0SlYsHLghVAIt3Oc/XGoWyP85C7nCYhlVKGc+bI9J43Dw/kaBioNOnjaceCtE6l+xXGsL8RtyLe/EnBWsvxb/h/+GlBWsej5nZ6febroZuNwl3Wxv9JdDWZh7r89Eal/OzTw4U9innniyMdeOTvzdrJL4donIRcEGoEuaLxq2/c654Om3S9AAS92iUHjc8TLet8zxdLuM99/XRZienG4ZCdEEIh+moNPLtRXgID+ElBWvX47u4B9cXrPEot+ZmEt76epMe6fFQ1gpH5ByN80+vl35qbTohWguDOLK3trW1XnjK+eNbNSoXAReEKsO6scleLwux1VZJJEptFa3J504kSPzOnqXfGfaY2XM+dYo2O7kydHycXoMnB1VqnlVuUtBf4h/wVfxlwZrHUyjOPl9h5kpTE4k4Z7dw6qHPR8/x+Uj0eR+Ah0xwGqPHU76NQfHv/L1ttahcBFwQqpRiW6VcNJ7LkYgnEvT4ZNJshkYiJMo2G1knXKHJUThntWQywDPPkNhbhdzhMN56ceYKUy6X/JO4FX+DT8A6hNnlMn43/+S0Qp7f2dZmZnp6PPQ4awpiLdUY5QdIs73CB1d6MlaLpVz+/VKi8iP9ERzqDaFnbxCX7SodZbdRiIALQhVTToTKCXk6bYqAcjkSYa5+jERMhD4wUNhIKp021Z7JJPDss7ThyUVCoRCJ78yMGdxcbhBzG4bxGC5BG0bza9/Au/Bu/J+CMn2HgzY6PR7yvTmXnEe5eb1krzQ2GvuF0xVra40dw6+VydDn4SZa/JNTEYGVR+VH+iP4s28eQiqTg8thw93v6qkaEZc8cEGoYqxl5db0QxYf/t3pJGHjAiCAxMrppMh2dpbEeNcuEvPBQbMhmMmYqfUeD7B/P/D00xSFu90k8l4vCerkJPnrU1OFg5hH5lrY1mMSD+MaHMTjuBHfxI34Jv4Fr8Qb8M9IwY1MhhpxWdvQBoMk4HV1xt93u03mSjxuslkaG40NxPYKQJ+JS/VZwFnQbbbCv1g46rZG5XyRLBeVH+oNIZXJIaeBdCaHQ72hqhHwSoiAC0IVYe0dYi0CsmZssK/sdpOlwjnkuRytdXSQOE5NAd3ddD8LeSZDz3E6zdDjeJyslbExU97vdhe2s+VsFhbDKTTgEhxFDRL4Aa7DtfgpXoUHkUQNHsUVeDEeys/0HB42PV/cbtNu1+83OeEjIyT4wSBdiCIROjfuU+520+Hzme8nFjNDJ6yRucNRmI7IWFv/cuMwa1TeszcIl8OGdCYHp8OGnr2lVavVhlgoglClzLfJCZhoM5Mx1gjng/PknKkpk6XCKYROp/HJWfg4xbCvjyJ47s2STpPYTUyQwE9OmqlDVuzI4Ha8A2/Bnfm1fnThCvwW42iB220iZx584fMZa4VHvjkcFH03NlLvFc4db2mh371euu3303uwEPNfIVYh501RhoW/OJvF2j3ydwPigVdEBFwQlk6xkFu9cWu2SipFoptMmtucnjc5aQZMcPMrtk20Nul87IcfP06PC4VM+1ul6PboKN3H7XEL0fgcPogP4PP5lSRcOIBn0Ic9eSHnClS/3/jl3DSLNyq5D3lTEwk4++JNTWZj1Ost3NTkv1ysAye4sZYVq5CzBFpbAFdbBosIuCBsYsoVs5Tb5MxmC+dyAkbIUykSYBbdsTFjp3CE39JCtyMRsjVOnCCx5kES6bRppjUxYQqEym14vhdfxBfx/oK1i/E4nsTFqKuj57E1wgLe2GjG0yllslI4KuduiOyd8+8uFz3OunGrdWG5vsNhOiYyi4nKqyGvXARcELYAxdF4pe592azpasgFQGybJJMkwNzKNhw23RB5Gn1jI90fiZB/3ttrhJy9ZI7sWcg5zbGYN+NO3Im3FKw9H4/gF3h+vkGXx0OfhfPHAwG63dhIj+epRRyVt7TMjYGbs1esrW+9XpOdwqLM4s1FRpstKl+2gCulbgfwSgBjWusL59Y+DuBGAONzD/uw1vpHC52ECLggrJxyFYmVbBXetGR/nP+58yagVZAnJkjYuMSfc7rTabpvYICOSMRMBNLapDBydWcsVpi5wrwC/4p/xSsL1l6LH+B+vDZf9MPC29hoeqmwR86RtstFlg+X6gcCply/vt7kkXs8pqLTWrVpzSfn9yyOsvm7qpaofCUC/ocAZgDcUSTgM1rrLyzlJETABWH1sEbj1oiRsUbjbKmw1cGWSS5HRT2xmBH8eJxEmyN89qPTabJdzpyh8nzuzcL++OysKQgKh+l1ywl5D36NX+N5BWvvxDdxO96Zj5BZWP1+ev/aWhLkmhpTlcl/KXi9ZKfwRqc199yaZuhw0Gfk74qHM1vfszgqZxEvHkTBwzbWKypfkYWilNoN4EERcEGoLhaKxoHK/rhSJNhc0clzO1nIp6cL0+64UjKZpNTA4WHKWolG6eCInDNaJieNoJcT8vPwNJ7GBQVrH8an8Bl8ON+dkDNJamvpqKkxGSsNDfTZuTe612sKgzgqtxYR8cXIOh2Ivx8Weo7K7fbSKJuF3Pp9c0S+1lH5Wgj42wBMAzgM4H1a67JFuEqpmwDcBABdXV2X9ff3L+8TCIJQkeKBEda+2kCprTI7a/pzs4CzAHHmCQ+P4FFvfHHw+0kEk0myVM6eJZ88FjNVogCJNkfp7JXH4ybFkdmJATyFC+GHmQn3Ffx3vAdfhs1ug4/SyeH10nM5Guec8Pp6k3kSCNDjgkHjlXMJf10dHUoV2idWe4Vfh6N8u700yt6IqHy1BbwVwAQADeATANq11u9Y6HUkAheEtaXYVimOCq3ReDpNIszdDq1RZCpFvjbbL9ksibN1U5DHqc3OUuvas2fpmJmh53PaIkfkMzNGyK19zJkgJvArPA/n4ER+7R68CW/GXcjZnPD5zEZnJmPmcHKuuM9nNie5F4vPR2Ju7YbI4s8ROfvibK+wCHP/Fqu9slFR+aoK+GLvK0YEXBDWnuLIcCFbhS0Vzgm3lqOz+FqFPBo1IuVymWh1dpa6Ho6Oms6HvFmqNb1+JGK6H7KQc4ojQK/r0TH8BNfi+fhF/nwfxgvwCvwISZsXtbX0vnV1RsiVIsFWis6HUwrdbmOvNDTQpmd9Pf0V4fWai5DVd+e9AaWMaHMfFxb2clF2Nrt2Ufmq9kJRSrVrrYfnbl4HFI3yEARhw7D2VQGMIFsjct6gtNuBY2MR/OpUCJftCOKCtkBeTLlQpqaGRJh9bhbc2VnTrpZ96ec8h0S6pYWaZQ0Pm0g7GiUBTSbNcIfJSfLIp6ZMRWjKUYsXZP8TDp3Cd/Bf8Abch2vwH4ijFsdyB3B19JeYcjTm+4dbhVdrEmfuq8J2kc1G7x0Ok4BzGqLXS4/nromcbsjCnU6bXjSzs2ZyEJfrW6NyPhduf1AcnVuHVKwWi8lC+S6AawA0ARgF8LG52wdBFkofgD+3CHpFJAIXhPWl3GgyazT4u4EIbrjddOC74209eRFnW4WfpzUJLtsunGvOQprLFXYQjEQoEh8ZITGPRuk1YjEzRWh6mm5PT1OZfzRKIhmPWypFczn8A/47bsZt+fOOoAGX2J/EQG5HPoqurze2iFImunY4zH0cqVt7l+/YYdIQ2S/nzof809rZkC+INTXGgrGuW+EUTS4qWi5SyCMI25hKm5xf+4+T+NLPnkVOA3YFvPdl5+K/vqA7PzwimTTzN6250Vz8w9klMzMmUlWKottslp4XDlPa4dAQiXkyadIVrR757KzJJ5+eptdNJo2Q53Iat+JT+CT+puCzXOQ4huO2A3C5SFR9PhMNW8vteZOTLRC3m7xx3uBsbSURb2gofB23m97H6pMDJurnqNy66bnaGSnSTlYQtjFWzxswkfVz95R24GNvm60JjrgBk0vNXQI5g4Uj1ESCxCsSMf1O6uuBSy+l5lR9fRRpDw/Te3DaosdDz62poedMTdFrTE6az+DxKHwu/RF8JvcRvCv3f/F/8W4AwJOZ8wAAz9eH8Nv0c/P2CeeEJxJ0Maqpoc/BZffZLGXPjI6STz45aeZ5BoN0UeINT5fL7A+wQFsHZ3ADMZ4Nyo8pF5WvJhKBC8I2o3ij7Uh/BI/2hfDcPUFc2hUoKBtnP5ejcY6w2W9mS4TTB1mQYzFT1cm52zyJfniYrJXhYSoM0trkp3PqIkfk09OmKAigtZoaM/D5dbgP9+ENBZ/v1c4f46fqWihlNio5h1xr+uly0We0TgziplrcKbG5mbxyq71SW2uyVgBjr/BfNFxJypu71myUlWxkioUiCEIBxbZKMVYrIJczueFsqxTnlsdiprEVF/XE48Yf5+wRbsR15gxFwENDJNI8YWh21vzOlZ7RqInIOeLlzoa5HHB15mE8jBcWnP/bnXfhbvxZXqi5oIezV5xO+j2XKxTpbNakIfIGZ2cnnT8Pbfb7zQBmLiZie0kpY69w73YuFFouIuCCIJRQvMlZDqsNwPnjfHDeNPvfPKPT6p/zwAmO2Bsa6LX4fQcGTMZKOEyP5Yk97MVPT9P7zcxQ+mE0alIbeSiF1sB5qaM4iksKzv+Dri/jq46/BmBSDHnTk+XPKrJcccopiZxP7vWSkPt85jZfFFjA2WJhy8k6KSi4gvkQIuCCIFSkOBovrpYsZ6twJSdvbvJzuMFVNGoidrZh7HbT3raxke6z20mo+/rMRufUlOmvwpWg/JqpFP2cnDQROW928kVkZ+oUfp89Fw6YD/ZF5y34mPPTgFL5BlncLIvTAb1eE1mz0HPPlMZGU9Lf2WnK9VnM2f/mvzjcbvNXi8tFrXCXiwi4IAgLYhVy9rrLdeQDjK3C4mqdjsObfImEmRaUShlbxeUyNkggQBE2e9vWjJXpaRJFru5Mp42ox+MUpXOJPl9Q3G7Tk6U5O4JDmcvRibP5z3CX4224ueabyCl7fvBDU5MRaqVMcQ+nI/p8xs9nK6WurrDKky0YznrhYcy8IdzcvPz/LiLggiAsinJtVIttlnK2CtsdQGEaHZfhW4UcMLMueehxUxNF3pwH3t9PGSKjo2Z9etpE/eyXJ5N0P/ckt871zOXmMlCSU/hp9sW4TB/Jn9dP7S/HDd7vI5atybeqDQRIjLn4iIWY+6NwH5hMhkScUw4bG2kWKeeRc0Ru7XTY1rb8/yYi4IIgLIlyZeGLsVW4yMf6ON54jEYLs03YWuGBy5x/zRPrJydpo3NsjDxyLgKamTH55MmkqQrl9MOZGbOZao3IndlZ3JN5Ha7N/Th/fr9Tl+L1gZ9jPOnPCy73TPF4jLfNEbk1tTCXo0icRdvvp8Ig9se5Ha7XC+zevfz/FiLggiAsmXLRODC/rZLNmmicszK4HwsL+cxM6SBmviCwKNbUmFzsiQnKWhkfp4icN0itG528ccqFQdxAiwdacNXk7CyQS2fxv9M34m36H/Of4yw68EfBIzibbcv3P7HO6mRriPPk2SfnqUKc6cL55zt3ktCz5XJ5ifwuHhFwQRCWTXE0XtxJkNcYFnGOsFn8OP2QM0h4U5IzVTh7gx/b0ECPTyRISIeHScTHxkjIi0v6edOUxdzaj9ya3sibqTqn8ZHErbhFf8acO2zoCRzHidw+ACTGLS2mLzl/FpfLVGLW15NY818TPp/xxNvbadPzZS9b/vcvAi4IwopZaJOznK3CG50cZVunB6XTZpYmWyKAEWyO7Hl4A2etDA2RiLO1wq/LQyi433kySQcXBLGQc+46V5rmcsC7Z/8nvpB9T8HnfWHDYziKg/mIvL6eBN3rNefG037YXmloME2t/H4S/aYm4D3vWX6JvQi4IAirQjlbxdojhNfK2SocMbPgsmWSyRi7g7NNOPukpsZYMY2N9Bi+3d9Pm5cckXNK4vS0yXixTiKaniaPPJk093FRTzJJ5/u6xN34x/SbCz7za/w/x69c1+Q3NjmdkPuKAyYVke0VzifPZKjPyt///fK/cxFwQRBWlXIWymJtFWpOZewUFsHZWZMeyIVC7JuzGHIv8ETC3D8wQBueIyPkl7NFYhVyzk+Px00eeSJhXodfmytBr5n9CX6YfHnB53lr7T/jp3Wvz+d2e70mKueCHf7MDocR+/Z24EtfWv53LQIuCMKqs5hNzkq2Clsi1h7lnKLHlZhsgfBrJpOmr4rTaXp/AyTEw8MUYY+MUGTudJryfp4SBJgURBZyvs2FQHa7ObeL4r/Bw7M9BZ/7fbX/B9/1/zlSKXoPHhjB0TdQWNBja4ngle8IoWdvEJftCiz5exYBFwRhzSi3yVkuDbGcrcI/2Vbh9rVctBOLGQHObz7O9dfOZk15PI9wi8XII5+YMMVAPCwiHjd9WwATkcdiJPxstbBvz8MrbDagK3YMh+PnF3zuz3j+Dv+r/iNIZ1Q+IucUQ05fTPkjGD3/EJSdeq7f/a6eJYt4JQFfg/GbgiBsN4oHGhRPAwKMaPPjuQrS6TRDGABTwcjzLrmpFI9Js9nITuGCoFTKRNs1NfSc/fuB888HLrgAOHDAlLk3NpoCHK6Y5En2XV2Uw93QQK/h9dLrc1OqQd956LgojIM9jyBiawAAfCjxUQyN2PDZmZuBXA5TU3TRGB2lC0ImAyTrQ9Aqh5wG0pkcDvWGVu17l37ggiCsGpx9wdG31d9mW4U3Ma3zJflx7I9be25zcY3HYzJWEglzAUgk6HU4qq6rM38B+P0kpHV1FImPjNB6Y6OpBOUURs4u4eZYoRC9NleZZgMROF92CBFbDgev/g5cD5yLB3tfiXOzx/DO2dvwzpHb8IDr9bi58btIJp15r93rDEJ12GBzmJ7rq4UIuCAIqwpHrBxtWwcEs7BbxZ3br1oHAHMJvPWCwHM3eZQZ2x+8cZhIkAjHYnS7tpaex0MlJidJ0KemyCvn3uDsf/NGp9drjtlZI+TJHSHkbDkoG6CRw+yeHF448zTsyTi+G/1j/EH6Ybw6dR9ePeLCIccf4M1NP8FsuhaTJwNoQA/+4uPL98ArIQIuCMKaUCka50gbMFE5R+LW+7mCk+0Ufg4PZ4jHTfofi7fWpqFVNGpGqmltSuQnJkjQQyEqCuILiNNphi/zhSKZNFN9xmaDmNZ0dVE5GzAWpFz4Wi+u9/wc2dk0vjZzA16bvAc9mV/g5EgdTtn34zXNvwZCQfy3F66ecDOyiSkIwppTbpOzOIOlXLYKj2vjzU67nSwNLpzhKJxtFe6nwuX13NCKo3bOfonHqbBnYoIslvFxUzXKzbK4b4vLRY+Px4GIPYKQPYTMUBDZ0UD+LwW+CGQyQC6Tw0cn34t3J7+S/2xnHbvQmTq97EoeyUIRBGFDKZdyyJWcC2WrsJBbS+05F5sbVXG6IHvh3CBLazPL0+2miJpFembGCHk4bIY1Fws5j4PjC0Y8TsLPvVi4fJ/7pHB73Zujn8GHZz5Mn//4Caj93cv67kTABUGoCspF48DCRUB8aF0o5FyAk82WCjl3EeTSfI78PR5TtMMl+JOTJiKPROi1ucdKJmMGLDscJvUwmTTdD1n42dpxOMwGaXN9CkeedC17yLFMpRcEoSpgm4QFm/3uYlulXLaK1R9na4VzxrnZlNtNgs0ZKrOzhf643W7yynmgsc1mOgc2NhohB8yGKkfWk5OmwjIWM+mJ4TDd5mEVfNFwu7GygZjzIAIuCMKGUGmTczHZKhyN22wk4izu6TQ9n/uQsJhzgY7HU2jJxOP0mjypvqWFxNjno1ayY2OUtcKl/dZJO5OT9Hp1dRSB80ZrOEy33W4j5F7vyqbSV2JBAVdK3Q7glQDGtNYXzq01ArgHwG4AfQDepLWOrP7pCYKwlSmXcgiU2irzZaswLN42m9nA5PzxWMxUYvKmJvcr5wZb1sKipiZT9DM9TWmHU1N0IWABdzrpuZGIEXKOwFMpI+Sc0ZLLYdkWSiUWE4H/E4CvArjDsnYLgIe01p9VSt0yd/uDq3tqgiBsF8rZJxyNL9ZWcTrpMdyfhC0WFnK32+SPz8wYf5wfx69v7WfS3GzmX05NUYVlNGrskWzW/IxEzICHWMykIbLlstriDSxCwLXWjyildhctvwbANXO/fxvAwxABFwRhBRQX+1ij7sXaKrmcyUzhlrQszpw/PjND61y8w1YLV4FaR6jxUVNDRUA+nxHyWKywPwtH5lNTpi84D2v2+80giNVkuR54q9Z6GAC01sNKqZZVPCdBELYxlTY5i9fL2SpWf5s9aH5uMmmm53D/cbfbZJN4veZCwSmEnBbodJro2uejg0vz+SLBeei82cniXV9Pz10L1nwTUyl1E4CbAKCrq2ut304QhC3CfJucC9kqLOJc8MMRMhf/sLCyVz47S4LO/nguZ3LQ+b2dTnptv5+iay7ND4XI73Y6zfvU1JhqUbZb1oLlvuyoUqp9LvpuBzBW6YFa668D+DpAeeDLfD9BELYhlfqqLCZbhR+by5G48tAGLpnn6Trc+pXTDjl/nKfOcx9yjq7ZH6+vJwH3emnTc2yMxJytFO6amEhQ5L4WLFfAHwDwVgCfnft5/6qdkSAIQhHz9VUBFm+rsChzOT73Ifd4zCanNX/cWvzDE32s0bjdTumGXOUZDJqqTs5GsdkoYl9t/xtYXBrhd0Eblk1KqUEAHwMJ971KqXcCGADwxtU/NUEQBMNCm5zlslj4OVZbhbsQsj9u3YSsrTVpfxyNc9tZnnlpndnJTbAAyljhronNzSb1kCP8tWAxWSh/WuGuF6/yuQiCICwID1GuJNbFAl/JVuFMk2Sy1B/ntEMut4/FTJTNgs0NtrjXCpfzNzcbeyaRIGuluE3AaiGVmIIgbDrmi8aXYqvwaDYus+cGWJzFwimEnH7ITapcLtMgixtZcZTORUJtbST+tbVrU4UJiIALgrCJmS/lcCm2Cmef8O3ijBJOJeRonO0X7qXCHQm5h7nTWTj+bS2KeAARcEEQtgCVNjkXa6twNM4bmlygA5Awc/MqLvrhboTceZArOrNZejy3unU6C/8qWG1EwAVB2BLM11dlsbYKYDxtm40ibs5c4TTCujpjq3BEzkJenHqYTpsIvpoqMQVBEKqShayTpdgqbrfxuTl/nAXe2vFwZobEOh43HQv5sbGY2TxdbUTABUHYcsy3yblYW4V7qORyZKuk06a6k/PIXS6zcRmLGSHnTU3ONecLh0TggiAIi2S+Tc6FbBXOSAFMgytOK2R/3Jp9wsMbEgmTsQKQ5cKbmquNCLggCFuehTY5K9kqnCrItgq3mwUKbRXOXPF6CwdJsJhLHrggCMIKmK+vylJtlWyWhFprU3LPAyXsdpOx4vFQ/3B+XbFQBEEQVsB80fhSbRWtKermnidcas+phTYbReOyiSkIgrBKzLfJyT8Xa6vwxiY3wbJ2PnS5zPPWgjVKLxcEQah+uGKT4dQ/wIg8iy974Hyfw2F6qHAFJvvfWptBD9Zxb6t+/mvzsoIgCJuHSkINlBd5q0fOUTbbL9yRkIc4OBwmI2W1EQEXBEHA/EINFFZ0soXC0Tq3lWUhZ8+7ttZE6GuBeOCCIAgW5tvkXKhAqDhbhScArVUzK4nABUEQimChZlZqq6xVHrgIuCAIQgWKOwlabZPi++ezVSQLRRAEYQOYLxtlofut2SprgQi4IAjCIlhok3O++yUCFwRBqALms00Wc/9qIgIuCIKwRMptclqj8XK2ylogaYSCIAjLZL6UQ6AwEl8LJAIXBEFYAQttcvJj1gIRcEEQhFVgoU3ONXnPtX15QRCE7YU1GmdbZa1YkQeulOoDEAWQBZDRWl++GiclCIKwmSk3ym0tyulXYxPzhVrriVV4HUEQhC0Fb3Ku1SamZKEIgiCsIWvZjXClHrgG8G9KqSNKqZtW44QEQRCExbHSCPxqrfWQUqoFwM+UUs9orR+xPmBO2G8CgK6urhW+nSAIgsCsKALXWg/N/RwD8AMAV5Z5zNe11pdrrS9vbm5eydsJgiAIFpYt4EqpWqWUj38H8DIAT63WiQmCIAjzsxILpRXADxS58w4A39Fa/2RVzkoQBEFYkGULuNa6F8BzVvFcBEEQhCUglZiCIAibFBFwQRCETYrSa1UiVO7NlBoH0L9ub1hKE4BqrxqVc1wd5BxXTrWfH7B9znGX1rokjW9dBXyjUUodrvZ+LXKOq4Oc48qp9vMD5BzFQhEEQdikiIALgiBsUrabgH99o09gEcg5rg5yjiun2s8P2ObnuK08cEEQhK3EdovABUEQtgwi4IIgCJuULS3gSqk+pdSTSqmjSqnDc2uNSqmfKaVOzP0MrPM53a6UGlNKPWVZq3hOSqkPKaVOKqWeVUr90Qad38eVUmfnvsejSqlXbNT5zb3nTqXUz5VSx5RSv1dK/dXcejV9j5XOsSq+S6VUjVLqUaXU43Pn97dz69X0HVY6x6r4DovO1a6Uekwp9eDc7fX5HrXWW/YA0AegqWjt7wHcMvf7LQA+t87n9IcALgXw1ELnBOB8AI8DcAPYA+AUAPsGnN/HAby/zGPX/fzm3rcdwKVzv/sAHJ87l2r6HiudY1V8lwAUgLq5350AfgOgp8q+w0rnWBXfYdF7vxfAdwA8OHd7Xb7HLR2BV+A1AL499/u3Abx2Pd9c08CL8CLP6TUAvqe1TmqtTwM4iTI919fh/Cqx7ucHAFrrYa317+Z+jwI4BqAT1fU9VjrHSqzrOWpiZu6mc+7QqK7vsNI5VmJD/n9USu0A8McAvll0Lmv+PW51AS838q1Vaz0M0D8yAC0bdnaGSufUCeCM5XGDmF8E1pKblVJPzFks/Ofghp+fUmo3gEtA0VlVfo9F5whUyXc592f/UQBjAH6mta6677DCOQJV8h3O8T8BfABAzrK2Lt/jVhfwq7XWlwJ4OYD/ppT6w40+oSVSbhTqRuR9fg3APgAHAQwD+OLc+oaen1KqDsB9AP5aaz0930PLrK3LeZY5x6r5LrXWWa31QQA7AFyplLpwnodvyHdY4Ryr5jtUSr0SwJjW+shin1JmbdnnuKUFXJcf+TaqlGoHgLmfYxt3hnkqndMggJ2Wx+0AMLTO5wat9ejcP6QcgG/A/Mm3YeenlHKChPFurfX355ar6nssd47V+F1qrScBPAzgWlTZd1juHKvsO7wawKuVUn0AvgfgRUqpu7BO3+OWFXBVeeTbAwDeOvewtwK4f2POsIBK5/QAgOuVUm6l1B4A+wE8ut4nx/8jznEdzOi8DTk/pZQC8C0Ax7TWX7LcVTXfY6VzrJbvUinVrJRqmPvdA+AlAJ5BdX2HZc+xWr5DANBaf0hrvUNrvRvA9QD+XWv9ZqzX97geO7QbcQDYC9rtfRzA7wHcOrceBPAQgBNzPxvX+by+C/qzLw26Gr9zvnMCcCtop/pZAC/foPO7E8CTAJ6Y+x+wfaPOb+49/wD0Z+cTAI7OHa+osu+x0jlWxXcJ4GIAj82dx1MAPjq3Xk3fYaVzrIrvsMz5XgOThbIu36OU0guCIGxStqyFIgiCsNURARcEQdikiIALgiBsUkTABUEQNiki4IIgCJsUEXBBEIRNigi4IAjCJuX/B3jcUMt7T7/+AAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "for m, b in res.bootstrap_distribution.T[::10]:\n", + " plt.plot(x, m*x + b, color='b', alpha=0.01)\n", + "plt.plot(x, y, '.', label='original data')\n", + "plt.plot(x, res_lr.intercept + res_lr.slope*x, 'r', label='fitted line')" + ] + }, + { + "cell_type": "markdown", + "id": "a38780c7-38c8-44fe-93be-7e5347e3aea4", + "metadata": {}, + "source": [ + "A major advantage of the bootstrap is that it can produce standard errors and confidence intervals even in more general regression models that have no simple analytical solutions, such as when the regression function is nonlinear in the parameters and when using fitting methods other than least squares." + ] + }, + { + "cell_type": "markdown", + "id": "1a886e39-f2b5-4ea4-8dff-110b7ddb5b45", + "metadata": {}, + "source": [ + "### Gotchas\n", + "\n", + "Our final example will show yet another application of the `bootstrap` chosen to illustrate common pitfalls.\n", + "\n", + "[An Introduction to the Bootstrap](https://books.google.com/books?id=MWC1DwAAQBAJ&printsec=frontcover) presents a study about whether regular doses of aspirin can prevent heart attacks. Subjects were randomly assigned two two groups: 11,037 received aspirin pills, and the remaining 11,034 received placebos. The subjects were instructed to take one pill every other day, and the scientists recorded the number of subject who experienced a heart attack during the study period: 104 in the aspirin group, and 189 in the placebo group. The statistic to assess the effectiveness of aspirin was the relative prevalence of heart attacks in the aspirin group versus the placebo group." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "f4153813-8a78-43c6-bd28-2cc6f89d7a33", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.5501149812103875" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "x = np.zeros(11037) # 11037 subjects in the aspirin group\n", + "x[:104] = 1 # 104 experience a heart attack\n", + "y = np.zeros(11034) # 11034 subjects in the placebo group\n", + "y[:189] = 1 # 189 experience a heart attack\n", + "def statistic(x, y):\n", + " return (np.sum(x)/len(x))/(np.sum(y)/len(y))\n", + "statistic(x, y)" + ] + }, + { + "cell_type": "markdown", + "id": "d1b65fcb-6f4b-480c-a163-085e272e3249", + "metadata": {}, + "source": [ + "The risk of heart atttack for aspirin-takers seemed to be approximately half that of placebo-takers.\n", + "\n", + "Suppose we wish to generate a 95% confidence interval to quantify our uncertainty." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "2b1bcd63-2a11-4643-b036-bce7121afb4f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "TypeError: bootstrap() takes 2 positional arguments but 3 positional arguments (and 1 keyword-only argument) were given\n" + ] + } + ], + "source": [ + "try:\n", + " stats.bootstrap(x, y, statistic, confidence_level=0.95)\n", + "except Exception as e:\n", + " print(f\"{type(e).__name__}: {e}\")" + ] + }, + { + "cell_type": "markdown", + "id": "f0b5caf3-3434-41a2-b8ea-da8d1ccfe679", + "metadata": {}, + "source": [ + "This reminds us that all the data needs to be passed in as a single sequence, not two separate arguments `x` and `y`." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "8d2738da-e79f-4ed3-968d-6533dadb7769", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "ValueError: `method = 'BCa' is only available for one-sample statistics\n" + ] + } + ], + "source": [ + "data = (x, y) \n", + "try:\n", + " stats.bootstrap(data, statistic, confidence_level=0.95)\n", + "except Exception as e:\n", + " print(f\"{type(e).__name__}: {e}\")" + ] + }, + { + "cell_type": "markdown", + "id": "aebd581b-d7df-4179-a6f1-a8d4feab9043", + "metadata": {}, + "source": [ + "`bootstrap` offers a `method` argument that selects how the confidence interval is to be estimated from the `bootstrap` distribution; the three methods `{'BCa', 'percentile', 'basic'}` vary in their performance and accuracy. `BCa` is the most computationally intensive but tends to be the most accurate, so it is the default. However, it is currently only available when our data has only one independent sample, whereas our data consists of two independent samples `x` and `y`. Let's try another option, `percentile`." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "33a84978-c534-4069-8f7e-3171614e0e20", + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " stats.bootstrap(data, statistic, method='basic', confidence_level=0.95)\n", + "except Exception as e:\n", + " print(f\"{type(e).__name__}: {e}\")" + ] + }, + { + "cell_type": "markdown", + "id": "47a2f909-b392-4738-ade4-4a80e07f7071", + "metadata": {}, + "source": [ + "Unlike `permutation_test`, `bootstrap` expects `statistic` to be vectorized by default. We can solve this by passing `vectorized=False`." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "15046441-1e6a-45c8-bdf6-2efa274f127c", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "ConfidenceInterval(low=0.4324405646124769, high=0.6926221426325354)" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "res = stats.bootstrap(data, statistic, method='percentile', confidence_level=0.95, vectorized=False)\n", + "res.confidence_interval" + ] + }, + { + "cell_type": "markdown", + "id": "09cc4689-4037-4ddb-a457-449015ed5256", + "metadata": {}, + "source": [ + "Alternatively, we can vectorize our statistic by making it accept a parameter `axis` and having it work along the specified axis-slice of N-dimensional arrays `x` and `y`." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "7588b203-8de2-4b52-9ced-64ad374c82a7", + "metadata": {}, + "outputs": [], + "source": [ + "def statistic(x, y, axis=0):\n", + " return (np.sum(x, axis=axis)/x.shape[axis])/(np.sum(y, axis=axis)/y.shape[axis])\n", + "\n", + "try:\n", + " res = stats.bootstrap(data, statistic, method='percentile', confidence_level=0.95)\n", + " res.confidence_interval\n", + "except Exception as e:\n", + " print(f\"{type(e).__name__}: {e}\")" + ] + }, + { + "cell_type": "markdown", + "id": "7800e793-92cb-4871-815b-06e3da89bb18", + "metadata": {}, + "source": [ + "Depending on your computer's hardware, you may run into a MemoryError there. Vectorized computations require a lot of memory. The default value of `n_resamples` is $9,999$, and there are a total of $11,037 + 11,034 = 22,071$ observations. Therefore, the resampled data arrays will contain a total of $9,999 \\cdot 22071 = 220,687,929$ elements. Each element is stored in double precision (8-bytes), so at least 1.7GB will be used during the calculation. To relax the memory requirement, we'll process the data in batches of 100 resamples rather than all at once." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "f6769306-af22-42b3-850b-377e40e3b703", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "ConfidenceInterval(low=0.4309040629913994, high=0.6944371938603752)" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "res = stats.bootstrap(data, statistic, method='percentile', confidence_level=0.95, batch=100)\n", + "res.confidence_interval" + ] + }, + { + "cell_type": "markdown", + "id": "19a3d153-fb2f-4f32-aa10-a347799c33e9", + "metadata": { + "tags": [] + }, + "source": [ + "## Conclusion\n", + "\n", + "The resampling approaches in SciPy can be used not only to replicate the results of most of SciPy's hypothesis tests, but to\n", + "\n", + "- improve the accuracy of statistical tests for small sample sizes and in the presence of ties,\n", + "- provide standard errors and confidence intervals for arbitrary statistics, and\n", + "- easily implement statistical tests that SciPy does not yet offer." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.5" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}