Statistics/How_to_Use_Statistical_Significance_Tests_to_Interpret_ML_Results.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to Use Statistical Significance Tests to Interpret Machine Learning Results\n",
    "It is good practice to gather a population of results when comparing two different machine learning algorithms or when comparing the same algorithm with different configurations.\n",
    "\n",
    "Repeating each experimental run 30 or more times gives you a population of results from which you can calculate the mean expected performance, given the stochastic nature of most machine learning algorithms.\n",
    "\n",
    "If the mean expected performance from two algorithms or configurations are different, how do you know that the difference is significant, and how significant?\n",
    "\n",
    "Statistical significance tests are an important tool to help to interpret the results from machine learning experiments. Additionally, the findings from these tools can help you better and more confidently present your experimental results and choose the right algorithms and configurations for your predictive modeling problem.\n",
    "\n",
    "In this tutorial, you will discover how you can investigate and interpret machine learning experimental results using statistical significance tests in Python.\n",
    "\n",
    "After completing this tutorial, you will know:\n",
    "\n",
    "* How to apply normality tests to confirm that your data is (or is not) normally distributed.\n",
    "* How to apply parametric statistical significance tests for normally distributed results.\n",
    "* How to apply nonparametric statistical significance tests for more complex distributions of results.\n",
    "\n",
    "Let’s get started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial Overview\n",
    "This tutorial is broken down into 6 parts. They are:\n",
    "\n",
    "1. Generate Sample Data\n",
    "2. Summary Statistics\n",
    "3. Normality Test\n",
    "4. Compare Means for Gaussian Results\n",
    "5. Compare Means for Gaussian Results with Different Variance\n",
    "6. Compare Means for Non-Gaussian Results\n",
    "\n",
    "This tutorial assumes Python 2 or 3 and a SciPy environment with NumPy, Pandas, and Matplotlib.\n",
    "\n",
    "## Generate Sample Data\n",
    "The situation is that you have experimental results from two algorithms or two different configurations of the same algorithm.\n",
    "\n",
    "Each algorithm has been trialed multiple times on the test dataset and a skill score has been collected. We are left with two populations of skill scores.\n",
    "\n",
    "We can simulate this by generating two populations of Gaussian random numbers distributed around slightly different means.\n",
    "\n",
    "The code below generates the results from the first algorithm. A total of 1000 results are stored in a file named results1.csv. The results are drawn from a Gaussian distribution with the mean of 50 and the standard deviation of 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from numpy.random import seed\n",
    "from numpy.random import normal\n",
    "from numpy import savetxt\n",
    "# define underlying distribution of results\n",
    "mean = 50\n",
    "stev = 10\n",
    "# generate samples from ideal distribution\n",
    "seed(1)\n",
    "results = normal(mean, stev, 1000)\n",
    "# save to ASCII file\n",
    "savetxt('results1.csv', results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now generate the results for the second algorithm. We will use the same method and draw the results from a slightly different Gaussian distribution (mean of 60 with the same standard deviation). Results are written to results2.csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from numpy.random import seed\n",
    "from numpy.random import normal\n",
    "from numpy import savetxt\n",
    "# define underlying distribution of results\n",
    "mean = 60\n",
    "stev = 10\n",
    "# generate samples from ideal distribution\n",
    "seed(1)\n",
    "results = normal(mean, stev, 1000)\n",
    "# save to ASCII file\n",
    "savetxt('results2.csv', results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Going forward, we will pretend that we don’t know the underlying distribution of either set of results.\n",
    "\n",
    "I chose populations of 1000 results per experiment arbitrarily. It is more realistic to use populations of 30 or 100 results to achieve a suitably good estimate (e.g. low standard error).\n",
    "\n",
    "Don’t worry if your results are not Gaussian; we will look at how the methods break down for non-Gaussian data and what alternate methods to use instead.\n",
    "\n",
    "## Summary Statistics\n",
    "The first step after collecting results is to review some summary statistics and learn more about the distribution of the data.\n",
    "\n",
    "This includes reviewing summary statistics and plots of the data.\n",
    "\n",
    "Below is a complete code listing to review some summary statistics for the two sets of results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                 A            B\n",
      "count  1000.000000  1000.000000\n",
      "mean     50.388125    60.388125\n",
      "std       9.814950     9.814950\n",
      "min      19.462356    29.462356\n",
      "25%      43.998396    53.998396\n",
      "50%      50.412926    60.412926\n",
      "75%      57.039989    67.039989\n",
      "max      89.586027    99.586027\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAFh1JREFUeJzt3X+MXeWd3/H3l7FhsM3ye6cjiBkkUDIWBJsdoWzDIhvHCZtUC6oCyrRqzWaoZVHGpPyBnRmpbKSa2lLVNAXJqVnTeKsyhLIbGXBlY+y53c4fpWsbEhFmt7AssEYGZ1lgM8B47dG3f8wx61+DPffc8R2feb+k0b3nuefc89X18Weeee5zzonMRJJUXec0uwBJ0tQy6CWp4gx6Sao4g16SKs6gl6SKM+glqeIMekmqOINekirOoJekipvV7AIALrvssuzo6Gh2GZXx8ccfM3fu3GaXIZ3AY7Ox9uzZ8zeZefmp1psWQd/R0cHu3bubXUZl1Go1Fi9e3OwypBN4bDZWRLx1Ous5dCNJFWfQS1LFGfSSVHEGvSRV3CmDPiIej4gDEfHKUW2XRMSOiHiteLy4aI+I+M8R8XpE/CIibpzK4iVJp3Y6PfqfALcd17YG2JmZ1wI7i2WA3wWuLX5WABsaU6aks9nAwADXXXcdS5cu5brrrmNgYKDZJc0op5xemZl/GhEdxzXfDiwunm8GasDqov2Pcvy2Vf8nIi6KiPbM3N+ogiWdXQYGBujv72fTpk2MjY3R0tJCT08PAN3d3U2ubmaodx5921Hh/S7QVjy/Avjro9bbV7SdEPQRsYLxXj9tbW3UarU6S9HxRkZG/Dw1bfT19bFq1SoigtHRUebNm0dvby99fX20t7c3u7wZIU7nnrFFj/65zLyuWP4wMy866vUPMvPiiHgOWJeZQ0X7TmB1Zn7u2VBdXV3pCVON40kpmk5aWloYHR1l9uzZnx2bhw4dorW1lbGxsWaXd1aLiD2Z2XWq9eqddfNeRLQXO2oHDhTt7wBfOGq9K4s2STNUZ2cnQ0NDx7QNDQ3R2dnZpIpmnnqD/hlgefF8ObDlqPZ/Wcy++QrwkePz0szW399PT08Pg4ODHD58mMHBQXp6eujv7292aTPGKcfoI2KA8S9eL4uIfcBDwDrgqYjoAd4C7ipW/5/AN4HXgU+A35+CmiWdRY584drb28vw8DCdnZ2sXbvWL2LPoNOZdTPRv8bSk6ybwL8uW5Skaunu7qa7u9vvj5rEM2MlqeIMekmqOINekirOoJekijPoJaniDHpJqjiDXpIqzqCXpIoz6CVNOa9H31z1XqZYkk6L16NvPnv0kqbU2rVr2bRpE0uWLGHWrFksWbKETZs2sXbt2maXNmMY9JKm1PDwMDfffPMxbTfffDPDw8NNqmjmMeglTSmvR998Br2kKeX16JvPL2MlTSmvR998pXr0EXF/RLwSEb+MiO8VbZdExI6IeK14vLgxpUqS6lF3jz4irgP+FXAT8PfAtuLm4CuAnZm5LiLWAGuA1Y0oVtLZx+mVzVemR98JvJiZn2TmYeB/Af8UuB3YXKyzGbijXIk6XZ6UounI6ZXNV2aM/hVgbURcCnzK+L1idwNtR90Q/F2grVyJOh32mjRdOb2y+eoO+swcjoj1wPPAx8DLwNhx62RE5Mm2j4gVjA/z0NbWRq1Wq7cUAX19faxatYqIYHR0lHnz5tHb20tfXx/t7e3NLk8z2Pz583n00UdZtGgRIyMj1Go1XnrpJebPn+//+zMkxu/n3YA3ingY2AfcDyzOzP0R0Q7UMvOLn7dtV1dX7t69uyF1zFQtLS2Mjo4ye/bsz27AfOjQIVpbWxkbGzv1G0hTZKK/Np15U15E7MnMrlOtV2p6ZUT8ZmYeiIj5jI/PfwW4GlgOrCset5TZh07PkZNSlixZ8lmbJ6VoOnB6ZfOV6tFHxP8GLgUOAQ9k5s5izP4pYD7wFnBXZv7t572PPfry7DXpbHDkr001xhnp0Wfm75yk7X1gaZn31eTZa5I0Ec+MrZDu7m66u7vtNUk6hte6kaSKM+glqeIMekmqOINekirOoJekijPoJaninF4pqeEiYtLbNOpyLDqRPXpJDZeZJ/25avVzE76mqWPQV4jXo5d0Mg7dVITXo5c0EXv0FeFdfCRNxKCvCO/iI2kiBn1FHLke/dG8Hr0kMOgro7+/n56eHgYHBzl8+DCDg4P09PTQ39/f7NIkNZlfxlaE16OXNJFSPfqI+DcR8cuIeCUiBiKiNSKujogXI+L1iPhpRJzbqGIlSZNXd48+Iq4AVgELMvPTiHgK+A7wTeCHmflkRPwY6AE2NKRaTcjplZImUnaMfhZwfkTMAuYA+4FbgaeL1zcDd5Tch06D0yslTaTuoM/Md4D/ALzNeMB/BOwBPszMw8Vq+4AryhapU3N6paSJlBm6uRi4Hbga+BD4H8Btk9h+BbACoK2tjVqtVm8pAubPn8+jjz7KokWLGBkZoVar8dJLLzF//nw/W00rHo9nXplZN18D/iozfwUQEX8CfBW4KCJmFb36K4F3TrZxZm4ENgJ0dXWlN7Mu5+GHH/5sjL61tZXM5JFHHuHhhx/2RuGaPrZt9XhsgjJB/zbwlYiYA3wKLAV2A4PAt4EngeXAlrJF6tScXilpInUHfWa+GBFPA3uBw8BLjPfQtwJPRsS/K9o2NaJQnVp3dzfd3d3UajV7TZI+U+qEqcx8CHjouOY3gJvKvK8kqXG8BIIkVZxBL0kVZ9BLUsUZ9JJUcQa9JFWcQS9JFef16M9SEVHXdpnZ4EokTXf26M9SmTnhz1Wrn5vwNUkzj0EvSRVn0EtSxRn0klRxBr0kVZxBL0kVZ9BLUsUZ9JJUcQa9JFVc3UEfEV+MiJeP+vm7iPheRFwSETsi4rXi8eJGFixJmpy6gz4z/yIzF2bmQuC3gE+AnwFrgJ2ZeS2ws1iWJDVJo4ZulgJ/mZlvAbcDm4v2zcAdDdqHJKkOjbqo2XeAgeJ5W2buL56/C7SdbIOIWAGsAGhra6NWqzWoFAF+npq2PDbPvNJBHxHnAr8HfP/41zIzI+KkV9LKzI3ARoCurq5cvHhx2VJ0xLat+HlqWvLYbIpGDN38LrA3M98rlt+LiHaA4vFAA/YhSapTI4K+m38YtgF4BlhePF8ObGnAPiRJdSoV9BExF1gG/MlRzeuAZRHxGvC1YlmS1CSlxugz82Pg0uPa3md8Fo4kaRrwzFhJqjiDXpIqzpuDS6rbDT94no8+PTSpbTrWbJ3U+heeP5ufP/T1SW2jYxn0kur20aeHeHPdt057/VqtNul59JP9xaATOXQjSRVn0EtSxRn0klRxBr0kVZxBL0kVZ9BLUsUZ9JJUcQa9JFWcQS9JFWfQS1LFGfSSVHFlbzxyUUQ8HRF/HhHDEfHbEXFJROyIiNeKx4sbVawkafLK9uh/BGzLzC8BNwDDwBpgZ2ZeC+wsliVJTVJ30EfEhcAtwCaAzPz7zPwQuB3YXKy2GbijbJGSpPqVuUzx1cCvgP8aETcAe4D7gbbM3F+s8y7QVq5ESdPVBZ1ruH7zJP9o33zqVY7dB8DpXwpZJyoT9LOAG4HezHwxIn7EccM0mZkRkSfbOCJWACsA2traqNVqJUrR8fw8dSb8engdP7lt7mmvPzIywrx58ya1j7u3fezxXFKZoN8H7MvMF4vlpxkP+vcioj0z90dEO3DgZBtn5kZgI0BXV1dO9mYE+hzbtk765g5SXSZ5rNVz4xGP5/LqHqPPzHeBv46ILxZNS4FXgWeA5UXbcmBLqQolSaWUvZVgL/DfI+Jc4A3g9xn/5fFURPQAbwF3ldyHJKmEUkGfmS8DXSd5aWmZ95UkNY5nxkpSxRn0klRxZcfoNcVu+MHzfPTpoUlv17Fm62mve+H5s/n5Q1+f9D4knR0M+mnuo08P8ea6yZ0sMtkpbJP5pSDp7OPQjSRVnEEvSRVn0EtSxRn0klRxBr0kVZxBL0kV5/RKSaVMenrutsmtf+H5syf3/jqBQS+pbpM9x6NjzdZJb6PyHLqRpIoz6CWp4gx6Sao4g16SKq7Ul7ER8Sbwa2AMOJyZXRFxCfBToAN4E7grMz8oV6YkqV6N6NEvycyFmXnkTlNrgJ2ZeS2ws1iWJDXJVAzd3A5sLp5vBu6Ygn1Ikk5T2Xn0CTwfEQn8l8zcCLRl5v7i9XeBtpL7mNEu6FzD9Zvr+KNo86lX+Yd9ADi3WaqqskF/c2a+ExG/CeyIiD8/+sXMzOKXwAkiYgWwAqCtrY1arVaylGr69fA6fnLb3EltMzIywrx58057/bu3feznrzPGY+3MKxX0mflO8XggIn4G3AS8FxHtmbk/ItqBAxNsuxHYCNDV1ZWTuSPSjLJt66TuFgWTv8NUPfuQ6uKx1hR1j9FHxNyIuODIc+DrwCvAM8DyYrXlwJayRUqS6lemR98G/CwijrzPE5m5LSL+DHgqInqAt4C7ypcpSapX3UGfmW8AN5yk/X1gaZmiJEmN45mxklRxBr0kVZxBL0kVZ9BLUsUZ9JJUcQa9JFWcQS9JFWfQS1LFGfSSVHFlr16pM6BjzdbJb7Tt9Le58PzZk39/SWcNg36ae3Pd5K8T37Fma13bSaomh24kqeIMekmqOINekirOoJekijPoJaniSgd9RLRExEsR8VyxfHVEvBgRr0fETyPi3PJlSpLq1Yge/f3A8FHL64EfZuY1wAdATwP2IUmqU6mgj4grgW8Bf1gsB3Ar8HSxymbgjjL7kCSVU/aEqf8EPAhcUCxfCnyYmYeL5X3AFSfbMCJWACsA2traqNVqJUvR0fw8NV15bJ55dQd9RPwT4EBm7omIxZPdPjM3AhsBurq6cvHiSb+FJrJtK36empY8NpuiTI/+q8DvRcQ3gVbgN4AfARdFxKyiV38l8E75MiVJ9ap7jD4zv5+ZV2ZmB/AdYFdm/nNgEPh2sdpyYEvpKiVJdZuKefSrgQci4nXGx+w3TcE+JEmnqSFXr8zMGlArnr8B3NSI95UkleeZsZJUcQa9JFWcQS9JFWfQS1LFGfSSVHEGvSRVnEEvSRXXkHn0knS08QvZTvDa+pO3Z+YUVSN79JIaLjOP+bnvvvs477zzADjvvPO47777TlhHU8cevaQp1dvby49//GPWr1/PggULePXVV1m9ejUAjzzySJOrmxns0UuaUo899hjr16/ngQceoLW1lQceeID169fz2GOPNbu0GcOglzSlDh48yNatWznnnHNYsmQJ55xzDlu3buXgwYPNLm3GMOglTamIYNeuXaxcuZJnn32WlStXsmvXrs/9wlaNZdBLmlKZSURwzTXXMGvWLK655hoiwi9gzyC/jJU05e655x76+vo4ePAg5513Hvfcc49j9GdQ3T36iGiNiP8bET+PiF9GxA+K9qsj4sWIeD0ifhoR5zauXElnm4hg1qxZjI6OMjg4yOjoKLNmzXLo5gwqM3RzELg1M28AFgK3RcRXgPXADzPzGuADoKd8mZLOVsuWLWPDhg3ce++9jIyMcO+997JhwwaWLVvW7NJmjLqHbnJ8gG2kWJxd/CRwK/DPivbNwB8AG+ovUdLZbPv27Xz5y19mw4YNbNgwHgXXX38927dvb3JlM0epL2MjoiUiXgYOADuAvwQ+zMzDxSr7gCvKlSjpbDYwMMDIyAi7du1ix44d7Nq1i5GREQYGBppd2oxR6svYzBwDFkbERcDPgC+d7rYRsQJYAdDW1katVitTio7j56npoq+vj1WrVhERjI6OMm/ePHp7e+nr66O9vb3Z5c0I0agpThHxb4FPgdXAP8rMwxHx28AfZOY3Pm/brq6u3L17d0PqEHSs2cqb677V7DIkAFpaWhgdHWX27NnUajUWL17MoUOHaG1tZWxsrNnlndUiYk9mdp1qvTKzbi4vevJExPnAMmAYGAS+Xay2HNhS7z4knf06OzsZGho6pm1oaIjOzs4mVTTzlBmjbwcGI+IXwJ8BOzLzOcZ79A9ExOvApcCm8mVKOlv19/fT09PD4OAghw8fZnBwkJ6eHvr7+5td2oxRZtbNL4BFJ2l/A7ipTFGSqqO7uxsYv4rl8PAwnZ2drF279rN2TT3PjJU05bq7u+nu7v5sjF5nlte6kaSKM+glqeIMekmqOINekirOoJekijPoJaniDHpJqjjn0Z+lTnXThlh/8nZv3ybNPPboz1KZeczPE088weWXX05HRwcRQUdHB5dffjlPPPHEMetJmnkM+op48MEHaWlp4fHHH+f555/n8ccfp6WlhQcffLDZpUlqModuKmLfvn3ceOONLF26lMwkIli0aBF79+5tdmmSmswefYXs3buXlStX8uyzz7Jy5UpDXhJg0FfKnDlzuPPOO2ltbeXOO+9kzpw5zS5J0jTg0E2FzJkzh+9+97u8/fbbzJ8/nzlz5vDJJ580uyxJTWaPviIigoULFzJ37lwA5s6dy8KFC085DVNS9ZW5leAXImIwIl6NiF9GxP1F+yURsSMiXiseL25cuZrIsmXLeOGFF7jlllvYsmULt9xyCy+88ALLli1rdmmSmqzum4NHRDvQnpl7I+ICYA9wB3A38LeZuS4i1gAXZ+bqz3svbw7eGN/4xjfYsWPHZ7Nuli1bxvbt25tdlvQZbzzSWFN+c/DM3J+Ze4vnv2b8xuBXALcDm4vVNjMe/joD7r77bhYsWMA555zDggULuPvuu5tdkqRpoCFfxkZEB+P3j30RaMvM/cVL7wJtjdiHPt/AwAD9/f1s2rSJsbExWlpa6OnpAfDenNIMVzroI2Ie8MfA9zLz747+8i8zMyJOOjYUESuAFQBtbW3UarWypcxofX19rFq1iohgdHSUefPm0dvbS19fH+3t7c0uTwJgZGTE/+tNUPcYPUBEzAaeA7Zn5n8s2v4CWJyZ+4tx/FpmfvHz3scx+vJaWloYHR1l9uzZn42DHjp0iNbWVsbGxppdngQ4Rt9oUz5GH+Nd903A8JGQLzwDLC+eLwe21LsPnb7Ozk6GhoaOaRsaGqKzs7NJFUmaLsrMo/8q8C+AWyPi5eLnm8A6YFlEvAZ8rVjWFOvv76enp4fBwUEOHz7M4OAgPT099Pf3N7s0SU1W9xh9Zg4BE52Ns7Te91V9jnzh2tvby/DwMJ2dnaxdu9YvYiV5CYQq6e7upru723FQScfwEgiSVHEGvSRVnEEvSRVn0EtSxRn0klRxpc6MbVgREb8C3mp2HRVyGfA3zS5COgmPzca6KjMvP9VK0yLo1VgRsft0TouWzjSPzeZw6EaSKs6gl6SKM+iraWOzC5Am4LHZBI7RS1LF2aOXpIoz6CskIu6IiIyILzW7FuloETFWXMr85xGxNyL+cbNrmkkM+mrpBoaKR2k6+TQzF2bmDcD3gX/f7IJmEoO+Iop7994M9ADfaXI50uf5DeCDZhcxk3g9+uq4HdiWmf8vIt6PiN/KzD3NLkoqnB8RLwOtQDtwa5PrmVHs0VdHN/Bk8fxJHL7R9HJk6OZLwG3AHxX3ndYZ4PTKCoiIS4B9wK+ABFqKx6vSf2BNAxExkpnzjlp+D7g+Mw80sawZwx59NXwb+G+ZeVVmdmTmF4C/An6nyXVJJyhmhbUA7ze7lpnCMfpq6AbWH9f2x0X7n575cqQTHBmjBwhgeWaONbOgmcShG0mqOIduJKniDHpJqjiDXpIqzqCXpIoz6CWp4gx6Sao4g16SKs6gl6SK+/+yG1euiOdvcwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x107ee6c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAFQJJREFUeJzt3X+wXHV5x/H3U4KIxBEQu40hY+iYatEMKHcQRzu9kVoRW4MzlGIZCUob/8BWbaZt9I+KY5niDJFqtUxjsUSrRIooGaAqxtxa/gBMlJIAMkYNkkwIUiN68cd449M/9kQXvD927/44537zfs3s3N3vOWf3yXef+8nZs2fvRmYiSSrXb9RdgCRpuAx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVDiDfgGIiImIOBgRx9RdizRMEbEnIn4SEZNVz98aEcvqrmuhM+gbLiKWA78HJPD6WouRRuOPM3MxsAQ4APxzzfUseAZ9810M3AlcB6yptxRpdDLzp8CNwKl117LQLaq7AM3pYuADwF3AnRHRyswDNdckDV1EPAP4U9o7OuqDQd9gEfFK4HnADZn5WER8C/gz4Op6K5OG6nMRMQUcB3wPeE3N9Sx4HrpptjXAFzPzser2p/Dwjcp3XmYeDzwdeBvw3xHxWzXXtKAZ9A0VEccCFwC/HxGPRMQjwDuB0yLitHqrk4YvMw9l5k3AIeCVddezkBn0zXUe7QY/FTi9uvwu8D+0j9tLRYu21cAJwAN117OQhX+Pvpki4vPAfZm57injFwAfAk7OzKlaipOGJCL2AC3aOzkJPAT8Y2Z+ss66FjqDXpIK56EbSSrcnEEfEU+PiLsj4n8j4r6IeG81fkpE3BURuyPi0xHxtGr8mOr27mr58uH+E6TBsudVmm726H8GvCozT6P9huA5EXEW8H7g6sx8PnAQuLRa/1LgYDV+dbWetJDY8yrKnEGfbZPVzaOrSwKvov3xZIBNtM8SAVhd3aZafnZExMAqlobMnldpuvpkbEQcBewAng98BPgW8IOOsz72Akur60uBhwEycyoiHgeeDTzGDE466aRcvnz5jI//xBNPcNxxx3VT6kg1sa4m1gSjqWvHjh2PZeZzBnFfdff8IB3JPdGrJtYEM9fVbc93FfSZeQg4PSKOBz4LvLDXQp8qItYCawFarRZXXXXVjOtOTk6yePHifh9y4JpYVxNrgtHUtWrVqocGdV919/wgHck90asm1gQz19V1z2dmTxfg74G/ob23sqgaeznwher6F4CXV9cXVevFbPd5xhln5Gy2bds26/K6NLGuJtaUOZq6gO3ZYz93c6mj5wfpSO6JXjWxpsyZ6+q257s56+Y51V7N4Y/lv5r2p9S2AedXq60Bbq6ub+FXf4/lfODLVUHSgmDPqzTdHLpZAmyqjln+Bu2/pHhLRNwPbI6IfwC+DlxbrX8t8ImI2A18H7hwCHVLw2TPqyhzBn1m3gu8ZJrxbwNnTjP+U+BPBlKdVAN7XqXxk7GSVDiDXpIKZ9BLUuEMekkqnEEvSYXzy8FrsHz9rT1vs+fK1w2hEmk0puv5dSunuGSW3wV7fnDco5ekwhn0klQ4g16SCmfQS1LhDHpJKpxBL0mFM+glqXAGvSQVzqCXpMIZ9JJUOINekgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVDiDXpIKZ9BLUuEMekkqnEEvSYWbM+gjYllEbIuI+yPivoh4ezV+eUTsi4h7qsu5Hdu8KyJ2R8SDEfGaYf4DpEGz51WaRV2sMwWsy8yvRcQzgR0RcXu17OrMvKpz5Yg4FbgQeBHwXOBLEfE7mXlokIVLQ2TPqyhz7tFn5v7M/Fp1/UfAA8DSWTZZDWzOzJ9l5neA3cCZgyhWGgV7XqWJzOx+5YjlwFeAFwN/DVwC/BDYTnsP6GBEfBi4MzP/o9rmWuC/MvPGp9zXWmAtQKvVOmPz5s0zPu7k5CSLFy/uus5RmW9dO/c93vM2K5c+q6v1SpurXqxatWpHZo4N8j7r6vlBakJPTNfzrWPhwE9m3qbbnh+kJszVdGaqq9ue7+bQDQARsRj4DPCOzPxhRFwDvA/I6ucG4C3d3l9mbgQ2AoyNjeX4+PiM605MTDDb8rrMt65L1t/a8zZ7LurucUqbqzrV2fOD1IS5n67n162cYsPOmSOo254fpCbM1XT6raurs24i4mjaDf/JzLwJIDMPZOahzPwF8FF+9VJ1H7CsY/OTqzFpwbDnVZJuzroJ4Frggcz8QMf4ko7V3gDsqq5vAS6MiGMi4hRgBXD34EqWhsueV2m6OXTzCuBNwM6IuKcaezfwxog4nfbL2D3AWwEy876IuAG4n/bZC5d59oEWGHteRZkz6DPzDiCmWXTbLNtcAVzRR11Sbex5lcZPxkpS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVDiDXpIKZ9BLUuEMekkqnEEvSYUz6CWpcAa9JBWu62+Y0vSWz+Pboob5OOtWTv3y23z2XPm6YZakI1TTer6TPT899+glqXAGvSQVzqCXpMIZ9JJUOINekgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVDiDXpIKN2fQR8SyiNgWEfdHxH0R8fZq/MSIuD0ivln9PKEaj4j4UETsjoh7I+Klw/5HSINkz6s03ezRTwHrMvNU4Czgsog4FVgPbM3MFcDW6jbAa4EV1WUtcM3Aq5aGy55XUeYM+szcn5lfq67/CHgAWAqsBjZVq20CzquurwY+nm13AsdHxJKBVy4NiT2v0vT0xSMRsRx4CXAX0MrM/dWiR4BWdX0p8HDHZnursf0dY0TEWtp7P7RaLSYmJmZ83MnJyVmX12VycpJ1Kw/VXcaTtI5tf/kI0Kg5a+pzOJe6en6QBj33h/urX529Oij9/jub2qf91tV10EfEYuAzwDsy84cR8ctlmZkRkb08cGZuBDYCjI2N5fj4+IzrTkxMMNvyukxMTLDhjifqLuNJ1q2cYsPO9tO656Lxeovp0NTncDZ19vwgDXruLxnQN0x19uqg9NvzTe3Tfuvq6qybiDiadsN/MjNvqoYPHH55Wv18tBrfByzr2PzkakxaMOx5laSbs24CuBZ4IDM/0LFoC7Cmur4GuLlj/OLqTISzgMc7Xu5KjWfPqzTdvG56BfAmYGdE3FONvRu4ErghIi4FHgIuqJbdBpwL7AZ+DLx5oBVLw2fPqyhzBn1m3gHEDIvPnmb9BC7rsy6pNva8SuMnYyWpcAa9JBXOoJekwhn0klQ4g16SCmfQS1LhDHpJKpxBL0mFM+glqXAGvSQVzqCXpMIZ9JJUOINekgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVDiDXpIKZ9BLUuEMekkqnEEvSYUz6CWpcAa9JBXOoJekws0Z9BHxsYh4NCJ2dYxdHhH7IuKe6nJux7J3RcTuiHgwIl4zrMKlYbLvVZJu9uivA86ZZvzqzDy9utwGEBGnAhcCL6q2+ZeIOGpQxUojdB32vQoxZ9Bn5leA73d5f6uBzZn5s8z8DrAbOLOP+qRa2PcqyaI+tn1bRFwMbAfWZeZBYClwZ8c6e6uxXxMRa4G1AK1Wi4mJiRkfaHJyctbldZmcnGTdykN1l/EkrWNh3copgEbNWVOfw3mYd9/30vODNOi5P9xf/ers1UHp99/Z1D7tt675Bv01wPuArH5uAN7Syx1k5kZgI8DY2FiOj4/PuO7ExASzLa/LxMQEG+54ou4ynmTdyik27Gw/rXsuGq+3mA5NfQ571Fff99LzgzToub9k/a0DuZ/OXh2Ufnu+qX3ab13zOusmMw9k5qHM/AXwUX71MnUfsKxj1ZOrMWnBs++1UM0r6CNiScfNNwCHz0zYAlwYEcdExCnACuDu/kqUmsG+10I15+umiLgeGAdOioi9wHuA8Yg4nfZL2D3AWwEy876IuAG4H5gCLsvMZh3Elrpg36skcwZ9Zr5xmuFrZ1n/CuCKfoqS6mbfqyR+MlaSCmfQS1LhDHpJKpxBL0mFM+glqXAGvSQVzqCXpMIZ9JJUOINekgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVDiDXpIKZ9BLUuEMekkqnEEvSYUz6CWpcAa9JBXOoJekwhn0klQ4g16SCmfQS1LhDHpJKtycQR8RH4uIRyNiV8fYiRFxe0R8s/p5QjUeEfGhiNgdEfdGxEuHWbw0LPa9SrKoi3WuAz4MfLxjbD2wNTOvjIj11e2/A14LrKguLwOuqX423vL1t/a8zbqVU3Q3hVqArsO+VyHm3KPPzK8A33/K8GpgU3V9E3Bex/jHs+1O4PiIWDKoYqVRse9VkvnujrYyc391/RGgVV1fCjzcsd7eamw/TxERa4G1AK1Wi4mJiRkfbHJyctblg9DeO+9N69j5bTdMnTUNe856MYrncAT66vteen6QZpv7Ovt3GL8//c5pU/u037r6Pu6QmRkROY/tNgIbAcbGxnJ8fHzGdScmJpht+SBcMs9DNxt2NuvQTWdNey4ar7eYDqN4DkdpPn3fS88P0mxzP5++H5Rh/P702/NN7dN+65rvWTcHDr80rX4+Wo3vA5Z1rHdyNSaVwL7XgjTfoN8CrKmurwFu7hi/uDoL4Szg8Y6XutJCZ99rQZrzdVNEXA+MAydFxF7gPcCVwA0RcSnwEHBBtfptwLnAbuDHwJuHULM0dPa9SjJn0GfmG2dYdPY06yZwWb9FSXWz71USPxkrSYUz6CWpcM06N1ADNZ9PPe658nVDqEQajfl+0rf0vnePXpIKZ9BLUuEMekkqnEEvSYUz6CWpcAa9JBXOoJekwhn0klQ4g16SCmfQS1LhDHpJKpxBL0mFM+glqXAGvSQVzqCXpMIZ9JJUOINekgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVLhF/WwcEXuAHwGHgKnMHIuIE4FPA8uBPcAFmXmwvzKl5rDvtdAMYo9+VWaenplj1e31wNbMXAFsrW5LpbHvtWAM49DNamBTdX0TcN4QHkNqGvtejdVv0CfwxYjYERFrq7FWZu6vrj8CtPp8DKlp7HstKJGZ8984Ymlm7ouI3wRuB/4S2JKZx3esczAzT5hm27XAWoBWq3XG5s2bZ3ycyclJFi9ePO86u7Fz3+M9b9M6Fg78ZAjF9KHfmlYufdbgiukwiudw1apVOzoOpQzNfPu+l54fpNnmfj59PyhN+v053Pej6NP5mKmubnu+r6B/0h1FXA5MAn8BjGfm/ohYAkxk5gtm23ZsbCy3b98+4/KJiQnGx8cHUudMlq+/tedt1q2cYsPOvt7PHrh+a9pz5esGWM2vjOI5jIiRBP1THvNy5tH3c/X8IM029/Pp+0Fp0u/P4b4fRZ/Ox0x1ddvz8z50ExHHRcQzD18H/hDYBWwB1lSrrQFunu9jSE1j32sh6ue/0xbw2Yg4fD+fyszPR8RXgRsi4lLgIeCC/suUGsO+14Iz76DPzG8Dp00z/n/A2f0UJTWVfa+FyE/GSlLhDHpJKpxBL0mFM+glqXDNOIl1wOo8N1iqw2w9v27lFJf4O3FEc49ekgpn0EtS4Qx6SSqcQS9JhTPoJalwBr0kFc6gl6TCGfSSVLgiPzCl+Zvvh82G9YUl0igc7vtePly2kHrePXpJKpxBL0mFM+glqXAGvSQVzqCXpMIZ9JJUOINekgpn0EtS4Qx6SSqcQS9JhWv8n0BYvv5Wv/NSRxS/81iD5h69JBXOoJekwhn0klS4oQV9RJwTEQ9GxO6IWD+sx5Gawp5XUw3lzdiIOAr4CPBqYC/w1YjYkpn3D+PxtDDN503Hpv4NcHte3air54d11s2ZwO7M/DZARGwGVgM2faHmauAj4Mwpe/4Is5DOjorMHPydRpwPnJOZf17dfhPwssx8W8c6a4G11c0XAA/OcpcnAY8NvND+NbGuJtYEo6nreZn5nCE/xrSG0PODdCT3RK+aWBPMXFdXPV/befSZuRHY2M26EbE9M8eGXFLPmlhXE2uC5tY1Sr30/CA1de6bWFcTa4L+6xrWm7H7gGUdt0+uxqRS2fNqrGEF/VeBFRFxSkQ8DbgQ2DKkx5KawJ5XYw3l0E1mTkXE24AvAEcBH8vM+/q4y5G/3O1SE+tqYk3Q3LoGYgg9P0hNnfsm1tXEmqDPuobyZqwkqTn8ZKwkFc6gl6TCNSroI2JZRGyLiPsj4r6IeHs1fmJE3B4R36x+nlBTfUdFxNcj4pbq9ikRcVf1kfdPV2/Cjbqm4yPixoj4RkQ8EBEvr3u+IuKd1fO3KyKuj4inN2GujhT2aU91NaJXI+JjEfFoROzqGJt2fqLtQ1V990bES+e6/0YFPTAFrMvMU4GzgMsi4lRgPbA1M1cAW6vbdXg78EDH7fcDV2fm84GDwKU11PRB4POZ+ULgtKq+2uYrIpYCfwWMZeaLab8xeSHNmKsjhX3ahYb16nXAOU8Zm2l+XgusqC5rgWvmvPfMbOwFuJn23w55EFhSjS0BHqyhlpOryX4VcAsQtD+ptqha/nLgCyOu6VnAd6jeVO8Yr22+gKXAw8CJtM/qugV4Td1zdaRc7NOe6mpUrwLLgV1zzQ/wr8Abp1tvpkvT9uh/KSKWAy8B7gJambm/WvQI0KqhpH8C/hb4RXX72cAPMnOqur2XduOM0inA94B/r16q/1tEHEeN85WZ+4CrgO8C+4HHgR3UP1dHCvu0SwugV2ean8P/QR02Z42NDPqIWAx8BnhHZv6wc1m2/wsb6TmhEfFHwKOZuWOUj9uFRcBLgWsy8yXAEzzl5e+o56s6jria9i/3c4Hj+PWXpBoC+7Q3C6lX+52fxgV9RBxNO+Q/mZk3VcMHImJJtXwJ8OiIy3oF8PqI2ANspv2y+IPA8RFx+ENndXzkfS+wNzPvqm7fSPsXqs75+gPgO5n5vcz8OXAT7fmre66OBPZpb5reqzPNT89/bqNRQR8RAVwLPJCZH+hYtAVYU11fQ/vY/chk5rsy8+TMXE77zZovZ+ZFwDbg/BrregR4OCJeUA2dTfvP4tY5X98FzoqIZ1TP5+Gaap2rI4F92rOm9+pM87MFuLg6++Ys4PGOQzzTG+WbH128GfFK2i9P7gXuqS7n0j7OuBX4JvAl4MQaaxwHbqmu/zZwN7Ab+E/gmBrqOR3YXs3Z54AT6p4v4L3AN4BdwCeAY5owV0fSxT7tuq5G9CpwPe33CX5O+xXQpTPND+032D8CfAvYSfusoVnv3z+BIEmFa9ShG0nS4Bn0klQ4g16SCmfQS1LhDHpJKpxBL0mFM+glqXD/DyfMuj0CWOGFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x103245e10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pandas import DataFrame\n",
    "from pandas import read_csv\n",
    "from matplotlib import pyplot\n",
    "%matplotlib inline\n",
    "# load results file\n",
    "results = DataFrame()\n",
    "results['A'] = read_csv('results1.csv', header=None).values[:, 0]\n",
    "results['B'] = read_csv('results2.csv', header=None).values[:, 0]\n",
    "# descriptive stats\n",
    "print(results.describe())\n",
    "# box and whisker plot\n",
    "results.boxplot()\n",
    "pyplot.show()\n",
    "# histogram\n",
    "results.hist()\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The example loads both sets of results and starts off by printing summary statistics. Data in *results1.csv* is called “A” and data in *results2.csv* is called “B” for brevity.\n",
    "\n",
    "We will assume that the data represents an error score on a test dataset and that minimizing the score is the goal.\n",
    "\n",
    "We can see that on average A (50.388125) was better than B (60.388125). We can also see the same story in the median (50th percentile). Looking at the standard deviations, we can also see that it appears both distributions have a similar (identical) spread.\n",
    "\n",
    "Next, a box and whisker plot is created comparing both sets of results. The box captures the middle 50% of the data, outliers are shown as dots and the green line shows the median. We can see the data indeed has a similar spread from both distributions and appears to be symmetrical around the median.\n",
    "\n",
    "The results for A look better than B.\n",
    "\n",
    "Finally, histograms of both sets of results are plotted.\n",
    "\n",
    "The plots strongly suggest that both sets of results are drawn from a Gaussian distribution.\n",
    "\n",
    "## Normality Test\n",
    "Data drawn from a Gaussian distribution can be easier to work with as there are many tools and techniques specifically designed for this case.\n",
    "\n",
    "We can use a statistical test to confirm that the results drawn from both distributions are Gaussian (also called the normal distribution).\n",
    "\n",
    "In SciPy, this is the normaltest() function.\n",
    "\n",
    "The null hypothesis of the test (H0), or the default expectation, is that the statistic describes a normal distribution.\n",
    "\n",
    "We fail to reject this hypothesis if the p-value is greater than 0.05. We reject this hypothesis if the p-value <= 0.05. In this case, we would believe the distribution is not normal with 95% confidence.\n",
    "\n",
    "The code below loads *results1.csv* and determines whether it is likely that the data is Gaussian."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.9901307811630056 0.22423394146329587\n",
      "It is likely that result1 is normal\n"
     ]
    }
   ],
   "source": [
    "from pandas import read_csv\n",
    "from scipy.stats import normaltest\n",
    "from matplotlib import pyplot\n",
    "result1 = read_csv('results1.csv', header=None)\n",
    "value, p = normaltest(result1.values[:,0])\n",
    "print(value, p)\n",
    "if p >= 0.05:\n",
    "\tprint('It is likely that result1 is normal')\n",
    "else:\n",
    "\tprint('It is unlikely that result1 is normal')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the example first prints the calculated statistic and the p-value that the statistic was calculated from a Gaussian distribution.\n",
    "\n",
    "We can see that it is very likely that *results1.csv* is Gaussian.\n",
    "\n",
    "We can repeat this same test with data from *results2.csv*.\n",
    "\n",
    "The complete code listing is provided below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.9901307811630233 0.2242339414632938\n",
      "It is likely that result2 is normal\n"
     ]
    }
   ],
   "source": [
    "from pandas import read_csv\n",
    "from scipy.stats import normaltest\n",
    "from matplotlib import pyplot\n",
    "result2 = read_csv('results2.csv', header=None)\n",
    "value, p = normaltest(result2.values[:,0])\n",
    "print(value, p)\n",
    "if p >= 0.05:\n",
    "\tprint('It is likely that result2 is normal')\n",
    "else:\n",
    "\tprint('It is unlikely that result2 is normal')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the example provides the same statistic p-value and outcome.\n",
    "\n",
    "Both sets of results are Gaussian.\n",
    "\n",
    "## Compare Means for Gaussian Results\n",
    "Both sets of results are Gaussian and have the same variance; this means we can use the Student t-test to see if the difference between the means of the two distributions is statistically significant or not.\n",
    "\n",
    "In SciPy, we can use the ttest_ind() function.\n",
    "\n",
    "The null hypothesis of the test (H0) or the default expectation is that both samples were drawn from the same population. If we fail to reject this hypothesis, it means that there is no significant difference between the means.\n",
    "\n",
    "If we get a p-value of <= 0.05, it means that we can reject the null hypothesis and that the means are significantly different with a 95% confidence. That means for 95 similar samples out of 100, the means would be significantly different, and not so in 5 out of 100 cases.\n",
    "\n",
    "An important assumption of this statistical test, besides the data being Gaussian, is that both distributions have the same variance. We know this to be the case from reviewing the descriptive statistics in a previous step.\n",
    "\n",
    "The complete code listing is provided below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-22.78226550283975 2.5159901708040968e-102\n",
      "Samples are likely drawn from different distributions (reject H0)\n"
     ]
    }
   ],
   "source": [
    "from pandas import read_csv\n",
    "from scipy.stats import ttest_ind\n",
    "from matplotlib import pyplot\n",
    "# load results1\n",
    "result1 = read_csv('results1.csv', header=None)\n",
    "values1 = result1.values[:,0]\n",
    "# load results2\n",
    "result2 = read_csv('results2.csv', header=None)\n",
    "values2 = result2.values[:,0]\n",
    "# calculate the significance\n",
    "value, pvalue = ttest_ind(values1, values2, equal_var=True)\n",
    "print(value, pvalue)\n",
    "if pvalue > 0.05:\n",
    "\tprint('Samples are likely drawn from the same distributions (fail to reject H0)')\n",
    "else:\n",
    "\tprint('Samples are likely drawn from different distributions (reject H0)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the example prints the statistic and the p-value. We can see that the p-value is much lower than 0.05.\n",
    "\n",
    "In fact, it is so small that we have a near certainty that the difference between the means is statistically significant.\n",
    "\n",
    "## Compare Means for Gaussian Results with Different Variance\n",
    "What if the means were the same for the two sets of results, but the variance was different?\n",
    "\n",
    "We would not be able to use the Student t-test as is. In fact, we would have to use a modified version of the test called Welch’s t-test.\n",
    "\n",
    "In SciPy, this is the same ttest_ind() function, but we must set the “equal_var” argument to “False” to indicate the variances are not equal.\n",
    "\n",
    "We can demonstrate this with an example where we generate two sets of results with means that are very similar (50 vs 51) and very different standard deviations (1 vs 10). We will generate 100 samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-2.6223313740555843 0.010087148378339763\n",
      "Samples are likely drawn from different distributions (reject H0)\n"
     ]
    }
   ],
   "source": [
    "from numpy.random import seed\n",
    "from numpy.random import normal\n",
    "from scipy.stats import ttest_ind\n",
    "# generate results\n",
    "seed(1)\n",
    "n = 100\n",
    "values1 = normal(50, 1, n)\n",
    "values2 = normal(51, 10, n)\n",
    "# calculate the significance\n",
    "value, pvalue = ttest_ind(values1, values2, equal_var=False)\n",
    "print(value, pvalue)\n",
    "if pvalue > 0.05:\n",
    "\tprint('Samples are likely drawn from the same distributions (fail to reject H0)')\n",
    "else:\n",
    "\tprint('Samples are likely drawn from different distributions (reject H0)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the example prints the test statistic and the p-value.\n",
    "\n",
    "We can see that there is good evidence (nearly 99%) that the samples were drawn from different distributions, that the means are significantly different.\n",
    "\n",
    "The closer the distributions are, the larger the sample that is required to tell them apart.\n",
    "\n",
    "We can demonstrate this by calculating the statistical test on different sized sub-samples of each set of results and plotting the p-values against the sample size.\n",
    "\n",
    "We would expect the p-value to get smaller with the increase sample size. We can also draw a line at the 95% level (0.05) and show at what point the sample size is large enough to indicate these two populations are significantly different."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/fromnumeric.py:3194: RuntimeWarning: Degrees of freedom <= 0 for slice\n",
      "  **kwargs)\n",
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/_methods.py:127: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  ret = ret.dtype.type(ret / rcount)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8W1eZ+P/P0S5L8r4ldvY4e5ek6UYLXaHLF1qgBdqBAQaGDtvAAANTZuHH9DsLzDDAdIbhC3TYt6GlTAMUCm0D3UuTtGn2xEmaxE7ifbd2nd8f915ZtiRbtiU7lp7365VXLen66ipKHz16zjnPUVprhBBCFBfbfF+AEEKI/JPgLoQQRUiCuxBCFCEJ7kIIUYQkuAshRBGS4C6EEEVIgrsQQhQhCe5CCFGEJLgLIUQRcszXE9fW1urly5fP19MLIcSCtHPnzm6tdd1Ux81bcF++fDk7duyYr6cXQogFSSl1IpfjpCwjhBBFSIK7EEIUoSmDu1Lqm0qpTqXU3iyPK6XUvUqpVqXUy0qpLfm/TCGEENORS+b+beDGSR6/CWgx/9wFfHX2lyWEEGI2pgzuWusngN5JDrkV+K42PAdUKqUW5esChRBCTF8+au5NwKmU223mfUIIIebJnA6oKqXuUkrtUErt6OrqmsunFkKIkpKP4N4OLEm53Wzel0Zr/XWt9Vat9da6uinn4Jekn+5sYygUne/LEEIscPkI7tuAd5qzZi4DBrTWZ/Jw3pJzqneUT9y/mwd2tk163On+IIPyASCEmEQuUyF/BDwLrFVKtSml3quUer9S6v3mIQ8Dx4BW4BvABwt2tfPscMcQe9sHCnb+3pFI8nkm89avPcuXfnu4YNchhFj4pmw/oLW+c4rHNfChvF3ROexzvzpI11CYn//5lQU5v5WNH+4YznpMMBKnrS9Ie1+wINcghCgOskJ1GoZDMToGQwU7/0DQCu5DGJ+Z6dr7jaDePyplGSFEdhLcpyEUi9MzEiGRyBx4Z8sK7kOhGB2D4YzHtPWNAtA3GinINQghioME92kIRePEEzoZhPNtMBhL/pyt7t5mlmP6JHMXQkxCgvs0hKIJALqHM2fVkznRM8IHf7CTzknKOqkfGkc6M9fdreDePxrJWroRQggJ7tMQisYB6JpBcP/D8V4e3nOWP/v+TsKxeMZjBoJRav0uqn0ujmTN3I2yTCyhGQrHMh4jhBAS3KfBCu49w9Ovdw+bgfjFk/387c/2Zsy6B0NRyj1OWur9U5ZlAPpHpDQjhMhMgvs0hGIzL8sMh4zg/v6rVnH/zja+88wraccMBqOUe52saQhwpGM44wdAW1+QWr8bkEFVIUR2EtxzlEhoIrMJ7uEYboeNT92wluvXN/B/f3mAV7pHxh0zEIxS4XWypsHPUDjG2Qn1+VA0TvdwmPOaygEJ7kKI7CS45yhsBnaYWVlmKBwj4HFgsyk+fO1q4gnN0a7xg6aDZnBfXR8A0hczWXPcNzVVADLXXQiRnQT3HFn1dph5WcbvNhYE1/hcQPqHxEAwSrnXwZoGP0DaoKpVbz/PDO5WuwIhhJhIgnuOQikzXLpmOKAa8DgBqPGbwT0lOGutGQzFqPA6qfG7qfG50gZVrZkyGxaXo5QxHVIIITKR4J4ja4673abomWXmXuZy4HHaxp1nJGIskKrwGh8ALQ3+tLnubX1BnHbFogovFV6nLGQSQmQlwT1HVllmUYWH7uHwtBcQDYVj+D1jfdpqfO5xZRVrAVO5md2vaQjQOmHGTFtfkMWVXuw2RVWZSwZUhRBZSXDPkRXcmyq9hKIJRiKZFyJlMxyOEnCnBHe/i+7U4G5m4cnMvd6YMXNmYGzGTFvfKM1VXgAqy5wyoCqEyEqCe46sskxzVRnAtEszQ6GJmbuL3pGxc1iZ+1hZxpoxM1Z3b+sL0lxpPL9k7kKIyUhwz5E1oGplztOZMaO1HldzB6j2ucfNlrF6uZebwX2tGdx3nzI2BwlF43QNhWlKydz7ZLaMECILCe45CltlGTO4dg3lHljDsQSxhB6Xudf6XfSMjDX/mpi5V/lcXLaymvt3niKe0Jw257hbHy7VZS4ZUBVCZCXBPUdjZRkjuPaM5J65D5mtBwLjMncXkVgi2XNmMDg+cwd4+6XLaOsL8sSRruQcd6ssVOVzEYzGx82/F0IIiwT3HKUOqAJ0TyNztwL4uJq72R/GmjEzGIyi1PgPgBs2NlLrd/GD506mBPexsgzIKlUhRGYS3HNkBXe/20FlmXNaNffhZOY+lpVbq1S7zbr7QNCYTWOzqeQxLoeNt25dwuMHO/jD8R4cNkVDuQcwBlRB+ssIITKT4J4jqyOkx2mnxueaXlkmbGTX4zN3IzhbmftAMEpFmTPtd++8ZCkaeGj36eQcdxjL3GVQVQiRiQT3HFmZu8dpp9bvnl5Zxszc/RNq7jA2pdJqPTDRkuoyrl5Th9ZjJRlIzdylLCOESCfBPUehaAKnXWG3KWoD7umVZcyae2DCClUY6y9jtfvN5O2XLgPGB3frw0HKMkKITCS45ygUjeNx2AGo9blmFNxTM3evy06Zy56c6z4QjCZbD0x0zbp6rl5bx9Vr65P3jQ2oSnAXQqRzTH2IAAjH4ridZnD3uxkMxYz7zIA/GWsqZGrNHYy6u7VKdXCSzN1uU3z7Ty4Zd5/bYXw4SFlGCJGJZO45CkUTeJzGX5c1jTHXTTuGQjFcdlvaB0G1z51TWSYbaUEghMhGgnuOQtE4nmTmnnmzjWyGw9G0rB2M8k7PcIRQNE44lhi3gCkX0oJACJGNBPccGcHd+OuqDRiZe65194l9ZSzV5pTKiX1lclUlLQiEEFlIcM9ROJZIGVA1gntXrsHd3D91ohq/0dN9YrvfXBltfyVzF0Kkk+Ceo3FlmcD0yjJDWTL3Gp+LaFzTZjYFm25wr/ZJ5i6EyEyCe45SB1TLXA7KXPbcyzJZM3fjQ+J41wgA5RmOmUxlmYvBUJR4Ynq7Qgkhip8E9xyFUqZCgrmT0jSCe7aaO8DxbiO4T3+2jBOtx9oFCyGEJafgrpS6USl1SCnVqpS6O8PjS5VS25VSLyqlXlZK3Zz/S51f4ehYzR2Mue45z5aZsAtT6jkAjnUbG2HPZCokMG4vViGEgByCu1LKDnwFuAnYANyplNow4bC/BX6itd4M3AH8V74vdL6lzpYBIzDnmrkPhWP43emBO5m5W2WZGQyogqxSFUKkyyVzvwRo1Vof01pHgB8Dt044RgPl5s8VwOn8XeK5IXVAFaydkKYOquFYnEgskbHmbgX30wMhylx2nPbpVcmkeZgQIptcokkTcCrldpt5X6rPAu9QSrUBDwN/nulESqm7lFI7lFI7urq6ZnC58ycUS4zL3CvKnDnVujN1hLR4nPbk/dMtyYA0DxNCZJevAdU7gW9rrZuBm4HvKaXSzq21/rrWeqvWemtdXV2enrrwovEE8YQeV3Mv9zgIRROEY5Nvc5epaVgqK0Bnaxo2mcnKMqFoPPncQojSk0twbweWpNxuNu9L9V7gJwBa62cBD1Cbjws8F6T2crdYmfZU2Xu2pmEWazrkTDJ3v9uBw6YylmX+/uf7eM+3Xpj2OYUQxSGX4P4C0KKUWqGUcmEMmG6bcMxJ4DoApdR6jOC+sOouk7A2x04ty1iDn4NTBPdMvdxTWdvtTXcwFUApRWWZK2N/mba+IMd7RqZ9TiFEcZgyuGutY8CHgUeAAxizYvYppe5RSt1iHvYJ4H1Kqd3Aj4B3a62LZmWNlbm7Z5C5Z9o/NZW1acdMMncw5rpn2iR7OByjbyRCEb0NQohpyGlJpNb6YYyB0tT7PpPy837givxe2rnDqqvPpCyTrLlnydyr/VbmPrPW+gGPI2NtfTgUI5bQWbfvE0IUN1mhmoNkWcaRMlsm15r7FAOqVllmpgHY73EmnyPViHmftAQWojRJcM/BpAOqU8wxT5ZlCjCgCuB32xkOZS7LwNgerUKI0iLBPQdjA6opUyGtAdXQ+Kz513vPcvtXn0k28xoOR3HYFG5H5r9qq+Y+k6mQYHwjmFiW0Von75PMXYjSJME9B2OZ+9hfl9Nuw+eyp5VlXnillx0n+jhu9oux+soopTKee0WtD6ddsbLON6Nr87udyW8HY9ebwGoU2SsLnIQoSbJBdg5CGQZUwcjeJwb3HrPfzL7Tg6yuD2Tt5W5ZUl3Gns/ekHbuXPk9DkYicRIJjc1mfICkZvLSVEyI0iSZew7GBlTHB+CKTMHdDKb7Tg8CVtOwyT9DZxrYAQLmuUciYwE9NbhLWUaI0iTBPQeZyjKQOXPvTQb3AcAoy8y0np4LnxncUwP6SMrPMqAqRGmS4J6DTIuYwMjcJ65QtXq8720fTA5sZpvjng/WuVPr7pK5CyEkuOcgHEtvPwDpZRmtNb0jEco9DgaCUdr7g1l3YcqXwCSZe7nHIQOqQpQoCe45CEXjKAUu++TBfSgcIxJPcGWL0TNt3+lBY0B1LjL3cHrmvrSmTAZUhShREtxzEIrG8TjsadMZK7xORiNxonEjs+81SzKXr6rFpozgPhyOJrPrQrC+FWQqyyytluAuRKmS4J6DUDSRVpKBsVWlVt3dGrxsrvKyqs7P7lP9hKKJgpZlrHMPZSjLLKkuYygUS374CCFKhwT3HEzcYs8ysb+MNce9xudi4+JydrzSC2RvGpYPGTN38+fmSi8gg6pClCIJ7jkwtthLD+5WJ0cruFslkBq/m42LKxiJGLNsCpm5W1MhR8bV3OP43Q5q/EZrAxlUFaL0SHDPQSgaz9gbJi1zt4K7mblbsjUNyweXw4bbYUubLeNz25MbaFtjAUKI0rGgg/svXj7Na/5lO11D4YI+T+5lmQhlLjsep50NKcHdn2WjjnwJeBzjau7D4Rg+tyPZcVIydyFKz4IO7rG45mTvKIMZWt7mUzjLgOrErfZ6R8LJgFpZ5qK5yqh5FzJzB6M0M3G2TMDtSGbuUnMXovQs6OBuBc2hUPpmFfkUiuWYuY9EqDZb+ALJ0kwhB1Qhve3viJm5V5U5k9clhCgtCzy4G8FrqMCZuzXPfSK3w47HaRtXlqk1d1YC2LS4wrzOuQ3uVlnGYbdR4XVK5i5ECVrQLX8zTQMshFA0gTtDWQas/jLG8/eORMYNpL79smU0lHuoD3gKen0Bj4PT/aHkbassA1Dtc9E7xW5RQojis8Az9zkqy2TJ3GGsBYHWmp6RcHLDazAC61svXlLQa4PsZRnrGnpHCjvgLIQ49yzo4G610i30gKoxWyZ75j4QjDIUjhGNa2pTau5zxZelLANQVeaid0QydyFKzYIO7j63kU1P3EM037ItYgLjA2YgGE3OJa9OqbnPFb9nbLZMOBYnGtfJbzXVPqm5C1GKFnRwd9htlLnsBS3LJBKaSCyR1svdYmXuPWbpI7UsM1cCbgeReIJwLM5I2FgV63MZ11vtc9M7EkFrPefXJYSYPws6uIO5gKeAZZlsvdwt5eaGHdYmHfNRlvEnWxDEk20IxmruTiLxRLIVghCiNBRBcHcWtCyT3GJvkgHVoXCMruH5y9z95tjDcCiW/BYzVpYx+8tICwIhSsqCD+5+t6OgZZlQzNo/NXtwB3ilewQw+srMNb859jAUjiY3yk7N3EFaEAhRahb0PHcwMtTBQgb36ORlGSu4H+sawWf2lZlrVu+a4VCMUfObRupsGZAWBEKUmgWfuZd7nAwXsOaeLMtMkbkf7x6Zl5IMjLU3GInEkrNmrEVMNWZZRloQCFFaFnxwL3hZJhncs2TuZv+Wk72jyUA615K7MYViaQOqVWZZRjJ3IUpLUZRlChvczbLMJAOqALGEnpd6O6S0YQjHCEbGl2X8bgcuu01q7kKUmJwyd6XUjUqpQ0qpVqXU3VmOeatSar9Sap9S6of5vczsAh4nwWicWIH2CbUGVLPNc7dWyQLJdr9zzSrLDIdiyZlDVsBXSlHlc8psGSFKzJSZu1LKDnwFeC3QBryglNqmtd6fckwL8GngCq11n1KqvlAXPFEysIVjVJblP7iGcxxQBca1+51LZU47Shk9ZYLROF6nHbtNJR+vKnNJ5i5Eicklc78EaNVaH9NaR4AfA7dOOOZ9wFe01n0AWuvO/F5mdlM1DxsKRbn1K09zuGNoRucPTzEV0uO04bIbf43zVZax2RR+l7Eb03A4nizJWGr8ruT+rkKI0pBLcG8CTqXcbjPvS7UGWKOUelop9ZxS6sZ8XeBUyqcI7id7R9l9qp/nj/fO6PxTzZZRSiV3ZJqvsgyM9ZcZDsfS+sdXlblkQFWIEpOvAVUH0AJcDTQDTyilztNa96cepJS6C7gLYOnSpXl5YmuOd7YWBFZw7hoMZXw8E6sPi1IqZUA1++dghddB93B4XpqGWazOkOFYItlQzVLtc8lUSCFKTC6ZezuQ2pS82bwvVRuwTWsd1VofBw5jBPtxtNZf11pv1Vpvraurm+k1jzNVWSYYMYJz5zQ20X7Tfz3DPb8whhSmytxhrO5e65+fmjuM9XQfDsfwucZ/Zlf7XAwEo0QLNOgshDj35BLcXwBalFIrlFIu4A5g24Rj/hcja0cpVYtRpjmWx+vMKpAyoJpJ0AzOHTlm7se7R3jpVD8P7monGk+krFCdOrjPZ+Ye8JjBPZRelqkxP3T6ZFBViJIxZXDXWseADwOPAAeAn2it9yml7lFK3WIe9gjQo5TaD2wHPqm17inURafyJzP3ycsyuWbujx80xoIHglGePdpDKBbHaVfjZp9MdC4Ed7/bqLmPRGLpA6rmdcmgqhClI6eau9b6YeDhCfd9JuVnDXzc/DOnxnZjmjxzzzW4bz/YyfKaMrqGwvxq7xk8TnvWBUyWhnIP1T7XvPSVsVhlmWg8kRbcrQ8dmesuROlY8O0H3A4bTrvKWpaxMvee4fCUC52GQlGeP97DDRsbuW59A4/s62A0HM+6gMnygatX8T93XTazF5AnPjNzHwrFkguYLFbmns9B1Vg8wV/ev5sDZwbzdk4hRP4s+PYDSimzv8zkZZmENoJbQ7kn67meOtJNNK65dl09faMRtu0+zZNHurIuYLJUlrkKsoBqOgIeY547kBbcqwtQljnZO8oDO9uo8btYv6g8b+cVQuTHgg/uYLQgmGq2DEDnYHjS4P7YwU7KPQ4uWlZFNK7xOu2cHgixut6f92vOt9SAPrEsU1nmQinj20u+tPcHAdh/WjJ3Ic5FC74sA+ZMkSlq7gCdQ9lnzCQSmu0HO7l6bT0Ouw2vy861640uClNl7ucCf8oMGf+Eee52m6KqLL9z3dv7jOC+7/Sg7M8qxDno3I9aOZisM2QoJbh3DGbPXHe39dMzEuHadWNtcW7etAjI3hHyXJKauVsLu1JV+/LbgsDK3HtHIpydxgIxIcTcKIrg7nc7GZyk5m7VnCfL3Lcf7MSm4Ko1Y4urrl5bh8dpm9dZMLkaX5ZJv958r1K1MneQ0owQ56KiCO7l5gKeTILROOUeBzU+16TTIR872MlFy6qoSpmr7nM7+Nj1a7jlgsV5v+Z8G5+5pw+l1OQ5c2/rD7JhUTlKGaUZIcS5pUgGVLOXZYKRuDFX3Wmnc5LywZHOYd79quVp9//ZVavydZkFNa7m7kl/W/NdljndH2TrsiqC0Tj7Tg/k7bxCiPwoiuDuNzN3rTVKjV9JGowawb3c68yauYeicSKxxLje7AtNIKXOPrG3DBgtCPpGI8QTetLVtrmIJzRnB0I0VXmJJjS7T/VP/UtCiDlVFGWZgMdJPKHHzYyxhKMJvE479QE3nVkGVAeCRr1+IQf38bNlMpdltIb+PPSX6RgMEUtomirL2Li4nLa+IAOjhdukXAgxfUUS3LN3hgxG43hddhrK3XQNh0kk0qftFUNwTx1EnTjPHfK7kMmaKdNU5WXj4goA9stKVSHOKUUR3K1MNdMqVaMsY6M+4CGe0BlnjBRDcHc77LjsNlwO489EVguC7jz0l7FmyjRVetlgrk6VursQ55aiqLlbzcMyZe4hs+ZeHzDa3nYOhagLjO+7bpUUFnJwh8wDqZZqfwEy90ovXpfxdyvTIYU4txRF5j5ZWSZkbhhdb7YdyDSoWgyZOxjfYDLV2yG1LDP7FgRtfUGqfS68LqMUtHFxuUyHFOIcUxTB3T9ZzT0yIXPPMB2ymIJ7pno7GPuoQn46Q7b3B2mq9CZvb1xcQWvX8LjVwEKI+VUUwT1glmWGw+Nr7lobM2i8TnuyFJNpxowV3MsXeHCvDbipzbJJt9Nuo8LrzE9Zpm90XHDfsLiceEJzuGMo53P84Xgv2w91zvpahBCZFUXNPVtZJhrXJDR4XcYipsqyzHPdB0NRAm7HrOd/z7d/fvN5kz5ek4cWBFpr2vuDXL12rAfPxsXWoOog5zdX5nSeLzxyiFN9ozz76etmdT1CiMyKInO3Fu1M3I0pOGFz6/qAO+NeqgPB6ILP2sEY4EzNqCeq9rmy7saktZ50Ba+ldyRCKJoY9zxLq8vwuewcOpt75v5KzwhnBkKcGQhOfbAQYtqKIrjbbSq5h2iqUDK4Gy+zPuDJnLkXSXCfymQtCB7Zd5YrPv84ZwcmD/Cn+43Hm6rGgrtSitUNgZzLMsFIPPk+vHhSVrcKUQhFEdzB6i8zvuZuBXevlbmXu+nKMlumwlsUFapJ1fjdWcsyB84MEY3rKeert/ePAqR9Q2ip93Okczin6zjVN5r8edeJvpx+RwgxPUUT3I2t9jKXZZLBPeChcyiUtrmEEdyLP3Ov8bnoG41kXKVrBdxDU2TfbeYCpuaq8cF9TYOfrqFwTu0NTvQYz+V3O9h1UoK7EIVQNME9kKHtbzCSXnOPxjV9E/qglEpwr/a5iCd0cnZQKitoH56ibt7eH8Tnsqf9fbXUBwByyt5P9IwAcNOmRva2DxKOyRRKIfKtiIK7M60skzagWj62SjVVqQT3Gn/2ue5WS4HDHZMH5/a+IE1V3rTumy0Nxj6zR6b4fTA21w64HVy3vp5IPCELoIQogCIK7ullmXDU2BzbWklpbY6dOtc9HIsTii7sdr+5ytY8LBpPcGYgiN2maO0aJhZPZPp1wMjcF2eYkbO4wkuZy57ToOqJnlGW1pSxZWkVIHV3IQqhuIL7xLJMWs3dyNxTp0MWy+rUXGRrQXB2IERCw9ZlVURiCU70jmb6dSB9darFZlO01PtpnVCWufexI/x89+lx953sHWVZTRn15R6aKr0yY0aIAiii4J6hLBMZPxWyIUN/mcEiWZ2aixqf8eE2sSxjDaZet95YmJSt7j4SjtE/Gh03DTLV6vrx0yEHQ1HufewI//3U8eR98YSmrW+UpdU+ADYvreRFGVQVIu+KJ7i7HYSiCaIpJYVQbHzmbq1STV04U0qZe5XPeI0TFzJZg6lXr61Hqex1dytwLzMD80QtDX46h8LJLptPH+kmljCmV1rTUs8MBInGNctqygDYsrSK0wOhKefXCyGmp2iCu9U8LHUhUzJzd41tZNFY7uHswFjmXkrB3e2wE3A70jL3tr4gNgUran0sqy7LWjd/9EAHdpviitU1GR9fYw2qdhq/b/WOicY1e9uN+fMnzWmQy6rN4L7MrLtL9i5EXhVNcA9k6Ok+cRETQGOFh7ODY5n7YNA4vhSCOxh93ScOqLb1jrKowovTbqOlIZB1rvtv9nVwyfJqKssyNydLnQ6pteZ3h7q4dEU1MBa8rXr+UjNz37CoHLfDJoOqQuRZEQV3q7/MWN09GI1jtymc9rGXWcqZO2RuQdBmTm8EWNsQ4Hj3SNrc8+PdIxzpHOZ1Gxuynrup0ovXacyY2X9mkM6hMLdf1MzS6jJ2nTAGTU/0jOK0KxZVGM/nctg4r6mCxw52jtvwIxJLsG33af7z8SN0D8++B70QpaZo1txbwXkwZYFOyNwcO1VjhYfu4TCRWAKXw1Y07X5zVeNzJWvslra+US5bZZRa1jQGiCc0x7pGWG9uoQfw2/1nAXjthuzB3WZTrDZnzPzuUBcAV62t4+nWbp4+2oPWmpO9IzRXlY3rwPmeK1fwyft3c/O9T/Lqllo2LCrnp7vaklsCfvV3R3nfa1byvlevzNqvXggxXk6Zu1LqRqXUIaVUq1Lq7kmOu00ppZVSW/N3ibmxgnvq6sugucVeqsbkjJlQ8vgyl31cdl/ManzucZl7JJbg7GCI5iqjTLK2wSitTKy7/2ZfBxsWlSePy6alwc/hjiG2H+zkvKYK6gMetiyromsoTFtf0JjjXj3+HDeft4hnPn0dn7pxLQfPDvGNJ49x4ZIqvvOeS3j041fxmjV1fPnRI9zw5ScYjaRvyCKESDdlGqSUsgNfAV4LtAEvKKW2aa33TzguAHwUeL4QFzqVyjIjuPenZu6ReHIapKWxwgjuZweMgFYqq1Mt9eVG87DRSIwylyM5x93qFbOi1ofDpsYF9+7hMDtP9vHR61qmPH9LfYAHd7XTNRTmw9esBhhbrHSyj5M9o1xkDqKmqvA6+eDVq/nTK1cyGomNq+t/9R0X8ZMdp/jUAy9z4MwgFy2rntXfgRClIJd09RKgVWt9TGsdAX4M3JrhuP8LfB6Ylzlt2TL3TGUZgLODY5l7KQX3S1fUEE9onj3aAxglGRgL7i6HjRW1Pg6dHZsO+diBDrSevCRjsWbMJDRcZW7osa4xgNdp5/GDnQyFY2mZeyqXw5ZxwPZVZtnowJnce8YLUcpyCe5NwKmU223mfUlKqS3AEq31L/N4bdPiddpx2hX9o6k193iy9YBlUbkRxKx51cWyUUeuLl5RRZnLCLQwNsd9SUq5ZU3j+MVIv9nXQVOllw0pNfhsrBkzlWVOLlxi7MrksNu4YEkFj+wz6vbLajLPk59MU6WXgNsxrQ1BhChlsy40K6VswBeBT+Rw7F1KqR1KqR1dXV2zfeqJ56bC65qy5l7udeBx2pLBfbDEMne3w86Vq2vZfrATrY3VojY19o0GjLr7qb5RHtl3lgd2tvFkazev29iQ1iwsk+YqL363g6vX1I0bNN2ytIqQ2evHWsA0HUop1i0KcPCsNBkTIhe5TD1oB5ak3G4277MEgE3A78z/+RuBbUqpW7TWO1JPpLX+OvB1gK1bt6Y3FZ+lCq+DgeAT9X9/AAAbYklEQVTYYGEwQ0MwpYxpeGdSyjKbSii4A1y7rp7f7O/gcMcwp/qCyTnulk1N5WgNf/a9nQDYFLzhgsU5ndtmU3z/Ty9lcaVn3P1W3R2YtCwzmbWNAR568TRa65w+aIQoZbkE9xeAFqXUCoygfgfwR9aDWusBoNa6rZT6HfCXEwP7XKgsG5+5h6NxvGab31QN5W46Bkqz5g4kN7d+/GAnbX2jaRtvXL2mnvvffzkuu43KMic1fjf+aUxBtMoxqayVqA3l7rRvU7la11jO98Mnae8PTjlrR4hSN2VZRmsdAz4MPAIcAH6itd6nlLpHKXVLoS9wOiq8zikHVAEjcx8IEY0nGI3ESy64N1Z42LConO2HOmnrSw+UNpvi4uXVXLCkkmU1vmkF9myqfS5W1vpYPoN6u2X9IqOef1AGVYWYUk7/12qtHwYennDfZ7Ice/XsL2tmKr3OcQOBwUj6gCoY3SE7h0LJwddSC+5glGa++vujJLROy9wL5Utvu3BW6wnWmHPwD3UMcX0OM3eEKGVFtXKnPEPm7nakB/fGcmO7vePdxnZvpRjcr1lXRzyh0Tp9P9RCuWBJJRsWTz3jJpuAx0lzlZcDZ2RQVYipFFVwr/A6GQrFiJsbQIejiYyZe6PZ18RqkFWKwf3CJVVUmQu/FlL9el1jOQdlOqQQUyqq4G6tUh0MRonFE0Ti6b1lYGzan7UpRbm39PqV2G2Kq9bUAXOXuefD+kVGYzOr46cQIrOiimqpq1SdDuNza2L7AYBFZnC3FsSUYuYO8K5XLUcplXFP1HPVusZy4glNa+cwm5oq5vtyhDhnFVVwT+0vY23ekSlzr/W7sdtUsixTSitUU21eWsXmpel9Xs5laxvNGTNnhyS4CzGJoirLpGbuY/unpgd3u01RH3CXXC/3YrC8pgy3w8ZBGVQVYlJFFtyNhlP9o5GxXZgyDKjC2GbZHqct44wacW5y2G2saQjIoKoQUyiy4D42oBo0g7snS+C26u6StS886xoluAsxlaIM7v2j0WSTqqkydwnuC8+6ReV0D4fpGpLt94TIpqiCu8tho8xlN2ru0ew1d5DMfSHbstToXfNUa347iwpRTIoquMNYf5mxAdXML7FRgvuCdUFzJfUBN4/s7ZjvSxHinFWUwb0/GCUcMwdUs2TuVlmmVKdBLmQ2m+KGjY38/nBX8kO8kL7028N84Ps7C/48QuRTUQb31Mw9W81dyjIL2w0bGwlG4zxxpPClmRde6eXRAx2yKlYsKMUZ3EfHau6TZe4uhy2ZwYuF5dKV1VR4ncmt+wqpayhMNK5liz+xoBRdcK8sc+Y0oOpx2nnoQ1fwx5ctm8vLE3nitNu4bn09j+7vIBpPFPS5uoeNWTkvtw8U9HmEyKeiC+5WWcaaCul2ZH+J6xeV48vDRhRiftywsZHBUIznj/UW7Dmi8QR9Zt//PW39BXseIfKt6IJ7ZZmLYDTOYDCK12mXvTaL2Gta6vA67QUtzfQMj+3J+3KbZO5i4Si64G7Nfjk7EMo6mCqKg9dl56o1dTyy7yyJRN73WwdILpRa1xjgSOfwnMzOESIfii64V1rBfTCEZ5KSjCgON25qpHMozK6TfQU5v1Vvv2ZdPfGEZr80LBMLRNFFP2tqY8dgCI9k7kXv+g0NeJ12frqrvSDntzL369bVA1J3FwtH0Qb3zqFw1mmQonj43Q5uOq+RX+w+XZB56F1m5r6pqYK6gFtmzIgFo+iCu7VhRzyhs06DFMXl9ouaGQrHCjKw2jUUJuB24HHaOb+pgj1zPKj6TGt3ciN3Iaaj6IJ76opTydxLw2Uramiq9PLAzra8n7trOExtwA3Aec0VtHYNMxKO5f15MglG4rz7Wy9w21ef4WjX8LjHtNbJjeCFyKTognvA48Sa/SiZe2mw2RS3XdTMU63dnBkI5vXc3UNh6vxGcD+/uQKtYd/puRlU3XGil0g8wXAoxh/f9zyn+43X9vjBDq78/Hb+9DsvoLUEeJFZ0QV3u00RMBcmyVTI0nHblia0hgfzPLDaNRymzszcrT1bX56jQdWnW3tw2BQ/eN+lDIVivPObf+BDP9zFe769g5FIjO2HuniqtXtOrkUsPEUX3AEqzLq7TIUsHctqfFyyopqf7mzLazbbPRSm1m9s31gf8NBY7mHPHA2qPnu0m81LK7l4eTXfeNdWTvaO8tt9HXzitWt46q+upanSy+d/fbBgc/zFwlaU0a/S3EtVMvfScvtFzRzrHuH3h/PTKTIUjTMYiiUzdzDq7nMxqDoQjLKnfYDLV9UCcNnKGrZ9+Aoe/fhV/Pl1LfjdDj7+2jXsbR/k4b1nCn49YuEpyuBuDarKgGppef35i1hZ5+PjP9lNW9/orM/XM2K0Hqj1jwX3DYvKOd4zUvD2v88f6yGh4YpVNcn71jWWs7SmLHn7jZubWNsQ4N9+czjZPE1q8MJSnMHdLMu4JbiXlDKXg2+8cyvReIK7vruT0cjsZrVYC5hSM/c1DQG0htbO4Wy/lhfPHO3B47RxobmlYCZ2m+KTN6zlePcId/90Dx/8wU4u/afHePN/PS1BXhRpcJfMvWStqvNz752bOXB2kE/e//Ksgly3GdxTM/c1DX4AjnQWtrf7M0e7uXh5NW7H5P+Gr1tfzyXLq/nprjZ2nxpgSXUZu072z9mMHnHuKsp+t5XJ4F6Un11iCtesreevblzH5351kPB3E9xz60YWV3qnfR5rdWpq5r681ofTrjjcUbjMvXMoxOGOYd68pXnKY5VSfOtPLmY4HKOh3EP/aISL//FRHnqpPTm7R5Smoox+ycxdBlRL1p+9ZiV/ffM6nmrt4rVf/D3ffOr4tBf9WJl7jTlbBoxNQlbU+jhcwF2Znj3aA8CrUurtk/G5HckdxSrLXFy1pp5tu09P+/WGonEe3nOGWIE3PxFzI6fgrpS6USl1SCnVqpS6O8PjH1dK7VdKvayUekwpNa/bG1nBXRYxlS6lFHe9ZhW//dhVbF1ezT2/2M+//ebQtM7RNRymwutMK420NAQ4XMCyzDOtPZR7HGxcPLPM+9YLF9MxGOYPx3PfxERrzV8/uIcP/mAXD75YmCZsYm5NGdyVUnbgK8BNwAbgTqXUhgmHvQhs1VqfDzwA/Eu+L3Q6rP4yEtzFkuoyvv0nF/OmzU3c9+RxXplGn5auofC4koxlTX2AU73BWQ/YZhKNJ3iqtZvLVtZgt81so5nr1zdQ5rKzbXfuQfp7z53gwRfbcdgUPytQh00xt3LJ3C8BWrXWx7TWEeDHwK2pB2itt2utrblnzwFTFwsLqNpn/A/ply30BEYW/+mb1uG0K/7hl/tz/r3u4bEFTKmsQdV8z5jRWvO3P9tLe3+QN21umvF5vC47N2xs5OE9ZwnHpp6yufNEL/f8fD/Xravnw9eu5tljPXmZSirmVy7BvQk4lXK7zbwvm/cCv5rNRc3W1mVVfPGtF3Dpiur5vAxxDqkv9/Dn17Xw6IHOnBc5GZm7J+3+loYAQN4HVe99rJX/2XGKj1y7mpvOWzSrc91y4WIGglGeODx5e4Ke4TAf+P4umqq8fPFtF3KbOYj70EunZ/X8Yv7ldUBVKfUOYCvwr1kev0sptUMptaOrKz+rCDOx2RRv3tKMw16U48Vihv7kiuUsrynj73++j0hs6kHD7uFIxsx9eU0ZLruNIx35q7v/ZMcpvvToYW7b0szHXrtm1ue7cnUt1T4X/ztF/fzhvWfpHArzH3dupsLrZEl1GZcsr+bBXflt4yDmXi7Rrx1YknK72bxvHKXU9cDfALdorcOZTqS1/rrWeqvWemtdXd1MrleIGXM77Pzd6zdwrGuE/3z8yKTHBiNxhsOxjDV3h93Gyjofh/MU3LuGwvzNz/bw6pZaPnfbeXnZ1N1pt/HGC5v45Z4zvOX/PcOj+zsy9qB5+VQ/1T4X56VMm3zzliaOdo3IhuALXC7B/QWgRSm1QinlAu4AtqUeoJTaDHwNI7B35v8yhciPa9fV8+YtTdz7eCvfe+5E1uOsvVNTFzClamkI5K0ss/1QJ9G45tM3rceZx2+bn7pxLZ99wwZO94f40+/u4Pb/90zaNMeX2wY4v7li3AfKzecvwuWw8eCu/PfHF3Nnyn9JWusY8GHgEeAA8BOt9T6l1D1KqVvMw/4V8AP3K6VeUkpty3I6IeaVUorP33Y+16+v5zMP7c1atujM0Hog1Zp6P+39wbxs3PHYgQ4WV3hYvygw63Ol8jjtvPuKFfz+k1fzsevXpK1cHQnHONI5xPnN41sclHucvG5DAz9/+UxO5StxbsopTdBaP6y1XqO1XqW1/kfzvs9orbeZP1+vtW7QWl9o/rll8jMKMX+cdhv/+UdbuHRFNZ+4fze/O5T+ZdPK3OuyZO5rGo1AfGSWM2ZC0ThPHunm2vX1eSnHZOKw27jzUqOy+uyxnuT9e9sHSGi4cEn6fPo3b2midyTC09IvfsGSEUdRkjxOO/e962JW1fn4/7btSytXZGoalmpNcsbM7Oruzx3rYTQS57r1DbM6z1TqAx5W1/uTq1+BZE19YuYO8KpVtbjsNp473pP2mFgYJLiLkuV3O/jE69Zyomc0bepf93AYpaDalz5bBmBpdRlux+xnzDx+sBOv087lK3NrNTAbl6+s4YVXepPtgXe39dNU6c04ruBx2tnUVM6OV/oKfl2iMCS4i5L2ug0NrF9Uzn9ubx3Xi6VrKExVmSvrAKfdplhV55/VoKrWmscOdHJlS+2crKa+fFUNo5F4cpvA3W39nN+cvcXBxcur2dM2UPDe9aIwJLiLkqaU4qPXreZ49wg/321k71prTvUFs9bbLWsa/Bw6OzTj+eAHzw7R3h/k+vX1M/r96brM/Hbw7NEeekcinOoNcsGS7P3ity6vJhJPzNm2giK/JLiLkve6DY2sbQjwH48foWc4zAd/sIsnDndx6crJVzi/alUtZwdDfPfZ7FMqJ/PYgQ4Arlk3N8G92udiXWOAZ4/1JLP3yTL3i5ZVAfDCK7k3IBPnDmm+Ikqezab4yHUtfOiHu7jqX39HKBrn7pvWcderV076e7df1Mwj+87yj788wJalVZw3SaDsHg7zdGs3L7cNsKLWx5alVfz2QCcXNFdQn6HFQaFcvqqGHz5/ks1L+lCKcYuXJqr2uVhV55O6+wIlwV0I4KZNjZzXVMFIOMa/37F50kBtsdkUX3jLBfyfe5/kQz/cxS8+ciXlHqMjaTSeYOeJPn53qIsnDnex/4wxv9xpV0TjY2Wcj+eh1cB0XL6yhm89/Qo/fuEkK2t9BMzrzebi5dX8au9ZEgmNbYZdKsX8kOAuBEagfuADl+O02aYVxKp8Lv7jj7bwtq89y3u+9QKLKr2c7B3laOcww+EYDpviomVVfPKGtVy5upZNTRW09wV58VQfRzqG+aNLlxbwVaW7dEUNShl9c16zZuoWIFuXV/PjF07R2jWcnP4pFgYJ7kKYptqvNJuLllXxmTds4PO/OkjXcJil1WW8aXMTV6yu4YrVtWnZ8dKaMpbWlOXjkqetoszJpsUV7Gkf4IIM89snunj5WN1dgvvCIsFdiDx45+XL+ePLlhVslWk+Xb6qhj3tA5MOplqWVpdR63ez45U+3n7pvG6wJqZJgrsQebIQAjvAnZcsJRSN57SBtlKKi5dXsePE2IyZ9v4gfrcjuZ2lODdJcBeixKyo9XHPrZtyPn6rOaj6g+dPsO2l0zxv7s26vKaMTU0VrKz1sajSS2OFh8tX1sj2lucICe5CiElZdfe/+dlemqu8fPKGtQDsaRvgxZP9/HLPGax1XNevb+C+d22d9nMc7Rpm+8FO3n7pMrwu+XDIBwnuQohJbVpcwd03rWN1nZ9r1tWnbdwdjSfoHArz3Wdf4Wu/P8bOE33JBVC50Frzl/fv5sWT/fzg+ZP86+3ns3W5bJE5W7JCVQgxKZtN8f6rVnH9hoa0wA5GC+WmSi8fubaFGp+LL/328LTO/9iBTl482c87L19GNJ7gLV97ln/85X7pJT9LEtyFEHnhczv4wNWreKq1m+eP5dYqOJHQfOE3h1hR6+PvXr+BX//Fa3j7pUv5xpPHeft9z9E5FCrwVRcvCe5CiLx5x2XLqA+4+bffHEZrTSKhef5YD/tOZ24+9vOXT3Pw7BAfe+0anHYbfreDf3jjedx752b2tA9wy388zUun+uf4VRQHCe5CiLzxOO186JrV/OGVXj794B6u+sJ23vb153jTfz3DI/vOjjs2Gk/wxd8eZv2icl5/3qJxj91ywWIe/MAVOOyKt37tWfZKZ8ppk+AuhMirOy5ZQlOll//ZcYpl1T6+8JYL2LConA98fyf37zgFwCvdI3zmob2c6BnlkzesydjyYcPich760BVUlTn56I9fJBiRvvLToWbai3q2tm7dqnfs2DEvzy2EKKzOwRBxrVlU4QWMzbjf//2dPHmkm01N5extH8Sm4LYtzfzL7edPugDsqSPdvOO/n+edly/LOD9/T9sAX/19K2+7eClX5dAvZ6FTSu3UWk8531SCuxBiToRjcf7qgZfZe3qQN21u4rYtzTRW5Nbu+B9+sZ/7njrOt959cbL/fTAS50uPHua+J49hRbGPXNvCR65ryTirp1hIcBdCFI1QNM4bv/I0ZwdDnNdUQTia4ETvCB2DYe68ZCkfu76Fz/36IA/uaufVLbX89c3rWdcYWDAtIaZDgrsQoqgc6Rji7x7aSziWwO2wEfA4ee+VK5LbB2qt+dEfTvHZbfuIxBM0VXq5dl09N2xs5LKV1Tiy7Ie70EhwF0KUpM6hEI8f6OSxg508daSbYDROjc/FjZsaaSj30DsSoX80whWra7n9ouYFl93nGtwXXvuBv/gLeOml+b4KIcQ5qh64w/yT0Jr+0Sg9IxH6RiIktMZuU9iUIhpPcLzcw/IaH3Neor/wQvjylwv6FAsvuAshRI5sSlHtc1Htc5HQ1n2ggVO9o5zuDxKMxGlp8OMqkrKNZeEF9wJ/2gkhilNq6FbAUmD37tO8+4HdRGIJti6v5vr19bz+/MUsrvTO01XmT3F9VAkhxDS84YLFPPyRV/Oha1YzFIrxTw8f5IYvPcHTrd3zfWmzJgOqQghhOt49wvu/t5OjXcN87rbzuf2i5vm+pDS5DqhK5i6EEKYVtT7u/8DlXLayhr+8fzf/9PABeobDWY9/5mg3f/2zPfxkxyk6Bs+tDpaSuQshxASRWILPPLSXH79wCpfDxhvOX8wdlyzhvKYKPE47Q6Eo//yrg/zw+ZM47Ypo3IijLfV+Ni4up6UhwLrGAFuXVVNRlt+9ZvM6z10pdSPw74AduE9r/bkJj7uB7wIXAT3A27TWr0x2TgnuQohzXWvnEN955gQ/3dXGaCSOw6ZY0xCgbzTC2cEQ771iBR9/3RpO9IzyxOEunjvWw+GOYdr7gwAoZexk9apVNWxZVsXmJZXUl+fWciGbvAV3pZQdOAy8FmgDXgDu1FrvTznmg8D5Wuv3K6XuAN6ktX7bZOeV4C6EWCgGQ1GePtLNnvYB9rQPEI4m+Kub1nLRsszbAQ6Gohw4Pchzx3p55mg3L57sJxI3dpZaVOHh7pvWceuFTTO6lnwG98uBz2qtbzBvfxpAa/3PKcc8Yh7zrFLKAZwF6vQkJ5fgLoQoFaFonP1nBnnpZD8vnernjkuW8KpVtTM6Vz5XqDYBp1JutwGXZjtGax1TSg0ANcDCn08khBCz5HHa2bK0ii1Lc984fLbmdLaMUuoupdQOpdSOrq6uuXxqIYQoKbkE93ZgScrtZvO+jMeYZZkKjIHVcbTWX9dab9Vab62rK/6m+kIIMV9yCe4vAC1KqRVKKRdGP55tE47ZBrzL/Pl24PHJ6u1CCCEKa8qau1lD/zDwCMZUyG9qrfcppe4BdmittwH/DXxPKdUK9GJ8AAghhJgnOTUO01o/DDw84b7PpPwcAt6S30sTQggxU9J+QAghipAEdyGEKEIS3IUQogjNW+MwpVQXcGKGv15LaS6QKsXXXYqvGUrzdZfia4bpv+5lWusp55LPW3CfDaXUjlyW3xabUnzdpfiaoTRfdym+Zijc65ayjBBCFCEJ7kIIUYQWanD/+nxfwDwpxdddiq8ZSvN1l+JrhgK97gVZcxdCCDG5hZq5CyGEmMSCC+5KqRuVUoeUUq1Kqbvn+3oKQSm1RCm1XSm1Xym1Tyn1UfP+aqXUb5VSR8z/zl1z6DmilLIrpV5USv3CvL1CKfW8+X7/j9m8rqgopSqVUg8opQ4qpQ4opS4vkff6Y+a/771KqR8ppTzF9n4rpb6plOpUSu1NuS/je6sM95qv/WWl1JbZPPeCCu7mln9fAW4CNgB3KqU2zO9VFUQM+ITWegNwGfAh83XeDTymtW4BHjNvF5uPAgdSbn8e+JLWejXQB7x3Xq6qsP4d+LXWeh1wAcbrL+r3WinVBHwE2Kq13oTRlPAOiu/9/jZw44T7sr23NwEt5p+7gK/O5okXVHAHLgFatdbHtNYR4MfArfN8TXmntT6jtd5l/jyE8T97E8Zr/Y552HeAN87PFRaGUqoZ+D/AfeZtBVwLPGAeUoyvuQJ4DUZnVbTWEa11P0X+XpscgNfcA6IMOEORvd9a6ycwOuWmyvbe3gp8VxueAyqVUotm+twLLbhn2vJvZrvMLhBKqeXAZuB5oEFrfcZ86CzQME+XVShfBj4FJMzbNUC/1jpm3i7G93sF0AV8yyxH3aeU8lHk77XWuh34AnASI6gPADsp/vcbsr+3eY1vCy24lxSllB/4KfAXWuvB1MfMzVCKZqqTUur1QKfWeud8X8sccwBbgK9qrTcDI0wowRTbew1g1plvxfhwWwz4SC9fFL1CvrcLLbjnsuVfUVBKOTEC+w+01g+ad3dYX9PM/3bO1/UVwBXALUqpVzDKbddi1KIrza/tUJzvdxvQprV+3rz9AEawL+b3GuB64LjWuktrHQUexPg3UOzvN2R/b/Ma3xZacM9ly78Fz6w1/zdwQGv9xZSHUrczfBfw0FxfW6ForT+ttW7WWi/HeF8f11q/HdiOsXUjFNlrBtBanwVOKaXWmnddB+yniN9r00ngMqVUmfnv3XrdRf1+m7K9t9uAd5qzZi4DBlLKN9OntV5Qf4CbgcPAUeBv5vt6CvQar8T4qvYy8JL552aMGvRjwBHgUaB6vq+1QK//auAX5s8rgT8ArcD9gHu+r68Ar/dCYIf5fv8vUFUK7zXw98BBYC/wPcBdbO838COMMYUoxre092Z7bwGFMRvwKLAHYybRjJ9bVqgKIUQRWmhlGSGEEDmQ4C6EEEVIgrsQQhQhCe5CCFGEJLgLIUQRkuAuhBBFSIK7EEIUIQnuQghRhP5/LMbeeu4LVPQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x105320780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from numpy.random import seed\n",
    "from numpy.random import normal\n",
    "from scipy.stats import ttest_ind\n",
    "from matplotlib import pyplot\n",
    "%matplotlib inline\n",
    "# generate results\n",
    "seed(1)\n",
    "n = 100\n",
    "values1 = normal(50, 1, n)\n",
    "values2 = normal(51, 10, n)\n",
    "# calculate p-values for different subsets of results\n",
    "pvalues = list()\n",
    "for i in range(1, n+1):\n",
    "\tvalue, p = ttest_ind(values1[0:i], values2[0:i], equal_var=False)\n",
    "\tpvalues.append(p)\n",
    "# plot p-values vs number of results in sample\n",
    "pyplot.plot(pvalues)\n",
    "# draw line at 95%, below which we reject H0\n",
    "pyplot.plot([0.05 for x in range(len(pvalues))], color='red')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the example creates a line plot of p-value vs sample size.\n",
    "\n",
    "We can see that for these two sets of results, the sample size must be about 90 before we have a 95% confidence that the means are significantly different (where the blue line intersects the red line).\n",
    "\n",
    "## Compare Means for Non-Gaussian Results\n",
    "We cannot use the Student t-test or the Welch’s t-test if our data is not Gaussian.\n",
    "\n",
    "An alternative statistical significance test we can use for non-Gaussian data is called the Kolmogorov-Smirnov test.\n",
    "\n",
    "In SciPy, this is called the ks_2samp() function.\n",
    "\n",
    "This test can be used on Gaussian data, but will have less statistical power and may require large samples.\n",
    "\n",
    "We can demonstrate the calculation of statistical significance on two sets of results with non-Gaussian distributions. We can generate two sets of results with overlapping uniform distributions (50 to 60 and 55 to 65). These sets of results will have different mean values of about 55 and 60 respectively.\n",
    "\n",
    "The code below generates the two sets of 100 results and uses the Kolmogorov-Smirnov test to demonstrate that the difference between the population means is statistically significant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47 2.1682585673653708e-10\n",
      "Samples are likely drawn from different distributions (reject H0)\n"
     ]
    }
   ],
   "source": [
    "from numpy.random import seed\n",
    "from numpy.random import randint\n",
    "from scipy.stats import ks_2samp\n",
    "# generate results\n",
    "seed(1)\n",
    "n = 100\n",
    "values1 = randint(50, 60, n)\n",
    "values2 = randint(55, 65, n)\n",
    "# calculate the significance\n",
    "value, pvalue = ks_2samp(values1, values2)\n",
    "print(value, pvalue)\n",
    "if pvalue > 0.05:\n",
    "\tprint('Samples are likely drawn from the same distributions (fail to reject H0)')\n",
    "else:\n",
    "\tprint('Samples are likely drawn from different distributions (reject H0)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the example prints the statistic and the p-value.\n",
    "\n",
    "The p-value is very small, suggesting a near certainty that the difference between the two populations is significant.\n",
    "\n",
    "## Summary\n",
    "In this tutorial, you discovered how you can use statistical significance tests to interpret machine learning results.\n",
    "\n",
    "You can use these tests to help you confidently choose one machine learning algorithm over another or one set of configuration parameters over another for the same algorithm.\n",
    "\n",
    "You learned:\n",
    "\n",
    "* How to use normality tests to check if your experimental results are Gaussian or not.\n",
    "* How to use statistical tests to check if the difference between mean results is significant for Gaussian data with the same and different variance.\n",
    "* How to use statistical tests to check if the difference between mean results is significant for non-Gaussian data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}