jupyter_english/tutorials/statistical_hypothesis_testing_in_python_panin_kirill.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "<img src=\"../../img/ods_stickers.jpg\" />\n",
    "    \n",
    "## [mlcourse.ai](https://mlcourse.ai) – Open Machine Learning Course \n",
    "## <center >  Tutorial: Statistical Hypothesis Testing in Python  \n",
    "#### <center >Author: Kirill Panin, @Kirill Panin  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " TOC:  \n",
    "[Statistical Hypothesis Tests](#hyptests):  \n",
    "  * [Normality Tests](#normality_t):  \n",
    "      * [Shapiro-Wilk Test](#norm1)\n",
    "      * [D’Agostino’s K^2 Test](#norm2)\n",
    "      * [Anderson-Darling Test](#norm3)\n",
    "      * [Visual Normality Checks](#norm4):  \n",
    "          * [Histogram](#histplot)\n",
    "          * [QQplot](#qqplot)\n",
    "  * [Correlation Tests](#cor_t):\n",
    "      * [Pearson’s Correlation Coefficient](#cor1)\n",
    "      * [Spearman’s Rank Correlation](#cor2)\n",
    "      * [Kendall’s Rank Correlation](#cor3)\n",
    "  * [Parametric Statistical Hypothesis Tests](#par_t)\n",
    "      * [Student’s t-test](#par1)\n",
    "      * [Paired Student’s t-test](#par2)\n",
    "      * [Analysis of Variance Test (ANOVA)](#par3)\n",
    "  * [Nonparametric Statistical Hypothesis Tests](#non_param_t)\n",
    "      * [Mann-Whitney U Test](#nonpar1)\n",
    "      * [Wilcoxon Signed-Rank Test](#nonpar2)\n",
    "      * [Kruskal-Wallis H Test](#nonpar3)\n",
    "      * [Friedman Test](#nonpar4)\n",
    "      * [Kolmogorov–Smirnov Test](#nonpar5)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "Hypothesis Testing so important for Data Scientist, because you should know how to check distributions and relations between data.\n",
    "\n",
    "In this tutorial, we will learn statistical hypothesis tests that you may need to use in your machine learning career. And after completing this tutorial, you will know how testing hypothesis for:  \n",
    "  \n",
    "  * _types of hypothesis tests_\n",
    "  * _normality checking_\n",
    "  * _relationships checking between variables_\n",
    "  * _differences between samples_\n",
    "  * _how to interpret the test result_\n",
    "  * _ how to implement the test using the Python_\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, you discovered the key statistical hypothesis tests that you may need to use in a machine learning project.\n",
    "\n",
    "Specifically, you learned:\n",
    "\n",
    "The types of tests to use in different circumstances, such as normality checking, relationships between variables, and differences between samples. The key assumptions for each test and how to interpret the test result. How to implement the test using the Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpretation of a Test  \n",
    "Before you can apply the statistical tests, you must know how to interpret the results.\n",
    "\n",
    "Each test will return at least two things:\n",
    "\n",
    "* __Statistic:__ A quantity calculated by the test that can be interpreted in the context of the test via comparing it to critical values from the distribution of the test statistic.\n",
    "* __$p-value$:__ Used to interpret the test, in first case wheher the sample will drawn from a Gaussian distribution.  \n",
    "\n",
    "Each test calculates a test-specific statistic. This statistic can aid in the interpretation of the result, although it may require a deeper proficiency with statistics and a deeper knowledge of the specific statistical test. Instead, the _p-value_ can be used to quickly and accurately interpret the statistic in practical applications.\n",
    "\n",
    "At first part we'll be testing data that the sample was drawn from a Gaussian distribution. Technically this is called the null hypothesis, or H0. A threshold level is chosen called alpha, typically 5% (or 0.05), that is used to interpret the p-value. Interpret the p value as follows:  \n",
    "\n",
    "* __$p <= alpha$__: reject H0, not normal.  \n",
    "* __$p > alpha$__: fail to reject H0, normal.  \n",
    "\n",
    "This means that, in general, __we are seeking results with a larger p-value to confirm that our sample was likely drawn from a Gaussian distribution__.\n",
    "\n",
    "A result above 5% __DOESN'T MEAN__ that the null hypothesis is __true__. It means that it is very likely true given available evidence. The p-value is not the probability of the data fitting a Gaussian distribution; it can be thought of as a value that helps us interpret the statistical test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Test Data <a id='gd'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "normal = np.random.standard_normal(\n",
    "    1000\n",
    ")  # Draw random samples from a normal (Gaussian) distribution.\n",
    "uniform = np.random.uniform(size=1000)\n",
    "log_normal = np.random.lognormal(size=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.distplot(log_normal)\n",
    "sns.distplot(normal);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.distplot(uniform);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistical Hypothesis Tests<a id='hyptests'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normality Tests <a id='normality_t'></a>\n",
    "This section lists statistical tests that you can use to check if your data has a Gaussian distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shapiro-Wilk Test  <a id='norm1'></a>\n",
    "\n",
    "Tests whether a data sample has a Gaussian distribution(Normal).\n",
    "\n",
    "Assumptions\n",
    "\n",
    "Observations in each sample are independent and identically distributed (iid).  \n",
    "Interpretation\n",
    "\n",
    "* H0: the sample has a Gaussian distribution.  \n",
    "* H1: the sample does not have a Gaussian distribution.  \n",
    "\n",
    "Returns the test statistic and the p-value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import shapiro\n",
    "\n",
    "shapiro(normal), shapiro(log_normal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [scipy.stats.shapiro](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.shapiro.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### D’Agostino’s $K^2$ Test <a id='norm2'></a>\n",
    "The D’Agostino’s K^2 test calculates summary statistics from the data, namely kurtosis and skewness, to determine if the data distribution departs from the normal distribution, named for Ralph D’Agostino.\n",
    "\n",
    "* __Skew__ is a quantification of how much a distribution is pushed left or right, a measure of asymmetry in the distribution.\n",
    "* __Kurtosis__ quantifies how much of the distribution is in the tail. It is a simple and commonly used statistical test for normality.  \n",
    "\n",
    "The D’Agostino’s K^2 test is available via the normaltest() SciPy function and returns the test statistic and the p-value.\n",
    "\n",
    "The complete example of the D’Agostino’s K^2 test on the dataset is listed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import normaltest\n",
    "\n",
    "stat, p = normaltest(normal)\n",
    "print(\"Statistics=%.3f, p=%.3f\" % (stat, p))\n",
    "\n",
    "alpha = 0.05\n",
    "if p > alpha:\n",
    "    print(\"Sample looks Gaussian (fail to reject H0)\")\n",
    "else:\n",
    "    print(\"Sample does not look Gaussian (reject H0)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import normaltest\n",
    "\n",
    "stat, p = normaltest(log_normal)\n",
    "print(\"Statistics=%.3f, p=%.3f\" % (stat, p))\n",
    "\n",
    "alpha = 0.05\n",
    "if p > alpha:\n",
    "    print(\"Sample looks Gaussian (fail to reject H0)\")\n",
    "else:\n",
    "    print(\"Sample does not look Gaussian (reject H0)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [scipy.stats.normaltest](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.normaltest.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Anderson-Darling Test <a id='norm3'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* H0: the sample has a Gaussian distribution.  \n",
    "* H1: the sample does not have a Gaussian distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import anderson\n",
    "\n",
    "result = anderson(normal)\n",
    "print(\"Statistic: %.3f\" % result.statistic)\n",
    "p = 0\n",
    "for i in range(len(result.critical_values)):\n",
    "    sl, cv = result.significance_level[i], result.critical_values[i]\n",
    "    if result.statistic < result.critical_values[i]:\n",
    "        print(\"%.3f: %.3f, data looks normal (fail to reject H0)\" % (sl, cv))\n",
    "    else:\n",
    "        print(\"%.3f: %.3f, data does not look normal (reject H0)\" % (sl, cv))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import anderson\n",
    "\n",
    "result = anderson(uniform)\n",
    "print(\"Statistic: %.3f\" % result.statistic)\n",
    "p = 0\n",
    "for i in range(len(result.critical_values)):\n",
    "    sl, cv = result.significance_level[i], result.critical_values[i]\n",
    "    if result.statistic < result.critical_values[i]:\n",
    "        print(\"%.3f: %.3f, data looks normal (fail to reject H0)\" % (sl, cv))\n",
    "    else:\n",
    "        print(\"%.3f: %.3f, data does not look normal (reject H0)\" % (sl, cv))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [scipy.stats.anderson](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.anderson.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Visual Normality Checks <a id='norm4'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Histogram Plot <a id='histplot'></a>\n",
    "A simple and commonly used plot to quickly check the distribution of a sample of data is the histogram. A sample of data has a Gaussian distribution of the histogram plot, showing the familiar bell shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Histogram Plot Gaussian distribution\n",
    "sns.distplot(normal);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Histogram Plot Uniform distribution\n",
    "sns.distplot(uniform);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quantile-Quantile Plot <a id='qqplot'></a>\n",
    "Another popular plot for checking the distribution of a data sample is the quantile-quantile plot, Q-Q plot, or QQ plot for short.\n",
    "\n",
    "This plot generates its own sample of the idealized distribution that we are comparing with, in this case the Gaussian distribution. The idealized samples are divided into groups (e.g. 5), called quantiles. Each data point in the sample is paired with a similar member from the idealized distribution at the same cumulative distribution.\n",
    "\n",
    "The resulting points are plotted as a scatter plot with the idealized value on the x-axis and the data sample on the y-axis.\n",
    "\n",
    "A perfect match for the distribution will be shown by a line of dots on a 45-degree angle from the bottom left of the plot to the top right. Often a line is drawn on the plot to help make this expectation clear. Deviations by the dots from the line shows a deviation from the expected distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "fig = sm.qqplot(normal, line=\"s\")\n",
    "plt.show()  # Gaussian distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = sm.qqplot(log_normal, line=\"s\")\n",
    "plt.show()  # Log Normal distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Correlation Tests <a id='cor_t'></a>\n",
    "This section lists statistical tests that you can use to check if two samples are related."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.random.normal(0, 1, 1000)\n",
    "y = (3 * x) - np.random.normal(0, 2, 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Pearson’s Correlation Coefficient <a id='cor1'></a>\n",
    "Tests whether two samples have a linear relationship.  \n",
    "* H0: the two samples are independent.\n",
    "* H1: there is a dependency between the samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import pearsonr\n",
    "\n",
    "corr, p = pearsonr(x, y)\n",
    "corr, p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [scipy.stats.pearsonr](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spearman’s Rank Correlation  <a id='cor2'></a>\n",
    "Tests whether two samples have a monotonic relationship. Observations in each sample can be ranked.\n",
    "* H0: the two samples are independent.\n",
    "* H1: there is a dependency between the samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import spearmanr\n",
    "\n",
    "corr, p = spearmanr(x, y)\n",
    "corr, p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [scipy.stats.spearmanr](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Kendall’s Rank Correlation <a id='cor3'></a> \n",
    "Tests whether two samples have a monotonic relationship. Like in previous"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import kendalltau\n",
    "\n",
    "corr, p = kendalltau(x, y)\n",
    "corr, p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [scipy.stats.kendalltau](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parametric Statistical Hypothesis Tests <a id='par_t'></a>\n",
    "This section lists statistical tests that you can use to compare data samples. Parametric statistical methods often mean those methods that assume the data samples have a Gaussian distribution.  Parametric statistical significance tests that quantify the difference between the means of two or more samples of data.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For __parametric statistical tests__ assume that a data sample was drawn from a specific population distribution.\n",
    "\n",
    "They often refer to statistical tests that assume the Gaussian distribution. Because it is so common for data to fit this distribution, parametric statistical methods are more commonly used.\n",
    "\n",
    "A typical question we may have about two or more samples of data is whether they have the same distribution. Parametric statistical significance tests are those statistical methods that assume data comes from the same Gaussian distribution, that is a data distribution with the same mean and standard deviation: the parameters of the distribution.\n",
    "\n",
    "The p-value can be thought of as the probability of observing the two data samples given the base assumption (null hypothesis) that the two samples were drawn from a population with the same distribution.\n",
    "\n",
    "__p <= alpha__: reject null hypothesis, different distribution.  \n",
    "__p > alpha__: fail to reject null hypothesis, same distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x1 = 3 * np.random.randn(1000) + 20\n",
    "x2 = 3 * np.random.randn(1000) + 21\n",
    "print(\"x1: mean=%.3f stdv=%.3f\" % (np.mean(x1), np.std(x1)))\n",
    "print(\"x2: mean=%.3f stdv=%.3f\" % (np.mean(x2), np.std(x2)))\n",
    "sns.distplot(x1)\n",
    "sns.distplot(x2);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Student’s t-test <a id='par1'></a>\n",
    "Tests whether the means of two independent samples are significantly different.\n",
    "One of the most commonly used t tests is the independent samples t test. You use this test when you want to compare the means of two independent samples on a given variable\n",
    "\n",
    "Assumptions\n",
    "\n",
    "* Observations in each sample are independent and identically distributed (iid).\n",
    "* Observations in each sample are normally distributed.\n",
    "* Observations in each sample have the same variance.  \n",
    "\n",
    "Interpretation\n",
    "\n",
    "* H0: the means of the samples are equal.\n",
    "* H1: the means of the samples are unequal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import ttest_ind\n",
    "\n",
    "stat, p = ttest_ind(x1, x2)\n",
    "print(\"P value:\", p)\n",
    "alpha = 0.05\n",
    "if p > alpha:\n",
    "    print(\"Same distributions (fail to reject H0)\")\n",
    "else:\n",
    "    print(\"Different distributions (reject H0)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The interpretation of the statistic finds that the sample means are different, with a significance of at least 5%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [scipy.stats.ttest_ind](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Paired Student’s t-test <a id='par2'></a>\n",
    "Tests whether the means of two paired samples are significantly different. The paired Student’s t-test for quantifying the difference between the mean of two dependent data samples.\n",
    "\n",
    "Assumptions\n",
    "\n",
    "* Observations in each sample are independent and identically distributed (iid).\n",
    "* Observations in each sample are normally distributed.\n",
    "* Observations in each sample have the same variance.\n",
    "* Observations across each sample are paired.\n",
    "\n",
    "Interpretation\n",
    "\n",
    "* H0: the means of the samples are equal.\n",
    "* H1: the means of the samples are unequal.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import ttest_rel\n",
    "\n",
    "stat, p = ttest_rel(x1, x2)\n",
    "print(\"P value:\", p)\n",
    "alpha = 0.05\n",
    "if p > alpha:\n",
    "    print(\"Same distributions (fail to reject H0)\")\n",
    "else:\n",
    "    print(\"Different distributions (reject H0)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The interpretation of the result suggests that the samples have different means and therefore different distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [scipy.stats.ttest_ind](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_rel.html)  \n",
    " * [Wikipedia](https://en.wikipedia.org/wiki/Student%27s_t-test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Analysis of Variance Test (ANOVA) <a id='par3'></a> \n",
    "Tests whether the means of two or more independent samples are significantly different. The ANOVA and repeated measures ANOVA for checking the similarity or difference between the means of 2 or more data samples.\n",
    "\n",
    "* H0: the means of the samples are equal.\n",
    "* H1: one or more of the means of the samples are unequal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import f_oneway\n",
    "\n",
    "stat, p = f_oneway(x1, x2)\n",
    "print(\"P value:\", p)\n",
    "alpha = 0.05\n",
    "if p > alpha:\n",
    "    print(\"Same distributions (fail to reject H0)\")\n",
    "else:\n",
    "    print(\"Different distributions (reject H0)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [scipy.stats.f_oneway](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.f_oneway.html)  \n",
    " * [Wikipedia](https://en.wikipedia.org/wiki/Analysis_of_variance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nonparametric Statistical Hypothesis Tests <a id='non_param_t'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mann-Whitney U Test <a id='nonpar1'></a>\n",
    "Tests whether the distributions of two independent samples are equal or not.\n",
    "\n",
    "Assumptions\n",
    "\n",
    "* Observations in each sample are independent and identically distributed (iid).\n",
    "* Observations in each sample can be ranked.\n",
    "\n",
    "Interpretation\n",
    "\n",
    "* H0: the distributions of both samples are equal.\n",
    "* H1: the distributions of both samples are not equal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.random.normal(0, 1, 1000)\n",
    "y = np.random.uniform(0, 1, 1000)\n",
    "z = np.random.uniform(0, 1, 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import mannwhitneyu\n",
    "\n",
    "stat, p = mannwhitneyu(y, z)\n",
    "stat, p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [scipy.stats.mannwhitneyu](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html)  \n",
    " * [Wikipedia](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wilcoxon Signed-Rank Test <a id='nonpar2'></a>\n",
    "Tests whether the distributions of two paired samples are equal or not.\n",
    "\n",
    "Assumptions\n",
    "\n",
    "* Observations in each sample are independent and identically distributed (iid).\n",
    "* Observations in each sample can be ranked.\n",
    "* Observations across each sample are paired.  \n",
    "\n",
    "Interpretation\n",
    "\n",
    "* H0: the distributions of both samples are equal.\n",
    "* H1: the distributions of both samples are not equal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import wilcoxon\n",
    "\n",
    "stat, p = wilcoxon(z, y)\n",
    "stat, p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [scipy.stats.wilcoxon](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html)  \n",
    " * [Wikipedia](https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Kruskal-Wallis H Test <a id='nonpar3'></a> \n",
    "Tests whether the distributions of two or more independent samples are equal or not.\n",
    "\n",
    "Assumptions\n",
    "\n",
    "* Observations in each sample are independent and identically distributed (iid).\n",
    "* Observations in each sample can be ranked.\n",
    "Interpretation\n",
    "\n",
    "* H0: the distributions of all samples are equal.\n",
    "* H1: the distributions of one or more samples are not equal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "from scipy.stats import kruskal\n",
    "z1 = np.random.uniform(0,1,1000)\n",
    "stat, p = kruskal(y,z,z1)\n",
    "stat, p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [scipy.stats.kruskal](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kruskal.html)  \n",
    " * [Wikipedia](https://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Friedman Test <a id='nonpar4'></a>\n",
    "Tests whether the distributions of two or more paired samples are equal or not.\n",
    "\n",
    "Assumptions\n",
    "\n",
    "* Observations in each sample are independent and identically distributed (iid).\n",
    "* Observations in each sample can be ranked.\n",
    "* Observations across each sample are paired.\n",
    "\n",
    "Interpretation\n",
    "\n",
    "* H0: the distributions of all samples are equal.\n",
    "* H1: the distributions of one or more samples are not equal.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import friedmanchisquare\n",
    "\n",
    "stat, p = friedmanchisquare(y, z, z1)\n",
    "stat, p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kolmogorov–Smirnov test <a id='nonpar5'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depends on what you want to determine. If you are interested in determining whether the distributions have the __same mean__, and don't care about the rest, then __K-S__ is not best. K-S is very good for determining whether two samples in essence come from the same population."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy.stats import ks_2samp\n",
    "\n",
    "np.random.seed(123456)\n",
    "x = np.random.normal(0, 1, 1000)\n",
    "y = np.random.normal(0, 1, 1000)\n",
    "z = np.random.normal(1.1, 0.9, 1000)\n",
    "\n",
    "ks_2samp(x, y), ks_2samp(x, z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Under the null hypothesis the two distributions are identical. If the K-S statistic is small or the p-value is high (greater than the significance level, say 5%), then we cannot reject the hypothesis that the distributions of the two samples are the same. Conversely, we can reject the null hypothesis if the p-value is low."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information:\n",
    " * [Wikipedia](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, you discovered the key statistical hypothesis tests that you may need to use in a machine learning project.\n",
    "\n",
    "Specifically, you learned:\n",
    "\n",
    "The types of tests to use in different circumstances, such as normality checking, relationships between variables, and differences between samples.\n",
    "The key assumptions for each test and how to interpret the test result.\n",
    "How to implement the test using the Python "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats  \n",
    "https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/  \n",
    "https://machinelearningmastery.com/a-gentle-introduction-to-normality-tests-in-python/  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}