-
Notifications
You must be signed in to change notification settings - Fork 856
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
6 changed files
with
344 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
329 changes: 329 additions & 0 deletions
329
docs/sources/user_guide/evaluate/paired_ttest_kfold_cv.ipynb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,329 @@ | ||
{ | ||
"cells": [ | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 1, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"%matplotlib inline" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"# K-fold cross-validated paired t-test" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"K-fold paired *t* test procedure to compare the performance of two models" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"> `from mlxtend.evaluate import paired_ttest_kfold_cv` " | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"## Overview" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"K-fold cross-validated paired t-test procedure is a common method for comparing the performance of two models (classifiers or regressors) and addresses some of the drawbacks of the [resampled t-test procedure](paired_ttest_resampled.md); however, this method has still the problem that the training sets overlap and is not recommended to be used in practice [1], and techniques such as the [`paired_ttest_5times2_cv`](paired_ttest_5times2_cv.md) should be used instead.\n", | ||
"\n", | ||
"To explain how this method works, let's consider to estimator (e.g., classifiers) A and B. Further, we have a labeled dataset *D*. In the common hold-out method, we typically split the dataset into 2 parts: a training and a test set. In the k-fold cross-validated paired t-test procedure, we split the test set into *k* parts of equal size, and each of these parts is then used for testing while the remaining *k-1* parts (joined together) are used for training a classifier or regressor (i.e., the standard k-fold cross-validation procedure).\n", | ||
"\n", | ||
"In each k-fold cross-validation iteration, we then compute the difference in performance between A and B in each so that we obtain *k* difference measures. Now, by making the assumption that these *k* differences were independently drawn and follow an approximately normal distribution, we can compute the following *t* statistic with *k-1* degrees of freedom according to Student's *t* test, under the null hypothesis that the models A and B have equal performance:\n", | ||
"\n", | ||
"$$t = \\frac{\\overline{p} \\sqrt{k}}{\\sqrt{\\sum_{i=1}^{k}(p^{(i) - \\overline{p}})^2 / (k-1)}}.$$\n", | ||
"\n", | ||
"Here, $p^{(i)}$ computes the difference between the model performances in the $i$th iteration, $p^{(i)} = p^{(i)}_A - p^{(i)}_B$, and $\\overline{p}$ represents the average difference between the classifier performances, $\\overline{p} = \\frac{1}{k} \\sum^k_{i=1} p^{(i)}$.\n", | ||
"\n", | ||
"Once we computed the *t* statistic we can compute the p value and compare it to our chosen significance level, e.g., $\\alpha=0.05$. If the p value is smaller than $\\alpha$, we reject the null hypothesis and accept that there is a significant difference in the two models.\n", | ||
"\n", | ||
"\n", | ||
"The problem with this method, and the reason why it is not recommended to be used in practice, is that it violates an assumption of Student's *t* test [1]:\n", | ||
"\n", | ||
"- the difference between the model performances ($p^{(i)} = p^{(i)}_A - p^{(i)}_B$) are not normal distributed because $p^{(i)}_A$ and $p^{(i)}_B$ are not independent\n", | ||
"- the $p^{(i)}$'s themselves are not independent because training sets overlap" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"### References\n", | ||
"\n", | ||
"- [1] Dietterich TG (1998) Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. *Neural Comput* 10:1895–1923." | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"## Example 1 - Paired resampled t test" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"Assume we want to compare two classification algorithms, logistic regression and a decision tree algorithm:" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 2, | ||
"metadata": {}, | ||
"outputs": [ | ||
{ | ||
"name": "stdout", | ||
"output_type": "stream", | ||
"text": [ | ||
"Logistic regression accuracy: 97.37%\n", | ||
"Decision tree accuracy: 94.74%\n" | ||
] | ||
} | ||
], | ||
"source": [ | ||
"from sklearn.linear_model import LogisticRegression\n", | ||
"from sklearn.tree import DecisionTreeClassifier\n", | ||
"from mlxtend.data import iris_data\n", | ||
"from sklearn.model_selection import train_test_split\n", | ||
"\n", | ||
"\n", | ||
"X, y = iris_data()\n", | ||
"clf1 = LogisticRegression(random_state=1)\n", | ||
"clf2 = DecisionTreeClassifier(random_state=1)\n", | ||
"\n", | ||
"X_train, X_test, y_train, y_test = \\\n", | ||
" train_test_split(X, y, test_size=0.25,\n", | ||
" random_state=123)\n", | ||
"\n", | ||
"score1 = clf1.fit(X_train, y_train).score(X_test, y_test)\n", | ||
"score2 = clf2.fit(X_train, y_train).score(X_test, y_test)\n", | ||
"\n", | ||
"print('Logistic regression accuracy: %.2f%%' % (score1*100))\n", | ||
"print('Decision tree accuracy: %.2f%%' % (score2*100))" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"Note that these accuracy values are not used in the paired t-test procedure as new test/train splits are generated during the resampling procedure, the values above are just serving the purpose of intuition.\n", | ||
"\n", | ||
"Now, let's assume a significance threshold of $\\alpha=0.05$ for rejecting the null hypothesis that both algorithms perform equally well on the dataset and conduct the k-fold cross-validated t-test:" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 3, | ||
"metadata": {}, | ||
"outputs": [ | ||
{ | ||
"name": "stdout", | ||
"output_type": "stream", | ||
"text": [ | ||
"t statistic: -1.861\n", | ||
"p value: 0.096\n" | ||
] | ||
} | ||
], | ||
"source": [ | ||
"from mlxtend.evaluate import paired_ttest_kfold_cv\n", | ||
"\n", | ||
"\n", | ||
"t, p = paired_ttest_kfold_cv(estimator1=clf1,\n", | ||
" estimator2=clf2,\n", | ||
" X=X, y=y,\n", | ||
" random_seed=1)\n", | ||
"\n", | ||
"print('t statistic: %.3f' % t)\n", | ||
"print('p value: %.3f' % p)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"Since $p > t$, we cannot reject the null hypothesis and may conclude that the performance of the two algorithms is not significantly different. \n", | ||
"\n", | ||
"While it is generally not recommended to apply statistical tests multiple times without correction for multiple hypothesis testing, let us take a look at an example where the decision tree algorithm is limited to producing a very simple decision boundary that would result in a relatively bad performance:" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 4, | ||
"metadata": {}, | ||
"outputs": [ | ||
{ | ||
"name": "stdout", | ||
"output_type": "stream", | ||
"text": [ | ||
"Decision tree accuracy: 63.16%\n", | ||
"t statistic: 13.491\n", | ||
"p value: 0.000\n" | ||
] | ||
} | ||
], | ||
"source": [ | ||
"clf2 = DecisionTreeClassifier(random_state=1, max_depth=1)\n", | ||
"\n", | ||
"score2 = clf2.fit(X_train, y_train).score(X_test, y_test)\n", | ||
"print('Decision tree accuracy: %.2f%%' % (score2*100))\n", | ||
"\n", | ||
"\n", | ||
"t, p = paired_ttest_kfold_cv(estimator1=clf1,\n", | ||
" estimator2=clf2,\n", | ||
" X=X, y=y,\n", | ||
" random_seed=1)\n", | ||
"\n", | ||
"print('t statistic: %.3f' % t)\n", | ||
"print('p value: %.3f' % p)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"Assuming that we conducted this test also with a significance level of $\\alpha=0.05$, we can reject the null-hypothesis that both models perform equally well on this dataset, since the p-value ($p < 0.001$) is smaller than $\\alpha$." | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"## API" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 5, | ||
"metadata": {}, | ||
"outputs": [ | ||
{ | ||
"name": "stdout", | ||
"output_type": "stream", | ||
"text": [ | ||
"## paired_ttest_kfold_cv\n", | ||
"\n", | ||
"*paired_ttest_kfold_cv(estimator1, estimator2, X, y, cv=10, scoring=None, shuffle=False, random_seed=None)*\n", | ||
"\n", | ||
"Implements the k-fold paired t test procedure\n", | ||
"to compare the performance of two models.\n", | ||
"\n", | ||
"**Parameters**\n", | ||
"\n", | ||
"- `estimator1` : scikit-learn classifier or regressor\n", | ||
"\n", | ||
"\n", | ||
"\n", | ||
"- `estimator2` : scikit-learn classifier or regressor\n", | ||
"\n", | ||
"\n", | ||
"\n", | ||
"- `X` : {array-like, sparse matrix}, shape = [n_samples, n_features]\n", | ||
"\n", | ||
" Training vectors, where n_samples is the number of samples and\n", | ||
" n_features is the number of features.\n", | ||
"\n", | ||
"\n", | ||
"- `y` : array-like, shape = [n_samples]\n", | ||
"\n", | ||
" Target values.\n", | ||
"\n", | ||
"\n", | ||
"- `cv` : int (default: 10)\n", | ||
"\n", | ||
" Number of splits and iteration for the\n", | ||
" cross-validation procedure\n", | ||
"\n", | ||
"\n", | ||
"- `scoring` : str, callable, or None (default: None)\n", | ||
"\n", | ||
" If None (default), uses 'accuracy' for sklearn classifiers\n", | ||
" and 'r2' for sklearn regressors.\n", | ||
" If str, uses a sklearn scoring metric string identifier, for example\n", | ||
" {accuracy, f1, precision, recall, roc_auc} for classifiers,\n", | ||
" {'mean_absolute_error', 'mean_squared_error'/'neg_mean_squared_error',\n", | ||
" 'median_absolute_error', 'r2'} for regressors.\n", | ||
" If a callable object or function is provided, it has to be conform with\n", | ||
" sklearn's signature ``scorer(estimator, X, y)``; see\n", | ||
" http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html\n", | ||
" for more information.\n", | ||
"\n", | ||
"\n", | ||
"- `shuffle` : bool (default: True)\n", | ||
"\n", | ||
" Whether to shuffle the dataset for generating\n", | ||
" the k-fold splits.\n", | ||
"\n", | ||
"\n", | ||
"- `random_seed` : int or None (default: None)\n", | ||
"\n", | ||
" Random seed for shuffling the dataset\n", | ||
" for generating the k-fold splits.\n", | ||
" Ignored if shuffle=False.\n", | ||
"\n", | ||
"**Returns**\n", | ||
"\n", | ||
"- `t` : float\n", | ||
"\n", | ||
" The t-statistic\n", | ||
"\n", | ||
"\n", | ||
"- `pvalue` : float\n", | ||
"\n", | ||
" Two-tailed p-value.\n", | ||
" If the chosen significance level is larger\n", | ||
" than the p-value, we reject the null hypothesis\n", | ||
" and accept that there are significant differences\n", | ||
" in the two compared models.\n", | ||
"\n", | ||
"\n" | ||
] | ||
} | ||
], | ||
"source": [ | ||
"with open('../../api_modules/mlxtend.evaluate/paired_ttest_kfold_cv.md', 'r') as f:\n", | ||
" s = f.read() \n", | ||
"print(s)" | ||
] | ||
} | ||
], | ||
"metadata": { | ||
"anaconda-cloud": {}, | ||
"kernelspec": { | ||
"display_name": "Python 3", | ||
"language": "python", | ||
"name": "python3" | ||
}, | ||
"language_info": { | ||
"codemirror_mode": { | ||
"name": "ipython", | ||
"version": 3 | ||
}, | ||
"file_extension": ".py", | ||
"mimetype": "text/x-python", | ||
"name": "python", | ||
"nbconvert_exporter": "python", | ||
"pygments_lexer": "ipython3", | ||
"version": "3.6.3" | ||
} | ||
}, | ||
"nbformat": 4, | ||
"nbformat_minor": 1 | ||
} |
Oops, something went wrong.