class10_Cross-Validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Data Mining:<br>Statistical Modeling and Learning from Data\n",
    "\n",
    "## Dr. Ciro Cattuto<br>Dr. Laetitia Gauvin<br>Dr. André Panisson\n",
    "\n",
    "### Exercises - Cross Validation and Regularization\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pylab inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load the Diabetes dataset\n",
    "\n",
    "The diabetes dataset consists of 10 physiological variables (age, sex, weight, blood pressure) measure on 442 patients, and an indication of disease progression after one year.\n",
    "\n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html#sklearn.datasets.load_diabetes\n",
    "\n",
    "Reference:\n",
    "   [http://web.stanford.edu/~hastie/Papers/LARS/LeastAngle_2002.pdf]\n",
    "\n",
    "Load the features in a variable with name $\\mathbf{X}$ and the target in a variable with name $\\mathbf{y}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_diabetes\n",
    "\n",
    "diabetes = load_diabetes()\n",
    "X = diabetes.data\n",
    "y = diabetes.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Evaluate a Linear Regression model using 10-fold cross-validation\n",
    "\n",
    "Create a Linear Regression model and assign it to a variable with name **`lr`**.   \n",
    "http://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares\n",
    "\n",
    "Create a cross validation generator with 10 folds. Set the parameter **`shuffle`** to **`True`**, in order to shuffle the data before creating the folds. Set the parameter **`random_state`** to 0 to create always the same combination of folds.\n",
    "Assign it to a variable with name **`cv`**.   \n",
    "http://scikit-learn.org/stable/modules/cross_validation.html#k-fold\n",
    "\n",
    "Use the function **`cross_val_score`** to evaluate the model using the cross validator. This function returns a list of scores.   \n",
    "http://scikit-learn.org/stable/modules/cross_validation.html#computing-cross-validated-metrics\n",
    "\n",
    "Finally, print the mean and standard deviation of the scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Redo the same evaluation, this time using the Mean Squared Error as score function\n",
    "\n",
    "Check which score function is used by the validator by default.\n",
    "\n",
    "Create a new scorer function that calculates the Mean Squared Error. You can use the function **`mean_squared_error`** from sklearn:   \n",
    "http://scikit-learn.org/stable/modules/model_evaluation.html#the-scoring-parameter-defining-model-evaluation-rules\n",
    "\n",
    "Use again the function **`cross_val_score`** to evaluate the model using the cross validator and the new scorer.\n",
    "\n",
    "Finally, print the mean and standard deviation of the new scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print lr.score.__doc__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Evaluate a Lasso model using 10-fold cross-validation\n",
    "\n",
    "Create a Lasso model with default parameters and assign it to a variable with name **`lasso`**.   \n",
    "http://scikit-learn.org/stable/modules/linear_model.html#lasso\n",
    "\n",
    "Create a cross validation generator with 10 folds. Set the parameter **`shuffle`** to **`True`**, in order to shuffle the data before creating the folds. Set the parameter **`random_state`** to 0 to create always the same combination of folds.\n",
    "Assign it to a variable with name **`cv`**.   \n",
    "http://scikit-learn.org/stable/modules/cross_validation.html#k-fold\n",
    "\n",
    "Use the function **`cross_val_score`** to evaluate the model using the cross validator.\n",
    "Use the scorer function that calculates the Mean Squared Error to calculate the scores.   \n",
    "http://scikit-learn.org/stable/modules/cross_validation.html#computing-cross-validated-metrics\n",
    "\n",
    "Finally, print the mean and standard deviation of the scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the average scores of the Lasso model with the LinearRegression. Is it better? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Find the best regularization parameter for a Lasso model using 10-fold cross-validation\n",
    "\n",
    "Create a Lasso model with default parameters and assign it to a variable with name **`lasso`**.   \n",
    "http://scikit-learn.org/stable/modules/linear_model.html#lasso\n",
    "\n",
    "Create a list of values for the regularization parameter (alpha) starting from $10^{-4}$ to $10^{-0.5}$, spaced evenly on a log scale.\n",
    "\n",
    "For each value of alpha, set the model parameter **`alpha`** to this value, calculate the 10-fold cross-validation scores, and add the average score to a list with name **`scores`**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's plot the scores and print the best regularization parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure(figsize=(8, 6))\n",
    "semilogx(alphas, scores)\n",
    "ylabel('CV score (MSE)')\n",
    "xlabel('alpha')\n",
    "axhline(np.max(scores), linestyle='--', color='.5')\n",
    "\n",
    "print (np.max(scores))\n",
    "print ('Best alpha:', alphas[np.argmax(scores)])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}