notebooks/metrics_ex_02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e3bd0b54",
   "metadata": {},
   "source": [
    "# 📝 Exercise M7.03\n",
    "\n",
    "As with the classification metrics exercise, we will evaluate the regression\n",
    "metrics within a cross-validation framework to get familiar with the syntax.\n",
    "\n",
    "We will use the Ames house prices dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0254e4aa",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T08:13:50.044567Z",
     "start_time": "2021-06-24T08:13:49.853691Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "ames_housing = pd.read_csv(\"../datasets/house_prices.csv\")\n",
    "data = ames_housing.drop(columns=\"SalePrice\")\n",
    "target = ames_housing[\"SalePrice\"]\n",
    "data = data.select_dtypes(np.number)\n",
    "target /= 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f95d92b2",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">If you want a deeper overview regarding this dataset, you can refer to the\n",
    "Appendix - Datasets description section at the end of this MOOC.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee44faf7",
   "metadata": {},
   "source": [
    "The first step will be to create a linear regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1405ebe1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T08:14:11.442047Z",
     "start_time": "2021-06-24T08:14:11.440077Z"
    }
   },
   "outputs": [],
   "source": [
    "# Write your code here.\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "lr = LinearRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52ff7701",
   "metadata": {},
   "source": [
    "Then, use the `cross_val_score` to estimate the statistical performance of\n",
    "the model. Use a `KFold` cross-validation with 10 folds. Make the use of the\n",
    "$R^2$ score explicit by assigning the parameter `scoring` (even though it is\n",
    "the default score)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "acc8e0d9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T08:16:07.429144Z",
     "start_time": "2021-06-24T08:16:07.375861Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tests score 0.79 +/- 0.10\n"
     ]
    }
   ],
   "source": [
    "# Write your code here.\n",
    "from sklearn.model_selection import cross_val_score, KFold\n",
    "\n",
    "cv = KFold(n_splits=10)\n",
    "\n",
    "test_score = cross_val_score(lr, data, target, cv=cv, scoring='r2')\n",
    "print(f'tests score {test_score.mean():0.02f} +/- {test_score.std():0.02f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d819515",
   "metadata": {},
   "source": [
    "Then, instead of using the $R^2$ score, use the mean absolute error. You need\n",
    "to refer to the documentation for the `scoring` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d3e35464",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T08:17:03.794050Z",
     "start_time": "2021-06-24T08:17:03.743627Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tests score 21.89 +/- -2.23\n"
     ]
    }
   ],
   "source": [
    "# Write your code here.\n",
    "test_score = cross_val_score(lr, data, target, cv=cv, scoring='neg_mean_absolute_error')\n",
    "print(f'tests score {-test_score.mean():0.02f} +/- {-test_score.std():0.02f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cfbeb2e",
   "metadata": {},
   "source": [
    "Finally, use the `cross_validate` function and compute multiple scores/errors\n",
    "at once by passing a list of scorers to the `scoring` parameter. You can\n",
    "compute the $R^2$ score and the mean absolute error for instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2a6ccbc1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T08:18:47.712862Z",
     "start_time": "2021-06-24T08:18:47.663121Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'fit_time': array([0.00356841, 0.002563  , 0.00250268, 0.00245285, 0.0023756 ,\n",
      "       0.00251913, 0.00349855, 0.00237823, 0.00275087, 0.002496  ]), 'score_time': array([0.00154924, 0.00171065, 0.00153542, 0.00150681, 0.00183582,\n",
      "       0.00164604, 0.00184298, 0.00187278, 0.00164914, 0.00166726]), 'test_r2': array([0.84390289, 0.85497435, 0.88752303, 0.74951104, 0.81698014,\n",
      "       0.82013355, 0.81554085, 0.81452472, 0.50115778, 0.83330693]), 'test_neg_mean_absolute_error': array([-20.48049905, -21.38003105, -21.26831487, -22.86887664,\n",
      "       -24.79955736, -18.95827641, -20.11793792, -20.5040172 ,\n",
      "       -26.76774564, -21.77871056])}\n"
     ]
    }
   ],
   "source": [
    "# Write your code here.\n",
    "from sklearn.model_selection import cross_validate\n",
    "\n",
    "test_score = cross_validate(lr, data, target, cv=cv, scoring=['r2', 'neg_mean_absolute_error'])\n",
    "print(test_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b13fdb7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}