notebooks/linear_models_sol_04.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \ud83d\udcc3 Solution for Exercise M4.04\n",
    "\n",
    "In the previous notebook, we saw the effect of applying some regularization\n",
    "on the coefficient of a linear model.\n",
    "\n",
    "In this exercise, we will study the advantage of using some regularization\n",
    "when dealing with correlated features.\n",
    "\n",
    "We will first create a regression dataset. This dataset will contain 2,000\n",
    "samples and 5 features from which only 2 features will be informative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_regression\n",
    "\n",
    "data, target, coef = make_regression(\n",
    "    n_samples=2_000,\n",
    "    n_features=5,\n",
    "    n_informative=2,\n",
    "    shuffle=False,\n",
    "    coef=True,\n",
    "    random_state=0,\n",
    "    noise=30,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When creating the dataset, `make_regression` returns the true coefficient\n",
    "used to generate the dataset. Let's plot this information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "feature_names = [f\"Features {i}\" for i in range(data.shape[1])]\n",
    "coef = pd.Series(coef, index=feature_names)\n",
    "coef.plot.barh()\n",
    "coef"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a `LinearRegression` regressor and fit on the entire dataset and\n",
    "check the value of the coefficients. Are the coefficients of the linear\n",
    "regressor close to the coefficients used to generate the dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "linear_regression = LinearRegression()\n",
    "linear_regression.fit(data, target)\n",
    "linear_regression.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names = [f\"Features {i}\" for i in range(data.shape[1])]\n",
    "coef = pd.Series(linear_regression.coef_, index=feature_names)\n",
    "_ = coef.plot.barh()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the coefficients are close to the coefficients used to generate\n",
    "the dataset. The dispersion is indeed cause by the noise injected during the\n",
    "dataset generation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, create a new dataset that will be the same as `data` with 4 additional\n",
    "columns that will repeat twice features 0 and 1. This procedure will create\n",
    "perfectly correlated features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "data = np.concatenate([data, data[:, [0, 1]], data[:, [0, 1]]], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit again the linear regressor on this new dataset and check the\n",
    "coefficients. What do you observe?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_regression = LinearRegression()\n",
    "linear_regression.fit(data, target)\n",
    "linear_regression.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names = [f\"Features {i}\" for i in range(data.shape[1])]\n",
    "coef = pd.Series(linear_regression.coef_, index=feature_names)\n",
    "_ = coef.plot.barh()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the coefficient values are far from what one could expect.\n",
    "By repeating the informative features, one would have expected these\n",
    "coefficients to be similarly informative.\n",
    "\n",
    "Instead, we see that some coefficients have a huge norm ~1e14. It indeed\n",
    "means that we try to solve an mathematical ill-posed problem. Indeed, finding\n",
    "coefficients in a linear regression involves inverting the matrix\n",
    "`np.dot(data.T, data)` which is not possible (or lead to high numerical\n",
    "errors)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a ridge regressor and fit on the same dataset. Check the coefficients.\n",
    "What do you observe?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "ridge = Ridge()\n",
    "ridge.fit(data, target)\n",
    "ridge.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "coef = pd.Series(ridge.coef_, index=feature_names)\n",
    "_ = coef.plot.barh()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the penalty applied on the weights give a better results: the\n",
    "values of the coefficients do not suffer from numerical issues. Indeed, the\n",
    "matrix to be inverted internally is `np.dot(data.T, data) + alpha * I`.\n",
    "Adding this penalty `alpha` allow the inversion without numerical issue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you find the relationship between the ridge coefficients and the original\n",
    "coefficients?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ridge.coef_[:5] * 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeating three times each informative features induced to divide the\n",
    "ridge coefficients by three."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition tip alert alert-warning\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Tip</p>\n",
    "<p class=\"last\">We always advise to use l2-penalized model instead of non-penalized model\n",
    "in practice. In scikit-learn, <tt class=\"docutils literal\">LogisticRegression</tt> applies such penalty\n",
    "by default. However, one needs to use <tt class=\"docutils literal\">Ridge</tt> (and even <tt class=\"docutils literal\">RidgeCV</tt> to tune\n",
    "the parameter <tt class=\"docutils literal\">alpha</tt>) instead of <tt class=\"docutils literal\">LinearRegression</tt>.</p>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}