class07_Linear-Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Data Mining:<br>Statistical Modeling and Learning from Data\n",
    "\n",
    "## Dr. Ciro Cattuto<br>Dr. Laetitia Gauvin<br>Dr. André Panisson\n",
    "\n",
    "### Exercises - Linear Regression\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pylab inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## These data points will be used in the exercises:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = 10 # training examples\n",
    "x = linspace(1, 10, m).reshape(m, 1)\n",
    "y = array([ 3.76405235,  3.20015721,  4.57873798,  6.6408932 ,  7.06755799,\n",
    "        5.02272212,  7.75008842,  7.44864279,  8.29678115,  9.6105985 ]).reshape(m, 1)\n",
    "\n",
    "print ('x = ', x)\n",
    "print ('y = ', y)\n",
    "\n",
    "plot(x, y, 'bx')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's prepare the data points for matrix manipulation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = insert(x, 0, ones(len(x)), axis=1)\n",
    "Y = array(y).reshape(m, 1)\n",
    "\n",
    "print ('X = \\n', X)\n",
    "print ('Y = \\n', Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1: Ordinary Least Squares\n",
    "\n",
    "Find the weight values $\\mathbf{w}$ that minimize the error $E_{\\mathbf{in}}(\\mathbf{w}) = \\frac{1}{N} \\sum_{n=1}^n {(\\mathbf{w}^T \\mathbf{X}_n - \\mathbf{y}_n)^2}$.\n",
    "\n",
    "For this, implement Linear Regression and use the Ordinary Least Squares (OLS) closed-form expression to find the estimated values of $\\mathbf{w}$:\n",
    "\n",
    "$$\\mathbf{w} = (\\mathbf{X}^{\\rm T}\\mathbf{X})^{-1} \\mathbf{X}^{\\rm T}\\mathbf{y}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "\n",
    "print ('w = \\n', w)\n",
    "\n",
    "p = plot (X[:,1], Y, 'bx')\n",
    "p = plot (X[:,1], X.dot(w), 'r-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2: Batch Gradient Descent\n",
    "\n",
    "Find the weight values $\\mathbf{w}$ that minimize the error $E_{\\mathbf{in}}(\\mathbf{w}) = \\frac{1}{N} \\sum_{n=1}^n {(\\mathbf{w}^T \\mathbf{X}_n - \\mathbf{y}_n)^2}$.\n",
    "\n",
    "For this, implement the Batch Gradient Descent algorithm with $\\mathbf{s}$ learning steps and learning rate $\\alpha$.  \n",
    "At each training step, update $\\mathbf{w}$ with this rule:\n",
    "\n",
    "$$\\mathbf{w}_i := \\mathbf{w}_i - \\alpha \\left(\\left(\\mathbf{X}\\mathbf{w} - \\mathbf{y}\\right)^T\\mathbf{X}_i\\right)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n, d = X.shape\n",
    "s = 1000 # learning steps\n",
    "alpha = 0.001 # learning rate\n",
    "\n",
    "w = zeros(d).reshape(d, 1)\n",
    "\n",
    "# YOUR CODE HERE\n",
    "\n",
    "print (w)\n",
    "\n",
    "p = plot (X[:,1], Y, 'bx')\n",
    "p = plot (X[:,1], X.dot(w), 'r-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3: Stochastic Gradient Descent\n",
    "\n",
    "Find the weight values $\\mathbf{w}$ that minimize the error $E_{\\mathbf{in}}(\\mathbf{w}) = \\frac{1}{N} \\sum_{n=1}^n {(\\mathbf{w}^T \\mathbf{X}_n - \\mathbf{y}_n)^2}$.\n",
    "\n",
    "For this, implement the Stochastic Gradient Descent algorithm with $\\mathbf{s}$ learning steps and learning rate $\\alpha$.\n",
    "In each step, iterate through all $j$ samples and, for each sample, update $\\mathbf{w}$ with this rule:\n",
    "\n",
    "$$\\mathbf{w}_i := \\mathbf{w}_i - \\alpha\\left(\\mathbf{X}^{(j)}\\mathbf{w} - \\mathbf{y}^{(j)}\\right)\\mathbf{X}^{(j)}_i$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n, d = X.shape\n",
    "s = 1000 # learning steps\n",
    "alpha = 0.001 # learning rate\n",
    "\n",
    "w = zeros(d).reshape(d, 1)\n",
    "\n",
    "# YOUR CODE HERE\n",
    "\n",
    "print (w)\n",
    "\n",
    "p = plot (X[:,1], Y, 'bx')\n",
    "p = plot (X[:,1], X.dot(w), 'r-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4: write a function fit(X,Y) \n",
    "\n",
    "The function fit(X,Y) receives a matrix $X \\in \\mathbb{R}^{m,n}$, where m is the number of samples and n is the number of features, and a matrix $Y \\in \\mathbb{R}^{m}$, and returns the matrix of coefficients $\\mathbf{w} \\in \\mathbb{R}^{n+1}$.   \n",
    "Implement the function with Ordinary Least Squares"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit(X, Y):\n",
    "    # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's test it!\n",
    "\n",
    "m = 10\n",
    "x = linspace(1, m, m).reshape(m,1)\n",
    "y = array([ 3.76405235,  3.20015721,  4.57873798,  6.6408932 ,  7.06755799,\n",
    "        5.02272212,  7.75008842,  7.44864279,  8.29678115,  9.6105985 ]).reshape(m,1)\n",
    "\n",
    "w = fit(x,y)\n",
    "\n",
    "print ('w = \\n', w)\n",
    "\n",
    "p = plot (X[:,1], Y, 'bx')\n",
    "p = plot (X[:,1], X.dot(w), 'r-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Scikit-Learn for Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if Scikit-Learn is installed in your system. If not, install it.\n",
    "\n",
    "Resources and documentation: http://scikit-learn.org/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5: find the coefficients $\\mathbf{w}$ using sklearn.linear_model.LinearRegression\n",
    "\n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "LinearRegression?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = 10\n",
    "x = linspace(1, m, m)\n",
    "y = array([ 3.76405235,  3.20015721,  4.57873798,  6.6408932 ,  7.06755799,\n",
    "        5.02272212,  7.75008842,  7.44864279,  8.29678115,  9.6105985 ])\n",
    "x.shape = (m,1)\n",
    "\n",
    "# YOUR CODE HERE\n",
    "\n",
    "print ('Coefficients = ', model.coef_, ', Intercept = ', model.intercept_)\n",
    "# compare the results to the Ordinary Least Squares result!\n",
    "\n",
    "p = plot (x, y, 'bx')\n",
    "p = plot (x, x*model.coef_+model.intercept_, 'r-')\n",
    "p = plot (x, model.predict(x), 'go')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6: find the coefficients $\\mathbf{w}$ using sklearn.linear_model.Lasso\n",
    "\n",
    "Regularization introduces a penalty in the model complexity, in order to prevent overfitting.  \n",
    "Lasso (least absolute shrinkage and selection operator) uses the constraint that $\\|\\mathbf{w}\\|_1$, the L1-norm of the parameter vector, is no greater than a given value.  \n",
    "The optimization objective for Lasso is:\n",
    "$$ \\frac{1}{2n} \\|\\mathbf{Xw} - \\mathbf{y}\\|^2_2 + \\alpha\\|\\mathbf{w}\\|_1$$\n",
    "\n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Lasso\n",
    "Lasso?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = 10\n",
    "x = linspace(1, m, m)\n",
    "y = array([ 3.76405235,  3.20015721,  4.57873798,  6.6408932 ,  7.06755799,\n",
    "        5.02272212,  7.75008842,  7.44864279,  8.29678115,  9.6105985 ])\n",
    "x.shape = (m,1)\n",
    "\n",
    "# YOUR CODE HERE\n",
    "\n",
    "print ('Coefficients = ', model.coef_, ', Intercept = ', model.intercept_)\n",
    "# compare the results to the Ordinary Least Squares result!\n",
    "\n",
    "p = plot (x, y, 'bx')\n",
    "p = plot (x, x*model.coef_+model.intercept_, 'r-')\n",
    "p = plot (x, model.predict(x), 'go')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with the Boston house-prices dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the dataset and print the description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "boston = load_boston()\n",
    "print (boston.DESCR)\n",
    "\n",
    "X = boston.data\n",
    "y = boston.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, create a plot for each feature, to have an idea of their correlation with the target variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure(figsize=(20,20))\n",
    "nr_plots = len(boston.feature_names)-1\n",
    "\n",
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 7: Create a Regression model using Scikit-Learn Linear Regression for the Boston house-prices dataset\n",
    "\n",
    "Use sklearn.linear_model.LinearRegression to predict the target variable y using only the average number of rooms (RM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Xtrain = X[:,[5]]\n",
    "\n",
    "# YOUR CODE HERE\n",
    "\n",
    "plot(Xtrain, y, 'bx', label='true data')\n",
    "plot(Xtrain, pred, 'gx', label='predicted')\n",
    "legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Train and Test sets\n",
    "\n",
    "Let's split our dataset in the Training and Test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nr_cases = y.shape[0]\n",
    "nTrain = np.floor(nr_cases *2.0 / 3.0)\n",
    "import random\n",
    "ids = list(range(nr_cases))\n",
    "random.shuffle(ids)\n",
    "\n",
    "trainX,trainY,testX,testY = [],[],[],[]\n",
    "for i, idx in enumerate(ids):\n",
    "    if i < nTrain:\n",
    "        trainX.append(X[idx, [5]])\n",
    "        trainY.append(y[idx])\n",
    "    else:\n",
    "        testX.append(X[idx, [5]])\n",
    "        testY.append(y[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 8: Calculate the MAE and RMSE\n",
    "\n",
    "Use only the samples in the training set to train your regression model (use sklearn.linear_model.LinearRegression).\n",
    "\n",
    "Predict the values for the samples in the test set and calculate the **mean absolute error (MAE)** and the **root-mean-square error (RMSE)**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mae(testY, predY):\n",
    "    # YOUR CODE HERE\n",
    "\n",
    "def rmse(testY, predY):\n",
    "    # YOUR CODE HERE\n",
    "\n",
    "# Train a Linear Regression model using only train data\n",
    "# YOUR CODE HERE\n",
    "\n",
    "# Predict the test values using the model\n",
    "predY = model.predict(testX)\n",
    "\n",
    "print(mae(testY, predY))\n",
    "print(rmse(testY, predY))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 9: Train with all features\n",
    "\n",
    "Split the Boston house-prices dataset into Training and Test sets, but using all features. Train a regression model with all features, calculate the MAE and RMSE and compare it with the previous results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nr_cases = y.shape[0]\n",
    "nTrain = np.floor(nr_cases *2.0 / 3.0)\n",
    "import random\n",
    "ids = list(range(nr_cases))\n",
    "random.shuffle(ids)\n",
    "\n",
    "trainX,trainY,testX,testY = [],[],[],[]\n",
    "for i, idx in enumerate(ids):\n",
    "    if i < nTrain:\n",
    "        trainX.append(X[idx])\n",
    "        trainY.append(y[idx])\n",
    "    else:\n",
    "        testX.append(X[idx])\n",
    "        testY.append(y[idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mae(testY, predY):\n",
    "    # YOUR CODE HERE\n",
    "\n",
    "def rmse(testY, predY):\n",
    "    # YOUR CODE HERE\n",
    "\n",
    "# Train a Linear Regression model using only train data\n",
    "# YOUR CODE HERE\n",
    "\n",
    "# Predict the test values using the model\n",
    "predY = model.predict(testX)\n",
    "\n",
    "print(mae(testY, predY))\n",
    "print(rmse(testY, predY))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}