3.Regression/Solution_Homework_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution Homework 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "\n",
    "#### Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "$\\widehat{y}=\\beta_{0}+\\beta_{1} x$\n",
    "\n",
    "Where $\\beta_{0}$ is 25 and $\\beta_{1}$ is 0.5\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 1a.\n",
    "\n",
    "a. What is the corresponding predicted value for the input x = 10?\n",
    "                       \n",
    "$\\widehat{y}$ = 25 + 0.5 * 10\n",
    "\n",
    "$\\widehat{y}$ = 30\n",
    "\n",
    "predicted value is 30.0 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 1b.\n",
    "\n",
    "b. Compute the residual corresponding to $x=6 \\quad y=30$\n",
    "\n",
    "The difference between the observed value of the dependent variable (y) and the predicted value ($\\widehat{y}$) is called the residual  \n",
    "\n",
    "\n",
    "Residual = Observed value - Predicted value\n",
    "\n",
    "Residual = 30 - (25+ 0.5*6)\n",
    "\n",
    "Residual = 30 - 28\n",
    "\n",
    "Residual = 2\n",
    "\n",
    "residual value is  2.0\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 1c\n",
    "c. How does $\\hat{y}$ change with respect to $x$ when $x$ increases by 1 unit?\n",
    "\n",
    "for x = 6, $\\hat{y}$ = 28\n",
    "\n",
    "for x = 7, $\\hat{y}$ = 28.5\n",
    "\n",
    " keeping $\\beta_{0}$ as constant , For every increase of x by 1 unit,  $\\hat{y}$ increases by 0.5 , because the coefficient of regression for x is $\\beta_{1}$ , which is 0.5\n",
    " \n",
    " \n",
    " Regression coefficients represent the mean change in the response variable for one unit of change in the predictor variable while holding other predictors in the model constant. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 1d.\n",
    "Original equation: \n",
    "\n",
    "\n",
    "$\\widehat{y}=\\beta_{0}+\\beta_{1} x$\n",
    "\n",
    "\n",
    "$ \\widehat{ y }_{ 1 }$ = 25 + 0.5*5\n",
    "\n",
    "\n",
    "$ \\widehat{ y }_{ 1 }$ = 27.5\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Second equation:\n",
    "\n",
    "\n",
    "\n",
    "$\\widehat{ y }_{ 2 } = \\beta_{0} + (\\frac { \\beta_{1}  }{ 3600 }) x^{*}$\n",
    "\n",
    "where $x^{*}$ is the number of\n",
    "working hours measured in seconds\n",
    "\n",
    "1 hour = 3600 seconds\n",
    "\n",
    "x hour = (x * 3600) seconds\n",
    "\n",
    "$ \\widehat{ y }_{ 2 }  = 25 + (\\frac { 0.5  }{ 3600 })* (5 * 3600) $ \n",
    "\n",
    "\n",
    "$ \\widehat{ y }_{ 2 }= 25 + 2.5 $\n",
    "\n",
    "$ \\widehat{ y }_{ 2 }= 27.5 $\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise 2.a. import necessary libraries\n",
    "import pandas as pd\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt \n",
    "\n",
    "import pandas as pd  \n",
    "import seaborn as sns "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>MEDV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.65660</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19.58</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8710</td>\n",
       "      <td>6.122</td>\n",
       "      <td>97.3</td>\n",
       "      <td>1.6180</td>\n",
       "      <td>5</td>\n",
       "      <td>403</td>\n",
       "      <td>14.7</td>\n",
       "      <td>14.10</td>\n",
       "      <td>21.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.10612</td>\n",
       "      <td>30.0</td>\n",
       "      <td>4.93</td>\n",
       "      <td>0</td>\n",
       "      <td>0.4280</td>\n",
       "      <td>6.095</td>\n",
       "      <td>65.1</td>\n",
       "      <td>6.3361</td>\n",
       "      <td>6</td>\n",
       "      <td>300</td>\n",
       "      <td>16.6</td>\n",
       "      <td>12.40</td>\n",
       "      <td>20.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.33099</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19.58</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8710</td>\n",
       "      <td>5.186</td>\n",
       "      <td>93.8</td>\n",
       "      <td>1.5296</td>\n",
       "      <td>5</td>\n",
       "      <td>403</td>\n",
       "      <td>14.7</td>\n",
       "      <td>28.32</td>\n",
       "      <td>17.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.06129</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.33</td>\n",
       "      <td>1</td>\n",
       "      <td>0.4429</td>\n",
       "      <td>7.645</td>\n",
       "      <td>49.7</td>\n",
       "      <td>5.2119</td>\n",
       "      <td>5</td>\n",
       "      <td>216</td>\n",
       "      <td>14.9</td>\n",
       "      <td>3.01</td>\n",
       "      <td>46.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.15172</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.14</td>\n",
       "      <td>0</td>\n",
       "      <td>0.5380</td>\n",
       "      <td>5.701</td>\n",
       "      <td>95.0</td>\n",
       "      <td>3.7872</td>\n",
       "      <td>4</td>\n",
       "      <td>307</td>\n",
       "      <td>21.0</td>\n",
       "      <td>18.35</td>\n",
       "      <td>13.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  INDUS  CHAS     NOX     RM   AGE     DIS  RAD  TAX  PTRATIO  \\\n",
       "0  1.65660   0.0  19.58     0  0.8710  6.122  97.3  1.6180    5  403     14.7   \n",
       "1  0.10612  30.0   4.93     0  0.4280  6.095  65.1  6.3361    6  300     16.6   \n",
       "2  2.33099   0.0  19.58     0  0.8710  5.186  93.8  1.5296    5  403     14.7   \n",
       "3  0.06129  20.0   3.33     1  0.4429  7.645  49.7  5.2119    5  216     14.9   \n",
       "4  1.15172   0.0   8.14     0  0.5380  5.701  95.0  3.7872    4  307     21.0   \n",
       "\n",
       "   LSTAT  MEDV  \n",
       "0  14.10  21.5  \n",
       "1  12.40  20.1  \n",
       "2  28.32  17.8  \n",
       "3   3.01  46.0  \n",
       "4  18.35  13.1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import data\n",
    "data = pd.read_csv(\"housing.csv\")\n",
    "data.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# preprocessing (apply any of imputation, scaling, feature selection, one hot encoding etc. as you feel necessary) \n",
    "\n",
    "#check if there are any missing values in the data \n",
    "data.isnull().sum()\n",
    "\n",
    "#Prepare the data for testing and training set\n",
    "X = data.iloc[:,:-1].values\n",
    "y = data.iloc[:,-1].values\n",
    "\n",
    "\n",
    "# Splitting the data into the Training set and Test set  \n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imputation\n",
    "from sklearn.impute import SimpleImputer\n",
    "imputer = SimpleImputer()\n",
    "X_train = imputer.fit_transform(X_train)\n",
    "X_test = imputer.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scaling\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "sc = StandardScaler()\n",
    "X_train = sc.fit_transform(X_train)\n",
    "X_test = sc.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,\n",
       "         normalize=False)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Exercise 2.b Linear regression\n",
    "\n",
    "#use scikit-learn’s LinearRegression to train  model on both the training and test sets.\n",
    "\n",
    "\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "\n",
    "#Fit the LinearRegression from sklearn\n",
    "lin_model = LinearRegression()\n",
    "lin_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model performance for testing set\n",
      "--------------------------------------\n",
      "R2 score is 0.772728165693154\n"
     ]
    }
   ],
   "source": [
    "# report results (r2_score)\n",
    "# model evaluation for testing set\n",
    "\n",
    "y_test_predict = lin_model.predict(X_test)\n",
    "\n",
    "# r-squared score of the model\n",
    "r_squared_score = r2_score(y_test, y_test_predict)\n",
    "\n",
    "print(\"The model performance for testing set\")\n",
    "print(\"--------------------------------------\")\n",
    "\n",
    "print('R2 score is {}'.format(r_squared_score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " linear model coefficients : \n",
      "   [-1.27794825  1.08351106  0.45178609  0.98272902 -2.44849385  2.61831927\n",
      "  0.20038205 -2.95561278  2.81952401 -2.42492555 -1.81620507 -3.87032126]\n",
      "\n",
      " lin_model.intercept_ :  22.331188118811916\n"
     ]
    }
   ],
   "source": [
    "# Exercise 2.c - \n",
    "print(\"\\n linear model coefficients : \\n  \",lin_model.coef_)\n",
    "\n",
    "print(\"\\n lin_model.intercept_ : \",lin_model.intercept_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "absoulte arr values \n",
      " [1.27794825 1.08351106 0.45178609 0.98272902 2.44849385 2.61831927\n",
      " 0.20038205 2.95561278 2.81952401 2.42492555 1.81620507 3.87032126]\n",
      " \n",
      " number of coefficients lin_model.coef_ (12,)\n",
      "\n",
      " indices of 5 largest absolute values  [11  7  8  5  4]\n",
      "\n",
      " Feature coefficients  are \n",
      "\n",
      " LSTAT : 3.8703212601298054\n",
      " DIS : 2.955612783956143\n",
      " RAD : 2.819524011090971\n",
      " RM : 2.6183192745614474\n",
      " NOX : 2.4484938523631556\n"
     ]
    }
   ],
   "source": [
    "arr = lin_model.coef_\n",
    "\n",
    "# find absoulte values for coefficients\n",
    "arr =np.absolute(arr) \n",
    "print(\"absoulte arr values \\n\", arr)\n",
    "print(\" \\n number of coefficients lin_model.coef_\",(arr.shape))\n",
    "\n",
    "# code to find 5 largest weights of the model\n",
    "print(\"\\n indices of 5 largest absolute values \",arr.argsort()[::-1][:5])\n",
    "print(\"\\n Feature coefficients  are \\n\")\n",
    "\n",
    "# print Feature coefficients\n",
    "temp_dict = {}\n",
    "\n",
    "for index,key in  enumerate(data.keys()):\n",
    "    temp_dict[index] = key\n",
    "\n",
    "max_5 = arr.argsort()[::-1][:5]\n",
    "for i in max_5: \n",
    "    print(\" {} : {}\".format(temp_dict[i],arr[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exercise 2c.\n",
    "1. Interpretation of intercept <br>\n",
    "When input features are  centered around 0 (scaled with mean set to 0), intercept imnplies the expected  value of the dependent variable when input features are set to their means. \n",
    "In the second case where input features  are not centered around 0, no meaningful interpretation can be done. \n",
    "\n",
    "2. Interpretation of coefficients <br>\n",
    "Coefficients represent change in dependent variable induced by one unit change in the input feature when all other features are kept constant. It is important to note that coefficients does not imply feature importance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}