Predicting House Price Regression (__all__).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "simplified-spokesman",
   "metadata": {},
   "source": [
    "### Predicting house prices with regression\n",
    "\n",
    "> Regression problems normally answers questions like **How much...?**, **How mayny...?**. In this notebook we will use different **Regression** Algorithms to predict the house prices from the well known dataset in **sklearn** `load_boston`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ahead-sacramento",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import datasets\n",
    "import pandas as pd\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regular-unknown",
   "metadata": {},
   "source": [
    "### Loading data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "spread-float",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1.0</td>\n",
       "      <td>296.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.90</td>\n",
       "      <td>4.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>392.83</td>\n",
       "      <td>4.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>394.63</td>\n",
       "      <td>2.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.33</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        0     1     2    3      4      5     6       7    8      9     10  \\\n",
       "0  0.00632  18.0  2.31  0.0  0.538  6.575  65.2  4.0900  1.0  296.0  15.3   \n",
       "1  0.02731   0.0  7.07  0.0  0.469  6.421  78.9  4.9671  2.0  242.0  17.8   \n",
       "2  0.02729   0.0  7.07  0.0  0.469  7.185  61.1  4.9671  2.0  242.0  17.8   \n",
       "3  0.03237   0.0  2.18  0.0  0.458  6.998  45.8  6.0622  3.0  222.0  18.7   \n",
       "4  0.06905   0.0  2.18  0.0  0.458  7.147  54.2  6.0622  3.0  222.0  18.7   \n",
       "\n",
       "       11    12  \n",
       "0  396.90  4.98  \n",
       "1  396.90  9.14  \n",
       "2  392.83  4.03  \n",
       "3  394.63  2.94  \n",
       "4  396.90  5.33  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "boston = datasets.load_boston()\n",
    "pd.DataFrame(boston.data).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "quick-thousand",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': array([[6.3200e-03, 1.8000e+01, 2.3100e+00, ..., 1.5300e+01, 3.9690e+02,\n",
      "        4.9800e+00],\n",
      "       [2.7310e-02, 0.0000e+00, 7.0700e+00, ..., 1.7800e+01, 3.9690e+02,\n",
      "        9.1400e+00],\n",
      "       [2.7290e-02, 0.0000e+00, 7.0700e+00, ..., 1.7800e+01, 3.9283e+02,\n",
      "        4.0300e+00],\n",
      "       ...,\n",
      "       [6.0760e-02, 0.0000e+00, 1.1930e+01, ..., 2.1000e+01, 3.9690e+02,\n",
      "        5.6400e+00],\n",
      "       [1.0959e-01, 0.0000e+00, 1.1930e+01, ..., 2.1000e+01, 3.9345e+02,\n",
      "        6.4800e+00],\n",
      "       [4.7410e-02, 0.0000e+00, 1.1930e+01, ..., 2.1000e+01, 3.9690e+02,\n",
      "        7.8800e+00]]), 'target': array([24. , 21.6, 34.7, 33.4, 36.2, 28.7, 22.9, 27.1, 16.5, 18.9, 15. ,\n",
      "       18.9, 21.7, 20.4, 18.2, 19.9, 23.1, 17.5, 20.2, 18.2, 13.6, 19.6,\n",
      "       15.2, 14.5, 15.6, 13.9, 16.6, 14.8, 18.4, 21. , 12.7, 14.5, 13.2,\n",
      "       13.1, 13.5, 18.9, 20. , 21. , 24.7, 30.8, 34.9, 26.6, 25.3, 24.7,\n",
      "       21.2, 19.3, 20. , 16.6, 14.4, 19.4, 19.7, 20.5, 25. , 23.4, 18.9,\n",
      "       35.4, 24.7, 31.6, 23.3, 19.6, 18.7, 16. , 22.2, 25. , 33. , 23.5,\n",
      "       19.4, 22. , 17.4, 20.9, 24.2, 21.7, 22.8, 23.4, 24.1, 21.4, 20. ,\n",
      "       20.8, 21.2, 20.3, 28. , 23.9, 24.8, 22.9, 23.9, 26.6, 22.5, 22.2,\n",
      "       23.6, 28.7, 22.6, 22. , 22.9, 25. , 20.6, 28.4, 21.4, 38.7, 43.8,\n",
      "       33.2, 27.5, 26.5, 18.6, 19.3, 20.1, 19.5, 19.5, 20.4, 19.8, 19.4,\n",
      "       21.7, 22.8, 18.8, 18.7, 18.5, 18.3, 21.2, 19.2, 20.4, 19.3, 22. ,\n",
      "       20.3, 20.5, 17.3, 18.8, 21.4, 15.7, 16.2, 18. , 14.3, 19.2, 19.6,\n",
      "       23. , 18.4, 15.6, 18.1, 17.4, 17.1, 13.3, 17.8, 14. , 14.4, 13.4,\n",
      "       15.6, 11.8, 13.8, 15.6, 14.6, 17.8, 15.4, 21.5, 19.6, 15.3, 19.4,\n",
      "       17. , 15.6, 13.1, 41.3, 24.3, 23.3, 27. , 50. , 50. , 50. , 22.7,\n",
      "       25. , 50. , 23.8, 23.8, 22.3, 17.4, 19.1, 23.1, 23.6, 22.6, 29.4,\n",
      "       23.2, 24.6, 29.9, 37.2, 39.8, 36.2, 37.9, 32.5, 26.4, 29.6, 50. ,\n",
      "       32. , 29.8, 34.9, 37. , 30.5, 36.4, 31.1, 29.1, 50. , 33.3, 30.3,\n",
      "       34.6, 34.9, 32.9, 24.1, 42.3, 48.5, 50. , 22.6, 24.4, 22.5, 24.4,\n",
      "       20. , 21.7, 19.3, 22.4, 28.1, 23.7, 25. , 23.3, 28.7, 21.5, 23. ,\n",
      "       26.7, 21.7, 27.5, 30.1, 44.8, 50. , 37.6, 31.6, 46.7, 31.5, 24.3,\n",
      "       31.7, 41.7, 48.3, 29. , 24. , 25.1, 31.5, 23.7, 23.3, 22. , 20.1,\n",
      "       22.2, 23.7, 17.6, 18.5, 24.3, 20.5, 24.5, 26.2, 24.4, 24.8, 29.6,\n",
      "       42.8, 21.9, 20.9, 44. , 50. , 36. , 30.1, 33.8, 43.1, 48.8, 31. ,\n",
      "       36.5, 22.8, 30.7, 50. , 43.5, 20.7, 21.1, 25.2, 24.4, 35.2, 32.4,\n",
      "       32. , 33.2, 33.1, 29.1, 35.1, 45.4, 35.4, 46. , 50. , 32.2, 22. ,\n",
      "       20.1, 23.2, 22.3, 24.8, 28.5, 37.3, 27.9, 23.9, 21.7, 28.6, 27.1,\n",
      "       20.3, 22.5, 29. , 24.8, 22. , 26.4, 33.1, 36.1, 28.4, 33.4, 28.2,\n",
      "       22.8, 20.3, 16.1, 22.1, 19.4, 21.6, 23.8, 16.2, 17.8, 19.8, 23.1,\n",
      "       21. , 23.8, 23.1, 20.4, 18.5, 25. , 24.6, 23. , 22.2, 19.3, 22.6,\n",
      "       19.8, 17.1, 19.4, 22.2, 20.7, 21.1, 19.5, 18.5, 20.6, 19. , 18.7,\n",
      "       32.7, 16.5, 23.9, 31.2, 17.5, 17.2, 23.1, 24.5, 26.6, 22.9, 24.1,\n",
      "       18.6, 30.1, 18.2, 20.6, 17.8, 21.7, 22.7, 22.6, 25. , 19.9, 20.8,\n",
      "       16.8, 21.9, 27.5, 21.9, 23.1, 50. , 50. , 50. , 50. , 50. , 13.8,\n",
      "       13.8, 15. , 13.9, 13.3, 13.1, 10.2, 10.4, 10.9, 11.3, 12.3,  8.8,\n",
      "        7.2, 10.5,  7.4, 10.2, 11.5, 15.1, 23.2,  9.7, 13.8, 12.7, 13.1,\n",
      "       12.5,  8.5,  5. ,  6.3,  5.6,  7.2, 12.1,  8.3,  8.5,  5. , 11.9,\n",
      "       27.9, 17.2, 27.5, 15. , 17.2, 17.9, 16.3,  7. ,  7.2,  7.5, 10.4,\n",
      "        8.8,  8.4, 16.7, 14.2, 20.8, 13.4, 11.7,  8.3, 10.2, 10.9, 11. ,\n",
      "        9.5, 14.5, 14.1, 16.1, 14.3, 11.7, 13.4,  9.6,  8.7,  8.4, 12.8,\n",
      "       10.5, 17.1, 18.4, 15.4, 10.8, 11.8, 14.9, 12.6, 14.1, 13. , 13.4,\n",
      "       15.2, 16.1, 17.8, 14.9, 14.1, 12.7, 13.5, 14.9, 20. , 16.4, 17.7,\n",
      "       19.5, 20.2, 21.4, 19.9, 19. , 19.1, 19.1, 20.1, 19.9, 19.6, 23.2,\n",
      "       29.8, 13.8, 13.3, 16.7, 12. , 14.6, 21.4, 23. , 23.7, 25. , 21.8,\n",
      "       20.6, 21.2, 19.1, 20.6, 15.2,  7. ,  8.1, 13.6, 20.1, 21.8, 24.5,\n",
      "       23.1, 19.7, 18.3, 21.2, 17.5, 16.8, 22.4, 20.6, 23.9, 22. , 11.9]), 'feature_names': array(['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD',\n",
      "       'TAX', 'PTRATIO', 'B', 'LSTAT'], dtype='<U7'), 'DESCR': \".. _boston_dataset:\\n\\nBoston house prices dataset\\n---------------------------\\n\\n**Data Set Characteristics:**  \\n\\n    :Number of Instances: 506 \\n\\n    :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.\\n\\n    :Attribute Information (in order):\\n        - CRIM     per capita crime rate by town\\n        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.\\n        - INDUS    proportion of non-retail business acres per town\\n        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\\n        - NOX      nitric oxides concentration (parts per 10 million)\\n        - RM       average number of rooms per dwelling\\n        - AGE      proportion of owner-occupied units built prior to 1940\\n        - DIS      weighted distances to five Boston employment centres\\n        - RAD      index of accessibility to radial highways\\n        - TAX      full-value property-tax rate per $10,000\\n        - PTRATIO  pupil-teacher ratio by town\\n        - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\\n        - LSTAT    % lower status of the population\\n        - MEDV     Median value of owner-occupied homes in $1000's\\n\\n    :Missing Attribute Values: None\\n\\n    :Creator: Harrison, D. and Rubinfeld, D.L.\\n\\nThis is a copy of UCI ML housing dataset.\\nhttps://archive.ics.uci.edu/ml/machine-learning-databases/housing/\\n\\n\\nThis dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.\\n\\nThe Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic\\nprices and the demand for clean air', J. Environ. Economics & Management,\\nvol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics\\n...', Wiley, 1980.   N.B. Various transformations are used in the table on\\npages 244-261 of the latter.\\n\\nThe Boston house-price data has been used in many machine learning papers that address regression\\nproblems.   \\n     \\n.. topic:: References\\n\\n   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.\\n   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.\\n\", 'filename': 'C:\\\\Users\\\\crisp\\\\Documents\\\\anaconda3\\\\lib\\\\site-packages\\\\sklearn\\\\datasets\\\\data\\\\boston_house_prices.csv'}\n"
     ]
    }
   ],
   "source": [
    "print(boston)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "occupied-decrease",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD',\n",
       "       'TAX', 'PTRATIO', 'B', 'LSTAT'], dtype='<U7')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "boston.feature_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "resident-allen",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((506,), (506, 13))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "boston.target.shape, boston.data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "raised-insider",
   "metadata": {},
   "source": [
    "### Spliting our data\n",
    "> We are goin to split our data into train, and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "fancy-classics",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "solved-innocent",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(boston.data, boston.target, test_size = .3, random_state =99)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "joined-curve",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((354, 13), (354,), (152, 13))"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape, y_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abstract-underwear",
   "metadata": {},
   "source": [
    "### Creating different `Regression` models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "traditional-candy",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import SGDRegressor\n",
    "from sklearn.svm import SVR\n",
    "from sklearn.ensemble import ExtraTreesRegressor, RandomForestRegressor\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.linear_model import LinearRegression, LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "violent-edinburgh",
   "metadata": {},
   "source": [
    "#### Scaling our X and y data for trsin and test\n",
    "> We are going to scale our data using the `StandardScaler` for both train and test datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "laughing-tender",
   "metadata": {},
   "outputs": [],
   "source": [
    "scaleX = StandardScaler()\n",
    "scaley = StandardScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "pleasant-spouse",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.35961633, -0.48960055, -0.48281904, -0.2511236 , -0.16357244,\n",
       "       -0.64911303, -0.43813561,  0.32717674, -0.65180049, -0.65554224,\n",
       "        1.11770735,  0.42946687, -0.59396908])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_scaled = scaleX.fit_transform(X_train)\n",
    "X_test_scaled = scaleX.transform(X_test)\n",
    "X_train_scaled[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "comic-palace",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([35.4, 35.2])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "quiet-classic",
   "metadata": {},
   "source": [
    "#### 1. SGDRegressor\n",
    "> SGD stands for Stochastic Gradient Descent: the gradient of the loss is estimated each sample at a time and the model is updated along the way with a decreasing strength schedule (aka learning rate)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "radical-wallpaper",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDRegressor(penalty=None, random_state=42)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_reg = SGDRegressor(loss='squared_loss', random_state=42, penalty=None)\n",
    "sgd_reg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "judicial-sensitivity",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDRegressor(penalty=None, random_state=42)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_reg.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "sharp-linux",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([34.09016066, 34.8356811 , 25.35752209, 25.31266708, 34.4594283 ]),\n",
       " array([35.4, 35.2, 24.8, 22.6, 34.9]))"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_reg.predict(X_test_scaled[:5]), y_test[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accessory-concord",
   "metadata": {},
   "source": [
    "#### 2. SVR\n",
    "> The implementation is based on libsvm. The fit time complexity is more than quadratic with the number of samples which makes it hard to scale to datasets with more than a couple of 10000 samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "surface-relay",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVR(C=1, epsilon=0.5, kernel='linear')"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svr_reg = SVR(C=1, epsilon=.5, kernel=\"linear\",degree=3 )\n",
    "svr_reg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "hollywood-coalition",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVR(C=1, epsilon=0.5, kernel='linear')"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svr_reg.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "sonic-polymer",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([32.77539084, 34.47704835, 25.07823343, 24.12403952, 33.66906661]),\n",
       " array([35.4, 35.2, 24.8, 22.6, 34.9]))"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svr_reg.predict(X_test_scaled[:5]), y_test[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accepted-retail",
   "metadata": {},
   "source": [
    "#### 3. DecisionTreeRegressor\n",
    "```\n",
    "class sklearn.tree.DecisionTreeRegressor(*, criterion='mse', splitter='best', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features=None, random_state=None, max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, ccp_alpha=0.0)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "sixth-calibration",
   "metadata": {},
   "outputs": [],
   "source": [
    "dst_reg = DecisionTreeRegressor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "deadly-carrier",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor()"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dst_reg.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "wired-celebration",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([19.9, 11.8, 13.1, 20. ,  7.2]), array([19.9, 11.8, 13.1, 20. ,  7.2]))"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dst_reg.predict(X_train_scaled[:5]), y_train[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "heated-desperate",
   "metadata": {},
   "source": [
    "#### 4. RandomForestRegressor\n",
    "> A random forest is a meta estimator that fits a number of classifying decision trees on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting. The sub-sample size is controlled with the max_samples parameter if bootstrap=True (default), otherwise the whole dataset is used to build each tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "discrete-championship",
   "metadata": {},
   "outputs": [],
   "source": [
    "randf_reg = RandomForestRegressor(n_estimators = 5, max_depth=5, random_state=45)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "fatty-tobacco",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestRegressor(max_depth=5, n_estimators=5, random_state=45)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "randf_reg.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "brazilian-perfume",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([20.17370258, 11.04782471, 14.90296757, 21.33863727,  9.60354437]),\n",
       " array([19.9, 11.8, 13.1, 20. ,  7.2]))"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "randf_reg.predict(X_train_scaled[:5]), y_train[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dominican-possession",
   "metadata": {},
   "source": [
    "#### 3. ExtraTreesRegressor\n",
    "> This class implements a meta estimator that fits a number of randomized decision trees (a.k.a. extra-trees) on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "minor-quebec",
   "metadata": {},
   "outputs": [],
   "source": [
    "ext_reg = ExtraTreesRegressor(n_estimators=100, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "essential-director",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ExtraTreesRegressor(random_state=0)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ext_reg.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "level-federal",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([32.415, 40.858, 28.084, 22.642, 34.329]),\n",
       " array([35.4, 35.2, 24.8, 22.6, 34.9]))"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ext_reg.predict(X_test_scaled[:5]), y_test[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "trying-savannah",
   "metadata": {},
   "source": [
    "#### 4. KNeighborsRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "corrected-cemetery",
   "metadata": {},
   "source": [
    "> The target is predicted by local interpolation of the targets associated of the nearest neighbors in the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "hairy-interest",
   "metadata": {},
   "outputs": [],
   "source": [
    "neigh_reg = KNeighborsRegressor(n_neighbors=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "dressed-liberty",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsRegressor(n_neighbors=2)"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neigh_reg.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "opposed-freeze",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([28.  , 32.75, 25.3 , 19.55, 34.05]),\n",
       " array([35.4, 35.2, 24.8, 22.6, 34.9]))"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neigh_reg.predict(X_test_scaled[:5]), y_test[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "second-diesel",
   "metadata": {},
   "source": [
    "#### 7. LinearRegression\n",
    "> LinearRegression fits a linear model with coefficients w = (w1, …, wp) to minimize the residual sum of squares between the observed targets in the dataset, and the targets predicted by the linear approximation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "wanted-thailand",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l_reg = LinearRegression()\n",
    "l_reg.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "opening-mineral",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l_reg.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "perfect-skating",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([34.39031078, 34.72259889, 25.11192059, 24.05948598, 34.85344369]),\n",
       " array([35.4, 35.2, 24.8, 22.6, 34.9]))"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l_reg.predict(X_test_scaled[:5]), y_test[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "essential-crash",
   "metadata": {},
   "source": [
    "### Scoring Different models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "fresh-radar",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ExtraTreesRegressor(random_state=0)"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svr_reg ## SVR\n",
    "sgd_reg ## SGDRegressor\n",
    "dst_reg ## DecisionTreeRegressor\n",
    "l_reg ## LinearRegression\n",
    "neigh_reg ## KNeighborsRegressor\n",
    "randf_reg ## RandomForestRegressor\n",
    "ext_reg   ## ExtraTreesRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "respected-absolute",
   "metadata": {},
   "source": [
    "#### SVR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "rubber-cache",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.7369778607349755, 0.6667837988608363)"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svr_reg.score(X_train_scaled, y_train), svr_reg.score(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "subtle-joseph",
   "metadata": {},
   "source": [
    "> `SVR` is `73%` accurate on the train and `67%` accurate on the test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "gross-whale",
   "metadata": {},
   "source": [
    "#### SGDRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "mechanical-reynolds",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.7624407538153944, 0.6695152132670618)"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_reg.score(X_train_scaled, y_train), sgd_reg.score(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wireless-money",
   "metadata": {},
   "source": [
    "> `SGDRegressor` is `76%` accurate on the train and `67%` accurate on the test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "retained-literature",
   "metadata": {},
   "source": [
    "#### DecisionTreeRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "bound-violin",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.0, 0.7514846521915527)"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dst_reg.score(X_train_scaled, y_train), dst_reg.score(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "collectible-calculator",
   "metadata": {},
   "source": [
    "`DecisionTreeRegressor` is `100%` accurate on the train and `75%` accurate on the test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "authorized-cologne",
   "metadata": {},
   "source": [
    "#### LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "northern-sunday",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.764467385910802, 0.6721726414424739)"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l_reg.score(X_train_scaled, y_train), l_reg.score(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "divided-hartford",
   "metadata": {},
   "source": [
    "`LinearRegression` is `76%` accurate on the train and `67%` accurate on the test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "senior-wisdom",
   "metadata": {},
   "source": [
    "#### KNeighborsRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "threatened-money",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.9396633008285875, 0.7092287520704416)"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neigh_reg.score(X_train_scaled, y_train), neigh_reg.score(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "nonprofit-threshold",
   "metadata": {},
   "source": [
    "`KNeighborsRegressor` is `94%` accurate on the train and `71%` accurate on the test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "reasonable-affairs",
   "metadata": {},
   "source": [
    "#### RandomForestRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "material-demographic",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.9269031662146822, 0.7741025482542851)"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "randf_reg.score(X_train_scaled, y_train), randf_reg.score(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "light-conflict",
   "metadata": {},
   "source": [
    "`RandomForestRegressor` is `93%` accurate on the train and `77%` accurate on the test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chubby-engagement",
   "metadata": {},
   "source": [
    "#### ExtraTreesRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "banned-journalist",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.0, 0.8325553079114401)"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ext_reg.score(X_train_scaled, y_train), ext_reg.score(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "divided-senator",
   "metadata": {},
   "source": [
    "`ExtraTreesRegressor` is `100%` accurate on the train and `83%` accurate on the test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "immediate-inflation",
   "metadata": {},
   "source": [
    "### Conlusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "brilliant-power",
   "metadata": {},
   "source": [
    "> The best model for predicting the house prices for `boston_dataset` is `ExtraTreesRegressor` according to the scoring outcome we get above comparing with other algos."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}