3_baseline_linear_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kaggle Competition - House Prices: Advanced Regression Techniques\n",
    "# Linear Regression Exploration\n",
    "\n",
    "This module is about creating a baseline linear regression model\n",
    "\n",
    "The code for this module is in: [src/models/LinearRegressionModel.py](../src/models/LinearRegressionModel.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/jamie/Dropbox/Programming/Kaggle/kaggle-house-prices'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "# sys.path.append( os.path.abspath( os.path.join(os.getcwd(), \"..\" ))) \n",
    "from src.utils import reset_root_dir\n",
    "reset_root_dir()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pandas import DataFrame, Series\n",
    "import numpy as np\n",
    "import math\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from src.utils.Charts import Charts\n",
    "from src.models.LinearRegressionModel import LinearRegressionModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline Linear Regression\n",
    "\n",
    "The basic workflow for linear regression is: \n",
    "- split the dataset into test and training datasets\n",
    "- create a 2 dimentional dataframe of X input features and a 1 dimentional vector of known Y outputs\n",
    "- generate artifical features as required\n",
    "- fit a linear model\n",
    "- score and output predictions against the test dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a naive baseline, we can use the raw data from all the numeric columns in the dataset as X to predict the Y price. Non-numeric fields will cause linear regression to crash. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>...</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1292</th>\n",
       "      <td>1293</td>\n",
       "      <td>70</td>\n",
       "      <td>60.0</td>\n",
       "      <td>6600</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>1892</td>\n",
       "      <td>1965</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>432</td>\n",
       "      <td>0</td>\n",
       "      <td>287</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1018</th>\n",
       "      <td>1019</td>\n",
       "      <td>80</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10784</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>1991</td>\n",
       "      <td>1992</td>\n",
       "      <td>76.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>402</td>\n",
       "      <td>164</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1213</th>\n",
       "      <td>1214</td>\n",
       "      <td>80</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10246</td>\n",
       "      <td>4</td>\n",
       "      <td>9</td>\n",
       "      <td>1965</td>\n",
       "      <td>2001</td>\n",
       "      <td>0.0</td>\n",
       "      <td>648</td>\n",
       "      <td>...</td>\n",
       "      <td>364</td>\n",
       "      <td>88</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 37 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        Id  MSSubClass  LotFrontage  LotArea  OverallQual  OverallCond  \\\n",
       "1292  1293          70         60.0     6600            5            4   \n",
       "1018  1019          80          0.0    10784            7            5   \n",
       "1213  1214          80          0.0    10246            4            9   \n",
       "\n",
       "      YearBuilt  YearRemodAdd  MasVnrArea  BsmtFinSF1  ...  GarageArea  \\\n",
       "1292       1892          1965         0.0           0  ...         432   \n",
       "1018       1991          1992        76.0           0  ...         402   \n",
       "1213       1965          2001         0.0         648  ...         364   \n",
       "\n",
       "      WoodDeckSF  OpenPorchSF  EnclosedPorch  3SsnPorch  ScreenPorch  \\\n",
       "1292           0          287              0          0            0   \n",
       "1018         164            0              0          0            0   \n",
       "1213          88            0              0          0            0   \n",
       "\n",
       "      PoolArea  MiscVal  MoSold  YrSold  \n",
       "1292         0        0      12    2009  \n",
       "1018         0        0       5    2007  \n",
       "1213         0        0       5    2006  \n",
       "\n",
       "[3 rows x 37 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "linearRegressionModel = LinearRegressionModel()\n",
    "linearRegressionModel.data[\"X_train\"].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1292    107500\n",
       "1018    160000\n",
       "1213    145000\n",
       "Name: SalePrice, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "linearRegressionModel.data[\"Y_train\"].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.1942, 'LinearRegressionModel', 'LinearRegression', '')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "linearRegressionModel.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scoring Methods\n",
    "\n",
    "[sklearn.linear_model.LinearRegression()](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)\n",
    "uses [R^2](https://www.investopedia.com/terms/r/r-squared.asp) as its default scoring method. 1 meaning perfect correleation between inputs and outputs.\n",
    "- R^2 = 0.823 | using training / validation dataset splitting \n",
    "\n",
    "[Kaggle](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/overview/evaluation) uses [Root-Mean-Squared-Error (RMSE)](https://www.statisticshowto.datasciencecentral.com/rmse/) between the logarithm of the predicted value and the logarithm of the observed sales price. \n",
    "- RMSLE = 0.18600 | local train      - without training / validation dataset splitting \n",
    "- RMSLE = 0.19416 | local validation - with    training / validation dataset splitting \n",
    "- RMSLE = 0.20892 | Kaggle test      - with    training / validation splitting\n",
    "- RMSLE = 0.43452 | Kaggle test      - without training / validation splitting\n",
    "\n",
    "## Submit to Kaggle\n",
    "- https://www.kaggle.com/c/house-prices-advanced-regression-techniques/submissions\n",
    "\n",
    "```\n",
    "$ kaggle competitions submit -c house-prices-advanced-regression-techniques -f data/submissions/LinearRegressionModel.csv -m \"LinearRegressionModel.py - raw numeric fields\n",
    "```\n",
    "    \n",
    "Before training / validation dataset splitting: \n",
    "- Your (Kaggle) submission scored 0.43452, which is an improvement of your previous score of 0.74279. Great job! \n",
    "- Kaggle Rank 4079 / 4339\n",
    "\n",
    "After training / validation dataset splitting:\n",
    "- Your (Kaggle) submission scored 0.20892, which is an improvement of your previous score of 0.43452. Great job! \n",
    "- Kaggle Rank 3751 / 4339\n",
    "\n",
    "Unable to explain why training / validation splitting has such a major impact on kaggle test scores, but minimal effect when applied locally. Maybe a smaller dataset leads to less overfitting."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}