housing.ipynb

{
  "cells": [
    {
      "metadata": {
        "_uuid": "06da34af0c189cb0c29beb679dfa202bc76b63df"
      },
      "cell_type": "markdown",
      "source": "# Within Top 10% with Simple Regression Model."
    },
    {
      "metadata": {
        "_uuid": "6f62f70159ddf19687dd1c679055f7cc27630cfb"
      },
      "cell_type": "markdown",
      "source": "# Step By Step Procedure To Predict House Price"
    },
    {
      "metadata": {
        "_uuid": "3306a9daf742759fff4e9a6959ab72fec3230e7f"
      },
      "cell_type": "markdown",
      "source": "# Importing packages\nWe have **numpy** and **pandas** to work with numbers and data, and we have **seaborn** and **matplotlib** to visualize data. We would also like to filter out unnecessary warnings. **Scipy** for normalization and skewing of data."
    },
    {
      "metadata": {
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "trusted": true
      },
      "cell_type": "code",
      "source": "#import some necessary librairies\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n%matplotlib inline\nimport matplotlib.pyplot as plt  # Matlab-style plotting\nimport seaborn as sns\ncolor = sns.color_palette()\nsns.set_style('darkgrid')\nimport warnings\ndef ignore_warn(*args, **kwargs):\n    pass\nwarnings.warn = ignore_warn #ignore annoying warning (from sklearn and seaborn)\n\nfrom scipy import stats\nfrom scipy.stats import norm, skew #for some statistics",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "e30ef2b9a5211e7e44031145e7dfbf54a0a429e2"
      },
      "cell_type": "markdown",
      "source": "# Loading and Inspecting data\nWith various Pandas functions, we load our training and test data set as well as inspect it to get an idea of the data we're working with. This is a large dataset we will be working on.\n"
    },
    {
      "metadata": {
        "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
        "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
        "trusted": true
      },
      "cell_type": "code",
      "source": "#Now let's import and put the train and test datasets in  pandas dataframe\n\ntrain = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')\n\ntrain.describe()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "scrolled": true,
        "_uuid": "e79d47658b82d49905c39b7e66e3fcf03a86ced2"
      },
      "cell_type": "code",
      "source": "print (\"Size of train data : {}\" .format(train.shape))\n\nprint (\"Size of test data : {}\" .format(test.shape))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "99ad10c6432a461389c5c9b5ffe2f56ceb70d7a9"
      },
      "cell_type": "markdown",
      "source": "> That is a very large data set! We are going to have to do a lot of work to clean it up\n\n**Drop the Id column because we dont need it currently.**"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "1209490182f8356c09e7f43c1834f18e9ec8ba9e"
      },
      "cell_type": "code",
      "source": "#Save the 'Id' column\ntrain_ID = train['Id']\ntest_ID = test['Id']\n\n#Now drop the  'Id' colum since it's unnecessary for  the prediction process.\ntrain.drop(\"Id\", axis = 1, inplace = True)\ntest.drop(\"Id\", axis = 1, inplace = True)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "fca66e4aa038a4310fc6b70122b7e184ee0b765f",
        "scrolled": true
      },
      "cell_type": "code",
      "source": "print (\"Size of train data after dropping Id: {}\" .format(train.shape))\nprint (\"Size of test data after dropping Id: {}\" .format(test.shape))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "adab25c81ca5515fa0dea82196134266e769e933"
      },
      "cell_type": "markdown",
      "source": "## Dealing with outliers\n\nOutlinear in the GrLivArea is recommended by the author of the data  to remove it. The author says in documentation “I would recommend removing any houses with more than 4000 square feet from the data set (which eliminates these five unusual observations) before assigning it to students.”\n"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "589e1b7432290d42ec3ba5f527000d6de5c7fa90",
        "scrolled": true
      },
      "cell_type": "code",
      "source": "fig, ax = plt.subplots()\nax.scatter(x = train['GrLivArea'], y = train['SalePrice'])\nplt.ylabel('SalePrice', fontsize=13)\nplt.xlabel('GrLivArea', fontsize=13)\nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "190cfe132f4616d905bba693b79b49fde3e89319"
      },
      "cell_type": "markdown",
      "source": "We can see that there are outlinear with low SalePrice and high GrLivArea. This looks odd.\nWe need to remove it."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "e215c8fe40fdcbefa4480a4ac0a8c2adf4f68a5e"
      },
      "cell_type": "code",
      "source": "#Deleting outliers\ntrain = train.drop(train[(train['GrLivArea']>4000) & (train['SalePrice']<300000)].index)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "502ebe387edeeeb1e4a3f454d5dd004645a07e75"
      },
      "cell_type": "markdown",
      "source": "## Correlation Analysis\n\nLet see the most correlated features."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "aa18c6e3a818d58c4a14addedb21888a2ec610cb",
        "scrolled": true
      },
      "cell_type": "code",
      "source": "# most correlated features\ncorrmat = train.corr()\ntop_corr_features = corrmat.index[abs(corrmat[\"SalePrice\"])>0.5]\nplt.figure(figsize=(10,10))\ng = sns.heatmap(train[top_corr_features].corr(),annot=True,cmap=\"RdYlGn\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "784b45b84c0d1fe24b7476907a871556f657d1df"
      },
      "cell_type": "markdown",
      "source": "- From this we can tell which features **(OverallQual, GrLivArea and TotalBsmtSF )** are highly positively correlated with the SalePrice. \n- **GarageCars and GarageArea ** also seems correlated with other, Since the no. of car that will fit into the garage will depend on GarageArea. "
    },
    {
      "metadata": {
        "trusted": true,
        "scrolled": false,
        "_uuid": "2a7ab1e4534d622b1fd8d1a9c656607c7243b24a"
      },
      "cell_type": "code",
      "source": "sns.barplot(train.OverallQual,train.SalePrice)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "8c066456b14f788bd5f85f7ffbb271f039df4b18"
      },
      "cell_type": "markdown",
      "source": "**Scatter plots between 'SalePrice' and correlated variables**"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "df041ffe64ef1807796a75237a79fe846d73be82",
        "scrolled": false
      },
      "cell_type": "code",
      "source": "sns.set()\ncols = ['SalePrice', 'OverallQual', 'GrLivArea', 'GarageCars', 'TotalBsmtSF', 'FullBath', 'YearBuilt']\nsns.pairplot(train[cols], size = 2.5)\nplt.show();",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "a3d3e339ff75571f28b9c2787d3c56ab2a9895e9"
      },
      "cell_type": "markdown",
      "source": "One of the figures we may find interesting is the one between ** 'TotalBsmtSF' and 'GrLiveArea'. **\n\nWe can see the dots drawing a linear line, which almost acts like a border. It totally makes sense that the majority of the dots stay below that line. Basement areas can be equal to the above ground living area, but it is not expected a basement area bigger than the above ground living area"
    },
    {
      "metadata": {
        "trusted": true,
        "scrolled": true,
        "_uuid": "535da1efb310261ff5d59751be4b267ddf1bcd6c"
      },
      "cell_type": "code",
      "source": "sns.scatterplot(train.GrLivArea,train.TotalBsmtSF)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "7a8333803c5f5efb2fee696d0a2e50429858e575"
      },
      "cell_type": "markdown",
      "source": "## Target Variable Transform\nDifferent features in the data set may have values in different ranges. For example, in this data set, the range of SalePrice feature may lie from thousands to lakhs but the range of values of YearBlt feature will be in thousands. That means a column is more weighted compared to other.\n\n**Lets check the skewness of data**\n![Skew](https://cdn-images-1.medium.com/max/800/1*hxVvqttoCSkUT2_R1zA0Tg.gif)"
    },
    {
      "metadata": {
        "trusted": true,
        "scrolled": true,
        "_uuid": "3e82f95ef5565aadf0ad956f3f70a6fd8a40fe13"
      },
      "cell_type": "code",
      "source": "def check_skewness(col):\n    sns.distplot(train[col] , fit=norm);\n    fig = plt.figure()\n    res = stats.probplot(train[col], plot=plt)\n    # Get the fitted parameters used by the function\n    (mu, sigma) = norm.fit(train[col])\n    print( '\\n mu = {:.2f} and sigma = {:.2f}\\n'.format(mu, sigma))\n    \ncheck_skewness('SalePrice')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "c368eceae3e58e9b7ef37bb36737e90a1aa4d87b"
      },
      "cell_type": "markdown",
      "source": "**This distribution is positively skewed.** Notice that the black curve is more deviated towards the right. If you encounter that your predictive (response) variable is skewed, it is **recommended to fix the skewness** to make good decisions by the model.\n\n## Okay, So how do I fix the skewness?\nThe best way to fix it is to perform a **log transform** of the same data, with the intent to reduce the skewness."
    },
    {
      "metadata": {
        "trusted": true,
        "scrolled": true,
        "_uuid": "ad4524d38a8b0c31b3bac90b40daaa558cf7e91c"
      },
      "cell_type": "code",
      "source": "#We use the numpy fuction log1p which  applies log(1+x) to all elements of the column\ntrain[\"SalePrice\"] = np.log1p(train[\"SalePrice\"])\n\ncheck_skewness('SalePrice')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "fe44621d16a89a2bca2543638690a27e23cee36f"
      },
      "cell_type": "markdown",
      "source": "After taking logarithm of the same data the curve seems to be normally distributed, although not perfectly normal, this is sufficient to fix the issues from a skewed dataset as we saw before.\n\n**Important : If you log transform the response variable, it is required to also log transform feature variables that are skewed.**"
    },
    {
      "metadata": {
        "_uuid": "c814c57999220ce366d39db839cb4d4b9374198a"
      },
      "cell_type": "markdown",
      "source": "# Feature Engineering"
    },
    {
      "metadata": {
        "_uuid": "c5370e7b469139b439350faed9cf17ef01af1516"
      },
      "cell_type": "markdown",
      "source": "Here is the [Documentation](http://ww2.amstat.org/publications/jse/v19n3/Decock/DataDocumentation.txt) you can refer , to know more about the dataset.\n\n**Concatenate both train and test values.**"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "039f107f3a6b7379ff2627d48ab98c22737825f9"
      },
      "cell_type": "code",
      "source": "ntrain = train.shape[0]\nntest = test.shape[0]\ny_train = train.SalePrice.values\nall_data = pd.concat((train, test)).reset_index(drop=True)\nall_data.drop(['SalePrice'], axis=1, inplace=True)\nprint(\"all_data size is : {}\".format(all_data.shape))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "7a6776afb61d2a1637e3003204671d0c0c013ebf"
      },
      "cell_type": "markdown",
      "source": "# Missing Data"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ad3c34ad8df7a1bbb2b818a1aa0750f91645956f"
      },
      "cell_type": "code",
      "source": "all_data_na = (all_data.isnull().sum() / len(all_data)) * 100\nall_data_na = all_data_na.drop(all_data_na[all_data_na == 0].index).sort_values(ascending=False)[:30]\nmissing_data = pd.DataFrame({'Missing Ratio' :all_data_na})",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "30af0ffa63d2424affea6028b53c9709e08e0099",
        "scrolled": true
      },
      "cell_type": "code",
      "source": "f, ax = plt.subplots(figsize=(15, 12))\nplt.xticks(rotation='90')\nsns.barplot(x=all_data_na.index, y=all_data_na)\nplt.xlabel('Features', fontsize=15)\nplt.ylabel('Percent of missing values', fontsize=15)\nplt.title('Percent missing data by feature', fontsize=15)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "00773902333384da2515e01f5cd550d5c6a6812f",
        "scrolled": true
      },
      "cell_type": "code",
      "source": "all_data.PoolQC.loc[all_data.PoolQC.notnull()]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "116000b37a24504895a7b8c7f50bb05831f203c8"
      },
      "cell_type": "markdown",
      "source": "**GarageType,  GarageFinish, GarageQual,  GarageCond, GarageYrBlt,  GarageArea,  GarageCars  these all features have same percentage of null values.**"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "dd7cdb893c4da5a59fed139f446c6afa4dbbae8e",
        "scrolled": false
      },
      "cell_type": "code",
      "source": "abc = ['GarageType', 'GarageFinish', 'GarageQual', 'GarageCond','GarageYrBlt', 'GarageArea', 'GarageCars']\ngrp = all_data[all_data['GarageType'].notnull() == True].groupby('GarageType')[abc]\nfor key,item in grp:\n    print(grp.get_group(key))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "8e2db4ef580f375e4dc7182afd83da4efea5241f"
      },
      "cell_type": "markdown",
      "source": "# Handle Missing Data"
    },
    {
      "metadata": {
        "_uuid": "226670f827161cbd4cd5f15c8fae4c83490f851f"
      },
      "cell_type": "markdown",
      "source": "Since PoolQC has the highest null values according to the data documentation says **null values means 'No Pool.**\nSince majority of houses has no pool.\nSo we will replace those null values with 'None'."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "e94513f7cb4dfd3f5e87774b736a39fa1e65a66b"
      },
      "cell_type": "code",
      "source": "all_data[\"PoolQC\"] = all_data[\"PoolQC\"].fillna(\"None\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "f0f24b2ecc5283adf1b777b4b6209ecaba69d2f9"
      },
      "cell_type": "markdown",
      "source": "* **MiscFeature** : Data documentation says NA means \"no misc feature\""
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "f8be0d01feb00595640a0ae90db0d877d361b46a"
      },
      "cell_type": "code",
      "source": "all_data[\"MiscFeature\"] = all_data[\"MiscFeature\"].fillna(\"None\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "22dd51000c69deb9b7debe72ec800cbd42d731e4"
      },
      "cell_type": "markdown",
      "source": "* **Alley** : data description says NA means \"no alley access\"\n"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "09c05ccc29ed49353365d6a0f74d715cb3b4e4da"
      },
      "cell_type": "code",
      "source": "all_data[\"Alley\"] = all_data[\"Alley\"].fillna(\"None\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "0b54e59904fc40a3bc1a1a937c890c145cef6f87"
      },
      "cell_type": "markdown",
      "source": "* **Fence** : data description says NA means \"no fence\"\n"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "c7bfad1d5982ea62ed418b5a556f5da363c8f99d"
      },
      "cell_type": "code",
      "source": "all_data[\"Fence\"] = all_data[\"Fence\"].fillna(\"None\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "ebfb8252b25107a6417882fb5244a8e808bdf324"
      },
      "cell_type": "markdown",
      "source": "* **FireplaceQu** : data description says NA means \"no fireplace\""
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "2eb64243b5e2eb759411e21d0d8e0c80cc6d16f7"
      },
      "cell_type": "code",
      "source": "all_data[\"FireplaceQu\"] = all_data[\"FireplaceQu\"].fillna(\"None\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "af166059bccd2bcc56d684f233a7b9a8a9c2e6ea"
      },
      "cell_type": "markdown",
      "source": "* **LotFrontage** : Since the area of each street connected to the house property most likely have a similar area to other houses in its neighborhood , we can fill in missing values by the median LotFrontage of the neighborhood."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "3ae0bce2d27efc9ed1eb629a73f93c643f309c66",
        "scrolled": false
      },
      "cell_type": "code",
      "source": "# Grouping by Neighborhood and Check the LotFrontage. Most of the grouping has similar areas\ngrouped_df = all_data.groupby('Neighborhood')['LotFrontage']\n\nfor key, item in grouped_df:\n    print(key,\"\\n\")\n    print(grouped_df.get_group(key), \"\\n\\n\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "1e899c274313d88114288c1e0fa720468da1afee"
      },
      "cell_type": "code",
      "source": "#Group by neighborhood and fill in missing value by the median LotFrontage of all the neighborhood\nall_data[\"LotFrontage\"] = all_data.groupby(\"Neighborhood\")[\"LotFrontage\"].transform(\n    lambda x: x.fillna(x.median()))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "e1493a84b2e9ebef26d03295511d52830343ace4"
      },
      "cell_type": "markdown",
      "source": "* **GarageType, GarageFinish, GarageQual and GarageCond** : Replacing missing data with None as per documentation. "
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "7a71d08cd1dd160b5184cfd8a681503ec0d92e95"
      },
      "cell_type": "code",
      "source": "for col in ['GarageType', 'GarageFinish', 'GarageQual', 'GarageCond']:\n    all_data[col] = all_data[col].fillna('None')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "0f30a319433e86234355ee6d08ef8e95a23285d3"
      },
      "cell_type": "code",
      "source": "abc = ['GarageType', 'GarageFinish', 'GarageQual', 'GarageCond','GarageYrBlt', 'GarageArea', 'GarageCars']\nall_data.groupby('GarageType')[abc].count()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "197c97edb46edc4ee356d587c66a9a9fca41a2be"
      },
      "cell_type": "markdown",
      "source": "* **GarageYrBlt, GarageArea and GarageCars** : Replacing missing data with 0 (Since No garage = no cars in such garage.)"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "e1c064cc7d91c1f318bbea0be6350a3da14b74cc"
      },
      "cell_type": "code",
      "source": "for col in ('GarageYrBlt', 'GarageArea', 'GarageCars'):\n    all_data[col] = all_data[col].fillna(0)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "8cd265faa79ccd0366ec63d691059933153feafc"
      },
      "cell_type": "markdown",
      "source": "* **BsmtFinSF1, BsmtFinSF2, BsmtUnfSF, TotalBsmtSF, BsmtFullBath and BsmtHalfBath** : missing values are likely zero for having no basement"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ed2d1168ba88ed3f1aed1e0843f9f57e7cd2046d"
      },
      "cell_type": "code",
      "source": "for col in ('BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF','TotalBsmtSF', 'BsmtFullBath', 'BsmtHalfBath'):\n    all_data[col] = all_data[col].fillna(0)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "587e73a91191f0ea1cc0fd12b21d8fe45eb403ec"
      },
      "cell_type": "markdown",
      "source": "* **BsmtQual, BsmtCond, BsmtExposure, BsmtFinType1 and BsmtFinType2** : For all these categorical basement-related features, NaN means that there is no basement."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "6de1156fca72633eb45fbb8c7c926613455ef25f"
      },
      "cell_type": "code",
      "source": "for col in ('BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2'):\n    all_data[col] = all_data[col].fillna('None')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "737f23b205e2cc84486ddb7a087509303f4b7a5b"
      },
      "cell_type": "markdown",
      "source": "* **MasVnrArea and MasVnrType** : NA most likely means no masonry veneer for these houses. We can fill 0 for the area and None for the type."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "de69391d40cc5c23765711f4ab248bf61b44bda5"
      },
      "cell_type": "code",
      "source": "all_data[\"MasVnrType\"] = all_data[\"MasVnrType\"].fillna(\"None\")\nall_data[\"MasVnrArea\"] = all_data[\"MasVnrArea\"].fillna(0)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "af7a8d719e79c38f25758488c4f4cb695e3e1976"
      },
      "cell_type": "markdown",
      "source": "* **MSZoning (The general zoning classification)** : 'RL' is by far the most common value. So we can fill in missing values with 'RL'"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ac0a52cd03ada0f9ab04f219972342753321798e"
      },
      "cell_type": "code",
      "source": "all_data['MSZoning'].value_counts()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "7b92b9424d4810edb0d60b03c9316bf8f3a85263"
      },
      "cell_type": "code",
      "source": "all_data['MSZoning'] = all_data['MSZoning'].fillna(all_data['MSZoning'].mode()[0])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "19602e120fbb71d1fb38290e2330df79b6a053a0"
      },
      "cell_type": "markdown",
      "source": "* **Utilities** : Since this is a categorical data and most of the data are of same category, Its not gonna effect on model. So we choose to drop it."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "7e44495f79c3deb3757a6f5c23ed597bea0782de"
      },
      "cell_type": "code",
      "source": "all_data['Utilities'].value_counts()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "25c2a42c8d23197a1db22b487ef8692da1063e8c"
      },
      "cell_type": "code",
      "source": "all_data = all_data.drop(['Utilities'], axis=1)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "b82cc3582a64e4d20ead3cd386b2bfb467684353"
      },
      "cell_type": "markdown",
      "source": "* **Functional** : data description says NA means typical"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "af05ef25e7ee2c0e4df51b2df09424112e7fc72f"
      },
      "cell_type": "code",
      "source": "all_data[\"Functional\"] = all_data[\"Functional\"].fillna(\"Typ\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "4e9723638293fceec4190dcccc49efd719a28147"
      },
      "cell_type": "markdown",
      "source": "* **Electrical,KitchenQual, Exterior1st, Exterior2nd, SaleType** : Since this all are categorical values so its better to replace nan values with the most used keyword."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "fd68c6cc53b0f09c99fbaae220455358acba10a5"
      },
      "cell_type": "code",
      "source": "mode_col = ['Electrical','KitchenQual', 'Exterior1st', 'Exterior2nd', 'SaleType']\nfor col in mode_col:\n    all_data[col] = all_data[col].fillna(all_data[col].mode()[0])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "9533bc18b3a5508e78f32471cbacc0d84114a0cc"
      },
      "cell_type": "markdown",
      "source": "* **MSSubClass** : Na most likely means No building class. We can replace missing values with None\n"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "0aa7e338dca4dbb303563fd401bddf69fa517a45"
      },
      "cell_type": "code",
      "source": "all_data['MSSubClass'] = all_data['MSSubClass'].fillna(\"None\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "1dece935c4d58ebbbdc20b476ac1ba0c1f49398d"
      },
      "cell_type": "markdown",
      "source": "## Lets check for any missing values"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "cb249dde6e25900ed562f1106c652c63a2aef72e"
      },
      "cell_type": "code",
      "source": "#Check remaining missing values if any \nall_data_na = (all_data.isnull().sum() / len(all_data)) * 100\nall_data_na = all_data_na.drop(all_data_na[all_data_na == 0].index).sort_values(ascending=False)\nmissing_data = pd.DataFrame({'Missing Ratio' :all_data_na})\nmissing_data.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "fce152364a35ce50b50dc77200f9c25c4709a3c5"
      },
      "cell_type": "markdown",
      "source": "**Now there any many features that are numerical but categorical.**"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "c99d01bd5a5ad8deccf993cc5c571b5f7f11741b"
      },
      "cell_type": "code",
      "source": "all_data['OverallCond'].value_counts()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "713f061e916a6d7326a86f64657c2a8573cf98b7"
      },
      "cell_type": "markdown",
      "source": "**Converting some numerical variables that are really categorical type.**\n\nAs you can see the category range from 1 to 9 which are numerical (**not ordinal type**). Since its categorical we need to change it to String type.\n\nIf we do not convert these to categorical, some model may get affect by this as model will compare the value 1<5<10 . We dont need that to happen with our model."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "62ebb5cfcd932164176a9614bc688a1107799415"
      },
      "cell_type": "code",
      "source": "#MSSubClass=The building class\nall_data['MSSubClass'] = all_data['MSSubClass'].apply(str)\n\n\n#Changing OverallCond into a categorical variable\nall_data['OverallCond'] = all_data['OverallCond'].astype(str)\n\n\n#Year and month sold are transformed into categorical features.\nall_data['YrSold'] = all_data['YrSold'].astype(str)\nall_data['MoSold'] = all_data['MoSold'].astype(str)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "4ca8b2ed84fcc54557f4d7b65e4aad367395ab6b"
      },
      "cell_type": "markdown",
      "source": "## Label Encoding \nAs you might know by now, we can’t have text in our data if we’re going to run any kind of model on it. So before we can run a model, we need to make this data ready for the model.\n\nAnd to convert this kind of categorical text data into model-understandable numerical data, we use the Label Encoder class.\n\nSuppose, we have a feature State which has 3 category i.e India , France, China . So, Label Encoder will categorize them as 0, 1, 2."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "8308b428bbf30e9e17f17b57a230ed2297081370"
      },
      "cell_type": "code",
      "source": "from sklearn.preprocessing import LabelEncoder\ncols = ('FireplaceQu', 'BsmtQual', 'BsmtCond', 'GarageQual', 'GarageCond', \n        'ExterQual', 'ExterCond','HeatingQC', 'PoolQC', 'KitchenQual', 'BsmtFinType1', \n        'BsmtFinType2', 'Functional', 'Fence', 'BsmtExposure', 'GarageFinish', 'LandSlope',\n        'LotShape', 'PavedDrive', 'Street', 'Alley', 'CentralAir', 'MSSubClass', 'OverallCond', \n        'YrSold', 'MoSold')\n# process columns, apply LabelEncoder to categorical features\nfor c in cols:\n    lbl = LabelEncoder() \n    lbl.fit(list(all_data[c].values)) \n    all_data[c] = lbl.transform(list(all_data[c].values))\n\n# shape        \nprint('Shape all_data: {}'.format(all_data.shape))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "865590a345058e1fd3bb76434441508a0204f6fd"
      },
      "cell_type": "markdown",
      "source": "Since area related features are very important to determine house prices, we add one more feature which is the total area of basement, first and second floor areas of each house"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b4ddb4163fc92e8a9b2efdb3957965e8d6d65573"
      },
      "cell_type": "code",
      "source": "# Adding total sqfootage feature \nall_data['TotalSF'] = all_data['TotalBsmtSF'] + all_data['1stFlrSF'] + all_data['2ndFlrSF']",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "2c4c4d9d412c284b0aa9f34653c193227e36ce07"
      },
      "cell_type": "markdown",
      "source": "**Lets see the highly skewed features we have**"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "0b3f732620e7b42d8c8592c0bef8e9030bcea37c",
        "scrolled": true
      },
      "cell_type": "code",
      "source": "numeric_feats = all_data.dtypes[all_data.dtypes != \"object\"].index\n\n# Check the skew of all numerical features\nskewed_feats = all_data[numeric_feats].apply(lambda x: skew(x.dropna())).sort_values(ascending=False)\nprint(\"\\nSkew in numerical features: \\n\")\nskewness = pd.DataFrame({'Skew' :skewed_feats})\nskewness.head(15)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "8917aa598b2df98fd173d63bde1f1f941d919d86"
      },
      "cell_type": "markdown",
      "source": "## Box Cox Transformation of (highly) skewed features\n\nWhen you are dealing with real-world data, you are going to deal with features that are heavily skewed. Transformation technique is useful to **stabilize variance**, make the data **more normal distribution-like**, improve the validity of measures of association.\n\nThe problem with the Box-Cox Transformation is **estimating lambda**. This value will depend on the existing data, and should be considered when performing cross validation on out of sample datasets."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "04b1a8240b20f470d326c1f480a0061712c30843"
      },
      "cell_type": "code",
      "source": "skewness = skewness[abs(skewness) > 0.75]\nprint(\"There are {} skewed numerical features to Box Cox transform\".format(skewness.shape[0]))\n\nfrom scipy.special import boxcox1p\nskewed_features = skewness.index\nlam = 0.15\nfor feat in skewed_features:\n    #all_data[feat] += 1\n    all_data[feat] = boxcox1p(all_data[feat], lam)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "aef52487095b221359382ce9135c709541c03958"
      },
      "cell_type": "markdown",
      "source": "**Getting dummy categorical features**"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "d3056bb177dd80797d752d80e0d9d943486e482a",
        "scrolled": true
      },
      "cell_type": "code",
      "source": "all_data = pd.get_dummies(all_data)\nall_data.shape",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "142bcae6537641dc5307beb7186a1a9a709fb21a"
      },
      "cell_type": "markdown",
      "source": "Creating train and test data."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "0c986a9c705e012679c661f03a017399978d6ebd"
      },
      "cell_type": "code",
      "source": "train = all_data[:ntrain]\ntest = all_data[ntrain:]\ntrain.shape",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "d5c3bdaf7c57955a06d4537e93ad7036af1e54f1"
      },
      "cell_type": "markdown",
      "source": "## Lets apply Modelling\n\n1. Importing Libraries\n\n2. We will use models\n - Lasso\n - Ridge\n - ElasticNet\n - Gradient Boosting\n \n3. Find the Cross Validation Score.\n4. Calculate the mean of all model's prediction.\n5. Submit the CSV file.\n "
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "a477c9212c17282e5c8a1767ca6b301e91afcce3"
      },
      "cell_type": "code",
      "source": "from sklearn.linear_model import ElasticNet, Lasso,  BayesianRidge, LassoLarsIC\nfrom sklearn.ensemble import RandomForestRegressor,  GradientBoostingRegressor\nfrom sklearn.kernel_ridge import KernelRidge\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import RobustScaler\nfrom sklearn.base import BaseEstimator, TransformerMixin, RegressorMixin, clone\nfrom sklearn.model_selection import KFold, cross_val_score, train_test_split\nfrom sklearn.metrics import mean_squared_error\nimport xgboost as xgb\nimport lightgbm as lgb",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "a907ce9421f8a0b628390c5db13920f9a392d476"
      },
      "cell_type": "markdown",
      "source": "## Cross Validation\nIt's simple way to calculate error for evaluation. \n\n**KFold( )** splits the train/test data into k consecutive folds, we also have made shuffle attrib to True.\n\n**cross_val_score ( )** evaluate a score by cross-validation."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "d7af5d935df2a27cfc54dc13f746aa64774d010f"
      },
      "cell_type": "code",
      "source": "#Validation function\nn_folds = 5\n\ndef rmsle_cv(model):\n    kf = KFold(n_folds, shuffle=True, random_state=42).get_n_splits(train.values)\n    rmse= np.sqrt(-cross_val_score(model, train.values, y_train, scoring=\"neg_mean_squared_error\", cv = kf))\n    return(rmse)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "ef4bf7c58fdfedd15102194023c24b94876f3559"
      },
      "cell_type": "markdown",
      "source": "# Modelling\nSince in this dataset we have a large set of features. So to make our model avoid Overfitting and noisy we will use Regularization.\nThese model have Regularization parameter.\n\nRegularization will reduce the magnitude of the coefficients."
    },
    {
      "metadata": {
        "_uuid": "c7cd0953ca1b7021b165aef700d1241732c09d18"
      },
      "cell_type": "markdown",
      "source": "## Ridge Regression\n- It shrinks the parameters, therefore it is mostly used to prevent multicollinearity.\n- It reduces the model complexity by coefficient shrinkage.\n- It uses L2 regularization technique."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b7f81c6d917d9c5325f3d3456bde7adce2899621"
      },
      "cell_type": "code",
      "source": "KRR = KernelRidge(alpha=0.6, kernel='polynomial', degree=2, coef0=2.5)\nscore = rmsle_cv(KRR)\nprint(\"Kernel Ridge score: {:.4f} ({:.4f})\\n\".format(score.mean(), score.std()))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "03559fcf57b62b49b0332acd9439274bf8dd9d8a"
      },
      "cell_type": "markdown",
      "source": "## Lasso Regression\nLASSO (Least Absolute Shrinkage Selector Operator), is quite similar to ridge.\n\nIn case of lasso, even at smaller alpha’s, our coefficients are reducing to absolute zeroes.\n Therefore, lasso selects the only some feature while reduces the coefficients of others to zero. This property is known as feature selection and which is absent in case of ridge.\n \n- Lasso uses L1 regularization technique.\n- Lasso is generally used when we have more number of features, because it automatically does feature selection.\n"
    },
    {
      "metadata": {
        "trusted": true,
        "scrolled": true,
        "_uuid": "a437402e2a37f26372fc03761fa05c6d7ea4e433"
      },
      "cell_type": "code",
      "source": "lasso = make_pipeline(RobustScaler(), Lasso(alpha =0.0005, random_state=1))\nscore = rmsle_cv(lasso)\nprint(\"Lasso score: {:.4f} ({:.4f})\\n\".format(score.mean(), score.std()))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "c854d61f20f4cdab8a37b5037c3908f783b5e644"
      },
      "cell_type": "markdown",
      "source": "## Elastic Net Regression\n\nElastic net is basically a combination of both L1 and L2 regularization. So if you know elastic net, you can implement both Ridge and Lasso by tuning the parameters."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "d06ca9a5f9db49890db7999ffe9db7333f02edc6"
      },
      "cell_type": "code",
      "source": "ENet = make_pipeline(RobustScaler(), ElasticNet(alpha=0.0005, l1_ratio=.9, random_state=3))\nscore = rmsle_cv(ENet)\nprint(\"ElasticNet score: {:.4f} ({:.4f})\\n\".format(score.mean(), score.std()))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "120e4a5ed78c963d8278803cc00956781f605691"
      },
      "cell_type": "markdown",
      "source": "## Gradient Boosting Regression\nRefer [here](https://medium.com/mlreview/gradient-boosting-from-scratch-1e317ae4587d)"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "221e05d63ac4d3f99900b36a9d06e3d8e10f1dc7"
      },
      "cell_type": "code",
      "source": "GBoost = GradientBoostingRegressor(n_estimators=3000, learning_rate=0.05,\n                                   max_depth=4, max_features='sqrt',\n                                   min_samples_leaf=15, min_samples_split=10, \n                                   loss='huber', random_state =5)\nscore = rmsle_cv(GBoost)\nprint(\"Gradient Boosting score: {:.4f} ({:.4f})\\n\".format(score.mean(), score.std()))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "7c0a8d859b163d5f10ae59afdd6fd77664bcd907"
      },
      "cell_type": "markdown",
      "source": "**Fit the training dataset on every model**"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "f2a04068b4c93f1a1851708d9f43edcef6990cb8"
      },
      "cell_type": "code",
      "source": "LassoMd = lasso.fit(train.values,y_train)\nENetMd = ENet.fit(train.values,y_train)\nKRRMd = KRR.fit(train.values,y_train)\nGBoostMd = GBoost.fit(train.values,y_train)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "b5b2fe94eaa417646e8a91b451db40b400e88bf6"
      },
      "cell_type": "markdown",
      "source": "## Mean of all model's prediction.\nnp.expm1 ( ) is used to calculate exp(x) - 1 for all elements in the array. "
    },
    {
      "metadata": {
        "trusted": true,
        "scrolled": true,
        "_uuid": "8b4e59cad1a1499ba00c3206676676658a4b1881"
      },
      "cell_type": "code",
      "source": "finalMd = (np.expm1(LassoMd.predict(test.values)) + np.expm1(ENetMd.predict(test.values)) + np.expm1(KRRMd.predict(test.values)) + np.expm1(GBoostMd.predict(test.values)) ) / 4\nfinalMd",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "0cd4ca41a0ecde7f82204ac5ed6fe6b502ea4a87"
      },
      "cell_type": "markdown",
      "source": "## Submission"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "708eb2603a04b74c2c19ed52f5130c5f5704cf0f"
      },
      "cell_type": "code",
      "source": "sub = pd.DataFrame()\nsub['Id'] = test_ID\nsub['SalePrice'] = finalMd\nsub.to_csv('submission.csv',index=False)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "b6797c8782e8aee1a187da8b2c65b67803853439"
      },
      "cell_type": "markdown",
      "source": "**If you found this notebook helpful or you just liked it , some upvotes would be very much appreciated.**\n\n**I'll be glad to hear suggestions on improving my models**"
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.6",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 1
}