Programming assignment, week 4: Ensembles/Programming_assignment_week_4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Version 1.0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check your versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numpy 1.13.1\n",
      "pandas 0.20.3\n",
      "scipy 0.19.1\n",
      "sklearn 0.19.0\n",
      "lightgbm 2.0.6\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd \n",
    "import sklearn\n",
    "import scipy.sparse \n",
    "import lightgbm \n",
    "\n",
    "for p in [np, pd, scipy, sklearn, lightgbm]:\n",
    "    print (p.__name__, p.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important!** There is a huge chance that the assignment will be impossible to pass if the versions of `lighgbm` and `scikit-learn` are wrong. The versions being tested:\n",
    "\n",
    "    numpy 1.13.1\n",
    "    pandas 0.20.3\n",
    "    scipy 0.19.1\n",
    "    sklearn 0.19.0\n",
    "    ligthgbm 2.0.6\n",
    "    \n",
    "\n",
    "To install an older version of `lighgbm` you may use the following command:\n",
    "```\n",
    "pip uninstall lightgbm\n",
    "pip install lightgbm==2.0.6\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ensembling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this programming assignment you are asked to implement two ensembling schemes: simple linear mix and stacking.\n",
    "\n",
    "We will spend several cells to load data and create feature matrix, you can scroll down this part or try to understand what's happening."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import gc\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline \n",
    "\n",
    "pd.set_option('display.max_rows', 600)\n",
    "pd.set_option('display.max_columns', 50)\n",
    "\n",
    "import lightgbm as lgb\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.metrics import r2_score\n",
    "from tqdm import tqdm_notebook\n",
    "\n",
    "from itertools import product\n",
    "\n",
    "\n",
    "def downcast_dtypes(df):\n",
    "    '''\n",
    "        Changes column types in the dataframe: \n",
    "                \n",
    "                `float64` type to `float32`\n",
    "                `int64`   type to `int32`\n",
    "    '''\n",
    "    \n",
    "    # Select columns to downcast\n",
    "    float_cols = [c for c in df if df[c].dtype == \"float64\"]\n",
    "    int_cols =   [c for c in df if df[c].dtype == \"int64\"]\n",
    "    \n",
    "    # Downcast\n",
    "    df[float_cols] = df[float_cols].astype(np.float32)\n",
    "    df[int_cols]   = df[int_cols].astype(np.int32)\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load data subset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the data from the hard drive first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sales = pd.read_csv('../readonly/final_project_data/sales_train.csv.gz')\n",
    "shops = pd.read_csv('../readonly/final_project_data/shops.csv')\n",
    "items = pd.read_csv('../readonly/final_project_data/items.csv')\n",
    "item_cats = pd.read_csv('../readonly/final_project_data/item_categories.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And use only 3 shops for simplicity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sales = sales[sales['shop_id'].isin([26, 27, 28])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get a feature matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now need to prepare the features. This part is all implemented for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create \"grid\" with columns\n",
    "index_cols = ['shop_id', 'item_id', 'date_block_num']\n",
    "\n",
    "# For every month we create a grid from all shops/items combinations from that month\n",
    "grid = [] \n",
    "for block_num in sales['date_block_num'].unique():\n",
    "    cur_shops = sales.loc[sales['date_block_num'] == block_num, 'shop_id'].unique()\n",
    "    cur_items = sales.loc[sales['date_block_num'] == block_num, 'item_id'].unique()\n",
    "    grid.append(np.array(list(product(*[cur_shops, cur_items, [block_num]])),dtype='int32'))\n",
    "\n",
    "# Turn the grid into a dataframe\n",
    "grid = pd.DataFrame(np.vstack(grid), columns = index_cols,dtype=np.int32)\n",
    "\n",
    "# Groupby data to get shop-item-month aggregates\n",
    "gb = sales.groupby(index_cols,as_index=False).agg({'item_cnt_day':{'target':'sum'}})\n",
    "# Fix column names\n",
    "gb.columns = [col[0] if col[-1]=='' else col[-1] for col in gb.columns.values] \n",
    "# Join it to the grid\n",
    "all_data = pd.merge(grid, gb, how='left', on=index_cols).fillna(0)\n",
    "\n",
    "# Same as above but with shop-month aggregates\n",
    "gb = sales.groupby(['shop_id', 'date_block_num'],as_index=False).agg({'item_cnt_day':{'target_shop':'sum'}})\n",
    "gb.columns = [col[0] if col[-1]=='' else col[-1] for col in gb.columns.values]\n",
    "all_data = pd.merge(all_data, gb, how='left', on=['shop_id', 'date_block_num']).fillna(0)\n",
    "\n",
    "# Same as above but with item-month aggregates\n",
    "gb = sales.groupby(['item_id', 'date_block_num'],as_index=False).agg({'item_cnt_day':{'target_item':'sum'}})\n",
    "gb.columns = [col[0] if col[-1] == '' else col[-1] for col in gb.columns.values]\n",
    "all_data = pd.merge(all_data, gb, how='left', on=['item_id', 'date_block_num']).fillna(0)\n",
    "\n",
    "# Downcast dtypes from 64 to 32 bit to save memory\n",
    "all_data = downcast_dtypes(all_data)\n",
    "del grid, gb \n",
    "gc.collect();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After creating a grid, we can calculate some features. We will use lags from [1, 2, 3, 4, 5, 12] months ago."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# List of columns that we will use to create lags\n",
    "cols_to_rename = list(all_data.columns.difference(index_cols)) \n",
    "\n",
    "shift_range = [1, 2, 3, 4, 5, 12]\n",
    "\n",
    "for month_shift in tqdm_notebook(shift_range):\n",
    "    train_shift = all_data[index_cols + cols_to_rename].copy()\n",
    "    \n",
    "    train_shift['date_block_num'] = train_shift['date_block_num'] + month_shift\n",
    "    \n",
    "    foo = lambda x: '{}_lag_{}'.format(x, month_shift) if x in cols_to_rename else x\n",
    "    train_shift = train_shift.rename(columns=foo)\n",
    "\n",
    "    all_data = pd.merge(all_data, train_shift, on=index_cols, how='left').fillna(0)\n",
    "\n",
    "del train_shift\n",
    "\n",
    "# Don't use old data from year 2013\n",
    "all_data = all_data[all_data['date_block_num'] >= 12] \n",
    "\n",
    "# List of all lagged features\n",
    "fit_cols = [col for col in all_data.columns if col[-1] in [str(item) for item in shift_range]] \n",
    "# We will drop these at fitting stage\n",
    "to_drop_cols = list(set(list(all_data.columns)) - (set(fit_cols)|set(index_cols))) + ['date_block_num'] \n",
    "\n",
    "# Category for each item\n",
    "item_category_mapping = items[['item_id','item_category_id']].drop_duplicates()\n",
    "\n",
    "all_data = pd.merge(all_data, item_category_mapping, how='left', on='item_id')\n",
    "all_data = downcast_dtypes(all_data)\n",
    "gc.collect();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To this end, we've created a feature matrix. It is stored in `all_data` variable. Take a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train/test split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a sake of the programming assignment, let's artificially split the data into train and test. We will treat last month data as the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Save `date_block_num`, as we can't use them as features, but will need them to split the dataset into parts \n",
    "dates = all_data['date_block_num']\n",
    "\n",
    "last_block = dates.max()\n",
    "print('Test `date_block_num` is %d' % last_block)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dates_train = dates[dates <  last_block]\n",
    "dates_test  = dates[dates == last_block]\n",
    "\n",
    "X_train = all_data.loc[dates <  last_block].drop(to_drop_cols, axis=1)\n",
    "X_test =  all_data.loc[dates == last_block].drop(to_drop_cols, axis=1)\n",
    "\n",
    "y_train = all_data.loc[dates <  last_block, 'target'].values\n",
    "y_test =  all_data.loc[dates == last_block, 'target'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# First level models "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to implement a basic stacking scheme. We have a time component here, so we will use ***scheme f)*** from the reading material. Recall, that we always use first level models to build two datasets: test meta-features and 2-nd level train-metafetures. Let's see how we get test meta-features first. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test meta-features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firts, we will run *linear regression* on numeric columns and get predictions for the last month."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lr = LinearRegression()\n",
    "lr.fit(X_train.values, y_train)\n",
    "pred_lr = lr.predict(X_test.values)\n",
    "\n",
    "print('Test R-squared for linreg is %f' % r2_score(y_test, pred_lr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the we run *LightGBM*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lgb_params = {\n",
    "               'feature_fraction': 0.75,\n",
    "               'metric': 'rmse',\n",
    "               'nthread':1, \n",
    "               'min_data_in_leaf': 2**7, \n",
    "               'bagging_fraction': 0.75, \n",
    "               'learning_rate': 0.03, \n",
    "               'objective': 'mse', \n",
    "               'bagging_seed': 2**7, \n",
    "               'num_leaves': 2**7,\n",
    "               'bagging_freq':1,\n",
    "               'verbose':0 \n",
    "              }\n",
    "\n",
    "model = lgb.train(lgb_params, lgb.Dataset(X_train, label=y_train), 100)\n",
    "pred_lgb = model.predict(X_test)\n",
    "\n",
    "print('Test R-squared for LightGBM is %f' % r2_score(y_test, pred_lgb))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, concatenate test predictions to get test meta-features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_test_level2 = np.c_[pred_lr, pred_lgb] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train meta-features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Now it is your turn to write the code**. You need to implement ***scheme f)*** from the reading material. Here, we will use duration **T** equal to month and **M=15**.  \n",
    "\n",
    "That is, you need to get predictions (meta-features) from *linear regression* and *LightGBM* for months 27, 28, 29, 30, 31, 32. Use the same parameters as in above models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dates_train_level2 = dates_train[dates_train.isin([27, 28, 29, 30, 31, 32])]\n",
    "\n",
    "# That is how we get target for the 2nd level dataset\n",
    "y_train_level2 = y_train[dates_train.isin([27, 28, 29, 30, 31, 32])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# And here we create 2nd level feeature matrix, init it with zeros first\n",
    "X_train_level2 = np.zeros([y_train_level2.shape[0], 2])\n",
    "\n",
    "# Now fill `X_train_level2` with metafeatures\n",
    "for cur_block_num in [27, 28, 29, 30, 31, 32]:\n",
    "    \n",
    "    print(cur_block_num)\n",
    "    \n",
    "    '''\n",
    "        1. Split `X_train` into parts\n",
    "           Remember, that corresponding dates are stored in `dates_train` \n",
    "        2. Fit linear regression \n",
    "        3. Fit LightGBM and put predictions          \n",
    "        4. Store predictions from 2. and 3. in the right place of `X_train_level2`. \n",
    "           You can use `dates_train_level2` for it\n",
    "           Make sure the order of the meta-features is the same as in `X_test_level2`\n",
    "    '''      \n",
    "    \n",
    "    #  YOUR CODE GOES HERE\n",
    "    \n",
    "    \n",
    "# Sanity check\n",
    "assert np.all(np.isclose(X_train_level2.mean(axis=0), [ 1.50148988,  1.38811989]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, the ensembles work best, when first level models are diverse. We can qualitatively analyze the diversity by examinig *scatter plot* between the two metafeatures. Plot the scatter plot below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# YOUR CODE GOES HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ensembling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, when the meta-features are created, we can ensemble our first level models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple convex mix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with simple linear convex mix:\n",
    "\n",
    "$$\n",
    "mix= \\alpha\\cdot\\text{linreg_prediction}+(1-\\alpha)\\cdot\\text{lgb_prediction}\n",
    "$$\n",
    "\n",
    "We need to find an optimal $\\alpha$. And it is very easy, as it is feasible to do grid search. Next, find the optimal $\\alpha$ out of `alphas_to_try` array. Remember, that you need to use train meta-features (not test) when searching for $\\alpha$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "alphas_to_try = np.linspace(0, 1, 1001)\n",
    "\n",
    "# YOUR CODE GOES HERE\n",
    "best_alpha = # YOUR CODE GOES HERE\n",
    "r2_train_simple_mix = # YOUR CODE GOES HERE\n",
    "\n",
    "print('Best alpha: %f; Corresponding r2 score on train: %f' % (best_alpha, r2_train_simple_mix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now use the $\\alpha$ you've found to compute predictions for the test set "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_preds = # YOUR CODE GOES HERE\n",
    "r2_test_simple_mix = # YOUR CODE GOES HERE\n",
    "\n",
    "print('Test R-squared for simple mix is %f' % r2_test_simple_mix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stacking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will try a more advanced ensembling technique. Fit a linear regression model to the meta-features. Use the same parameters as in the model above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# YOUR CODE GOES HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute R-squared on the train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_preds = # YOUR CODE GOES HERE\n",
    "r2_train_stacking = # YOUR CODE GOES HERE\n",
    "\n",
    "test_preds = # YOUR CODE GOES HERE\n",
    "r2_test_stacking = # YOUR CODE GOES HERE\n",
    "\n",
    "print('Train R-squared for stacking is %f' % r2_train_stacking)\n",
    "print('Test  R-squared for stacking is %f' % r2_test_stacking)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interesting, that the score turned out to be lower than in previous method. Although the model is very simple (just 3 parameters) and, in fact, mixes predictions linearly, it looks like it managed to overfit. **Examine and compare** train and test scores for the two methods. \n",
    "\n",
    "And of course this particular case does not mean simple mix is always better than stacking."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We all done! Submit everything we need to the grader now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from grader import Grader\n",
    "grader = Grader()\n",
    "\n",
    "grader.submit_tag('best_alpha', best_alpha)\n",
    "\n",
    "grader.submit_tag('r2_train_simple_mix', r2_train_simple_mix)\n",
    "grader.submit_tag('r2_test_simple_mix',  r2_test_simple_mix)\n",
    "\n",
    "grader.submit_tag('r2_train_stacking', r2_train_stacking)\n",
    "grader.submit_tag('r2_test_stacking',  r2_test_stacking)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "STUDENT_EMAIL = # EMAIL HERE\n",
    "STUDENT_TOKEN = # TOKEN HERE\n",
    "grader.status()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grader.submit(STUDENT_EMAIL, STUDENT_TOKEN)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}