docs/examples/tutorial/wrappers/08_ensembles.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ensembles and Predictions Clipping\n",
    "The combination of predictions from several methods to one forecast often leads to great performance improvements. \n",
    "\n",
    "## Simple Ensembles\n",
    "The most common strategy just takes an average of all the forecast, which often leads to surprisingly good results, for more on this topic, see forecast combination chapter from [Forecasting: Principles and Practice](https://otexts.com/fpp2/combinations.html). hcrystalball implements `SimpleEnsemble` which provides a simple interface for putting together very diverse models. `SimpleEnsemble` model takes a list of any hcrystalball model wrapper instance(s) as base learners and aggregates their prediction using `ensemble_func`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('seaborn')\n",
    "plt.rcParams['figure.figsize'] = [12, 6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hcrystalball.utils import get_sales_data\n",
    "\n",
    "df = get_sales_data(n_dates=100, \n",
    "                    n_assortments=1, \n",
    "                    n_states=1, \n",
    "                    n_stores=1)\n",
    "X, y = pd.DataFrame(index=df.index), df['Sales']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hcrystalball.ensemble import SimpleEnsemble\n",
    "from hcrystalball.wrappers import ProphetWrapper\n",
    "from hcrystalball.wrappers import ExponentialSmoothingWrapper\n",
    "from hcrystalball.wrappers import get_sklearn_wrapper\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.ensemble import RandomForestRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prophet = ProphetWrapper(\n",
    "    extra_seasonalities = [{\n",
    "        'name':'bi-weekly',\n",
    "        'period': 14.,\n",
    "        'fourier_order': 5,\n",
    "        'prior_scale': 15.0,\n",
    "        'mode': None\n",
    "    }]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "simple_ensemble = SimpleEnsemble(\n",
    "    base_learners=[\n",
    "        prophet,\n",
    "        ExponentialSmoothingWrapper(),\n",
    "        get_sklearn_wrapper(RandomForestRegressor)\n",
    "    ],\n",
    "    ensemble_func = 'median')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(simple_ensemble.fit(X[:-10], y[:-10])\n",
    "                .predict(X[-10:])\n",
    "                .merge(y, left_index=True, right_index=True, how='outer')\n",
    "                .tail(50)\n",
    "                .plot()\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stacking Ensembles\n",
    "hcrystalball `StackingEnsemble` model is very similar to sklearn [StackingRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingRegressor.html), nice explanation of the concept could be also found [here](http://rasbt.github.io/mlxtend/user_guide/regressor/StackingCVRegressor/). Unfortunately, the sklearn version of the `StackingEnsemble` doesn't allow us to enrich the `meta_training_data` by other features than predictions of base learners. \n",
    "\n",
    "In the case of time-series forecasts, the additional features are the key to enabling the `meta_model` to successfully learn when each model performs best. It's often the case that some model performs better during specific days (i.e. weekends/holidays/spikes) and other is better during more stable periods. The goal is to combine them appropriately. \n",
    "\n",
    "To do that you can specify `weekdays_as_features` as `True`, which will enrich the `meta_training_data` with the day of the week features. Another way how to improve the performance of the stacking ensemble is to take into account that some models perform better on short horizons, and some on longer ones. To take this effect into account the `horizons_as_features` can be set to `True`, which creates an additional column per each horizon and enables the meta_model to learn this representation. Another important aspect of `StackingRegressor` is that to obtain `meta_model` which can generalize well - it needs to be fitted on out-of-sample predictions - `train_horizon` and `train_n_splits`. If you set `train_horizon`=5 and`train_n_splits`=4 then the training set for meta_model will be 20 observations. It's advisable to have \n",
    "`train_horizon` * `train_n_splits` = training set for `meta_model` as big as possible and have `train_horizon` which will match the horizon of the `StackingEnsemble` itself.\n",
    "\n",
    "### Predictions Clipping\n",
    "Meta model's predictions can be clipped to certain range given `clip_predictions_lower` and `clip_predictions_upper` parameters and/or we can in the same way also restrict `base_learners` (see/try commented lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hcrystalball.ensemble import StackingEnsemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacking_ensemble = StackingEnsemble(\n",
    "    base_learners=[\n",
    "        prophet,\n",
    "        ExponentialSmoothingWrapper(\n",
    "            # prediction bounds for ExponentialSmoothingWrapper base_learner predictions\n",
    "            # clip_predictions_lower=0. \n",
    "            # clip_predictions_upper=50_000. \n",
    "        ),\n",
    "        get_sklearn_wrapper(\n",
    "            RandomForestRegressor\n",
    "            # prediction bounds for RandomForestRegressor base_learner predictions\n",
    "            # clip_predictions_lower=0. \n",
    "            # clip_predictions_upper=50_000. \n",
    "        )\n",
    "    ],\n",
    "    train_horizon=10,\n",
    "    train_n_splits=3,\n",
    "    meta_model=LinearRegression(),\n",
    "    # prediction bounds for meta_model predictions \n",
    "    # clip_predictions_lower=0. \n",
    "    # clip_predictions_upper=50_000.\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(stacking_ensemble.fit(X[:-10], y[:-10])\n",
    "                  .predict(X[-10:])\n",
    "                  .merge(y, left_index=True, right_index=True, how='outer')\n",
    "                  .tail(50)\n",
    "                  .plot()\n",
    ");"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}