12_gradient_boosting_machines/05_trading_signals_with_lightgbm_and_catboost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Long-Short Strategy, Part 2: Trading signals with LightGBM and CatBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we'll start designing, implementing, and evaluating a trading strategy for US equities driven by daily return forecasts produced by gradient boosting models.\n",
    "\n",
    "As in the previous examples, we'll lay out a framework and build a specific example that you can adapt to run your own experiments. There are numerous aspects that you can vary, from the asset class and investment universe to more granular aspects like the features, holding period, or trading rules. See, for example, the **Alpha Factor Library** in the [Appendix](../24_alpha_factor_library) for numerous additional features.\n",
    "\n",
    "We'll keep the trading strategy simple and only use a single ML signal; a real-life application will likely use multiple signals from different sources, such as complementary ML models trained on different datasets or with different lookahead or lookback periods. It would also use sophisticated risk management, from simple stop-loss to value-at-risk analysis.\n",
    "\n",
    "**Six notebooks** cover our workflow sequence:\n",
    "\n",
    "1. [preparing_the_model_data](04_preparing_the_model_data.ipyny): we engineer a few simple features from the Quandl Wiki data \n",
    "2. `trading_signals_with_lightgbm_and_catboost`  (this noteboook): we tune hyperparameters for LightGBM and CatBoost to select a model, using 2015/16 as our validation period. \n",
    "3. [evaluate_trading_signals](06_evaluate_trading_signals.ipynb): we compare the cross-validation performance using various metrics to select the best model. \n",
    "4. [model_interpretation](07_model_interpretation.ipynb): we take a closer look at the drivers behind the best model's predictions.\n",
    "5. [making_out_of_sample_predictions](08_making_out_of_sample_predictions.ipynb): we generate predictions for our out-of-sample test period 2017.\n",
    "6. [backtesting_with_zipline](09_backtesting_with_zipline.ipynb): evaluate the historical performance of a long-short strategy based on our predictive signals using Zipline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll subset the dataset created in the preceding notebook through the end of 2016 to cross-validate several model configurations for various lookback and lookahead windows, as well as different roll-forward periods and hyperparameters. \n",
    "\n",
    "Our approach to model selection will be similar to the one we used in the previous chapter and uses the custom `MultipleTimeSeriesCV` introduced in [Chapter 7, Linear Models – From Risk Factors to Return Forecasts](../07_linear_models)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports & Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:53.361121Z",
     "start_time": "2020-06-21T03:15:53.359422Z"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:54.473652Z",
     "start_time": "2020-06-21T03:15:53.619170Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from pathlib import Path\n",
    "import sys, os\n",
    "from time import time\n",
    "from tqdm import tqdm\n",
    "\n",
    "from collections import defaultdict\n",
    "from itertools import product\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import lightgbm as lgb\n",
    "from catboost import Pool, CatBoostRegressor\n",
    "\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from scipy.stats import spearmanr\n",
    "\n",
    "from alphalens.tears import (create_summary_tear_sheet,\n",
    "                             create_full_tear_sheet)\n",
    "\n",
    "from alphalens.utils import get_clean_factor_and_forward_returns\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:54.477216Z",
     "start_time": "2020-06-21T03:15:54.474618Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sys.path.insert(1, os.path.join(sys.path[0], '..'))\n",
    "from utils import MultipleTimeSeriesCV, format_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:54.489618Z",
     "start_time": "2020-06-21T03:15:54.478409Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sns.set_style('whitegrid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:54.626839Z",
     "start_time": "2020-06-21T03:15:54.624975Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "YEAR = 252\n",
    "idx = pd.IndexSlice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We select the train and validation sets, and identify labels and features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:58.610627Z",
     "start_time": "2020-06-21T03:15:56.699840Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "MultiIndex: 1749266 entries, ('A', Timestamp('2010-01-04 00:00:00')) to ('ZION', Timestamp('2016-12-30 00:00:00'))\n",
      "Data columns (total 34 columns):\n",
      " #   Column           Non-Null Count    Dtype  \n",
      "---  ------           --------------    -----  \n",
      " 0   dollar_vol       1749266 non-null  float64\n",
      " 1   dollar_vol_rank  1749266 non-null  float64\n",
      " 2   rsi              1735336 non-null  float64\n",
      " 3   bb_high          1730361 non-null  float64\n",
      " 4   bb_low           1730359 non-null  float64\n",
      " 5   NATR             1735336 non-null  float64\n",
      " 6   ATR              1735336 non-null  float64\n",
      " 7   PPO              1724391 non-null  float64\n",
      " 8   MACD             1716431 non-null  float64\n",
      " 9   sector           1749266 non-null  int64  \n",
      " 10  r01              1748271 non-null  float64\n",
      " 11  r05              1744291 non-null  float64\n",
      " 12  r10              1739316 non-null  float64\n",
      " 13  r21              1728371 non-null  float64\n",
      " 14  r42              1707476 non-null  float64\n",
      " 15  r63              1686581 non-null  float64\n",
      " 16  r01dec           1748271 non-null  float64\n",
      " 17  r05dec           1744291 non-null  float64\n",
      " 18  r10dec           1739316 non-null  float64\n",
      " 19  r21dec           1728371 non-null  float64\n",
      " 20  r42dec           1707476 non-null  float64\n",
      " 21  r63dec           1686581 non-null  float64\n",
      " 22  r01q_sector      1748271 non-null  float64\n",
      " 23  r05q_sector      1744291 non-null  float64\n",
      " 24  r10q_sector      1739316 non-null  float64\n",
      " 25  r21q_sector      1728371 non-null  float64\n",
      " 26  r42q_sector      1707476 non-null  float64\n",
      " 27  r63q_sector      1686581 non-null  float64\n",
      " 28  r01_fwd          1749266 non-null  float64\n",
      " 29  r05_fwd          1749266 non-null  float64\n",
      " 30  r21_fwd          1749251 non-null  float64\n",
      " 31  year             1749266 non-null  int64  \n",
      " 32  month            1749266 non-null  int64  \n",
      " 33  weekday          1749266 non-null  int64  \n",
      "dtypes: float64(30), int64(4)\n",
      "memory usage: 461.2+ MB\n"
     ]
    }
   ],
   "source": [
    "data = (pd.read_hdf('data.h5', 'model_data')\n",
    "            .sort_index()\n",
    "            .loc[idx[:, :'2016'], :]) # train & validation period\n",
    "data.info(null_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:58.627071Z",
     "start_time": "2020-06-21T03:15:58.611792Z"
    }
   },
   "outputs": [],
   "source": [
    "labels = sorted(data.filter(like='_fwd').columns)\n",
    "features = data.columns.difference(labels).tolist() # features are columns not containing '_fwd'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Selection: Lookback, lookahead and roll-forward periods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:58.713463Z",
     "start_time": "2020-06-21T03:15:58.628189Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tickers = data.index.get_level_values('symbol').unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We may want to predict 1, 5 or 21-day returns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:58.716838Z",
     "start_time": "2020-06-21T03:15:58.714860Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "lookaheads = [1, 5, 21]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:58.726739Z",
     "start_time": "2020-06-21T03:15:58.718248Z"
    }
   },
   "outputs": [],
   "source": [
    "categoricals = ['year', 'month', 'sector', 'weekday']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We select 4.5 and one years as the length of our training periods; test periods are one and three months long. Since we are using two years (2015/16) for validation, a one-month test period implies 24 folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:58.735386Z",
     "start_time": "2020-06-21T03:15:58.728023Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train_lengths = [int(4.5 * 252), 252]\n",
    "test_lengths = [63, 21]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:58.743174Z",
     "start_time": "2020-06-21T03:15:58.737182Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "test_params = list(product(lookaheads, train_lengths, test_lengths))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:15:58.751241Z",
     "start_time": "2020-06-21T03:15:58.744605Z"
    }
   },
   "outputs": [],
   "source": [
    "results_path = Path('results', 'us_stocks')\n",
    "if not results_path.exists():\n",
    "    results_path.mkdir(parents=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline: Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We always want to know how much our (gradient boosting) is improving over a simpler baseline (if at all..)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:16:05.129548Z",
     "start_time": "2020-06-21T03:16:05.127693Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "lr = LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:37.679350Z",
     "start_time": "2020-06-21T03:16:05.270137Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12/12 [07:08<00:00, 35.72s/it]\n"
     ]
    }
   ],
   "source": [
    "lr_metrics = []\n",
    "\n",
    "# iterate over our three CV configuration parameters\n",
    "for lookahead, train_length, test_length in tqdm(test_params):\n",
    "    label = f'r{lookahead:02}_fwd'\n",
    "    df = pd.get_dummies(data.loc[:, features + [label]].dropna(), \n",
    "                        columns=categoricals, \n",
    "                        drop_first=True)\n",
    "    X, y = df.drop(label, axis=1), df[label]\n",
    "\n",
    "    n_splits = int(2 * YEAR / test_length)\n",
    "    cv = MultipleTimeSeriesCV(n_splits=n_splits,\n",
    "                              test_period_length=test_length,\n",
    "                              lookahead=lookahead,\n",
    "                              train_period_length=train_length)\n",
    "\n",
    "    ic, preds = [], []\n",
    "    for i, (train_idx, test_idx) in enumerate(cv.split(X=X)):\n",
    "        X_train, y_train = X.iloc[train_idx], y.iloc[train_idx]\n",
    "        X_test, y_test = X.iloc[test_idx], y.iloc[test_idx]\n",
    "        lr.fit(X_train, y_train)\n",
    "        y_pred = lr.predict(X_test)\n",
    "        preds.append(y_test.to_frame('y_true').assign(y_pred=y_pred))\n",
    "        ic.append(spearmanr(y_test, y_pred)[0])\n",
    "    preds = pd.concat(preds)\n",
    "    lr_metrics.append([lookahead, \n",
    "                       train_length, \n",
    "                       test_length,\n",
    "                       np.mean(ic),\n",
    "                       spearmanr(preds.y_true, preds.y_pred)[0]\n",
    "                      ])\n",
    "\n",
    "columns = ['lookahead', 'train_length', 'test_length', 'ic_by_day', 'ic']\n",
    "lr_metrics = pd.DataFrame(lr_metrics, columns=columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Information Coefficient - Distribution by Lookahead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:37.897475Z",
     "start_time": "2020-06-21T03:19:37.680445Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA+gAAAFgCAYAAAAo31N4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAps0lEQVR4nO3de3hldX3v8XcywEAcEDMg0AEEmfHroOINwVs9oEJDiuKl3hDPaGstVprWW296DmixD/VCJRwKKmDHC4K1WikN4WILVq06M4AXZvyaFEEGUJyEy9DAMDA5f+wd3cZJsjOTtddK9vv1PHmy11q/tX/fPcnktz/7ty4d4+PjSJIkSZKkcnWWXYAkSZIkSTKgS5IkSZJUCQZ0SZIkSZIqwIAuSZIkSVIFGNAlSZIkSaoAA7okSZIkSRVgQJc0pyLikIgYj4hdyq5FkiQVJyKui4i31h+/OSK+UXZN0nznG2hJvyEibgXempnX1pcPAM4EeoElwB3AZcCHM/N/CqzjDOB9wEP1VXcBVwMfysy7iupXkqQi7Oz4GhGLgTOANwL7AhuBTwIfzczxFrwESQVzBl3StCKiG/gvYA/geZm5J3AcsDdwWAtKuKzeZzfwSmB/YF39TY0kSfPSDo6v/wS8hFqg3xN4E/A24JwC6nMiTyqB//EkzeRdwGbglMzcBpCZtwN/OsN+v1+fAe+g9sn+xyJif+AW4KDMHAGIiGcDg8BvZebWqZ6svu3miHgdcAPwbuA9EfE44LPA0dT+pn0TODUzN0bEa4C/zMxnTzxPRLwb+O3MfMUs/x0kSZpLsxpfI+IlwPHAino7gG9HxCnAtyKiHzgSeE9mHtmw3zuBYzPz5fUZ+A8BrwUWA18B3pmZD0bEMcDngHOBdwLXREQfU4yxc/jvIKmBM+iSZvJS4MsTbx5m4VhgBbU3E38ZES/NzJ8B11F7YzDhFODS6cJ5o8x8FPgq8Nv1VZ3Ap4EnAAcDDwL/r77tcuDQiFg5qb/PzvK1SJI012Y7vh4HfKchnAOQmd+hdqj7S6iNexERKxqanAxcUn/8d8CTgGcAy4FlwP9taLs/tSPWnkBtZn66MVZSAZxBlzSTpdTO/Z6tD9TPn/tBRHwaeANwLbAa6APOj4hF9fUvn+Vz30ntDQT1mfh/ntgQER8C/qO+bUtEXEYtlL8vIp4CHAJcsQOvR5KkuTTb8XWfadrfBeyTmWMR8VVqY+sH60H9ycDlEdEB/CFwRGaOAkTE31IL739Vf55twOmZuaW+/CBTjLGSimFAlzSTEWBHzvdu/IT/NuBp9cdfBS6IiCdS+xT/vsz87iyfexkw8eaiC/h7oAd4XH37nhGxqD7bvhr4QkS8n9q5el9seOMhSVJZZju+bqJ2ZNr2HFDfDrXA/THgg9Rmz/+lHtwfD3RRu47LxH4dwKKG5/lFZk5cmLWZMVbSHPMQd0kzuRZ4ZUTM9u/FQQ2PD6Y260194P8itSvQvolZHm5er+NlwH/WV70bCODozNwLeFF9fUe9v28DD1M7JP7k2fYnSVJBZju+XgscHRGN4ysRcRS1Mfff66uuBvaJiGdQm0mfOLx9E7UZ8adk5t71r8dm5pKGp5t8Jfhpx1hJc8+ALmkmZwN7Aasj4gkAEbEsIs6OiCOm2e//RERX/bDyt1C7bcyEzwBvpnZo++eaKSIidq2fS/4FaufInV3ftCe1Nxz31q+Ie/p2dv8MtXPmHslM79EqSaqCWY2v9VuzfQ3454h4SkQsiojnAp8Hzs/MoXq7R4AvAR+hdjrYNfX124BPAX9fn02f6O93pqmxmTFW0hwyoEuaVv08tecDW4HvRMRmam8Q7gOGp9n1+vr2r1G7ivvVDc/5TWrnud2QmbfOUMLrIuIB4F5qF78ZAZ6dmXfWt3+c2i1qNgHfpnZF+Mk+CzwVZ88lSRWxg+Prq6mdAz4IPEDtQ+6LgD+Z1O4Saheh+6d6YJ/wF/Xn/nZE3E9tVj6Y2seZeYyVNIc6xscnH8kiScWLiH8HLsnMC1vQ1x7A3cCzJmYYJEmSpKrxInGSWi4ingM8CzipRV2+HVhjOJckSVKVGdAltVRErAZeAfxpZm5uQX+3UruYzSuK7kuSJEnaGR7iLkmSJElSBZQ6gx4RPcA51O6/eGFmnjVp+xupXcwCahfCeHtmfq+ZfSVJkiRJmk9Km0GPiEXAj4HjgI3AGuANmbm+oc3zgQ2ZeU9EnACckZlHN7Pv9tx0003jixcvLuYFSZK0wIyNjW169rOfvW8zbR1jJUlq3lRjbJkz6EcBw5l5C0BEXErtglG/DNmZ+a2G9t8GDmx23+1ZvHgxK1eunLMXIEnSQrZu3brbmm3rGCtJUvOmGmPLDOjLgNsbljcCR0/T/g+AK3dwXwC2bNnChg0bZlmmJEmaiWOsJEk7r8yA3rGddds93j4ijqUW0F84230b+em+JEnNW7duXdNtHWMlSWreVGNsmQF9I3BQw/KBwJ2TG0XEEcCFwAmZOTKbfSVJkiRJmi/KDOhrgBURcShwB/B64OTGBhFxMPBl4E2Z+ePZ7CtJkiRJ0nzSWVbHmfkIcBpwFbAB+GJm3hwRp0bEqfVm/xdYCvxDRNwUEWun27flL0KSJEmSpDlS6n3QM3MAGJi07oKGx28F3trsvpIkSZIkzVelzaBLkiRJkqRfMaBLkiRJklQBBnRJkiRJkirAgC5JkiRJUgUY0CVJkiRJqoBSr+IuqRoGBwcZGCjnpgijo6MAdHd3l9J/b28vPT09pfQtSZIkNTKgSyrVyMgIUF5AlyRJkqrCgC6Jnp6e0maR+/r6AOjv7y+lf0mSJKkqPAddkiRJkqQKMKBLkiRJklQBBnRJkiRJkirAgC5JkiRJUgUY0CVJkiRJqgADuiRJkiRJFWBAlyRJkiSpAgzokiRJkiRVgAFdkiRJkqQKMKBLkiRJklQBBnRJkiRJkirAgC5JkiRJUgUY0CVJkiRJqgADuiRJkiRJFbBL2QVIkiRJknbc4OAgAwMDpfQ9OjoKQHd3d8v77u3tpaenp+X9FsmALkmSJEnaISMjI0A5AX0hMqBLkiRJ0jzW09NT2kxyX18fAP39/aX0v9B4DrokSZIkSRVgQJckSZIkqQIM6JIkSZIkVYABXZIkSZKkCjCgS5IkSZJUAV7FXaqI/v5+hoeHyy6j5YaGhoBfXQG0nSxfvrwtX7ckSZK2z4AuVcTw8DA//uENHLzk0bJLaam9xjsAeOjWNSVX0lo/fWBR2SVIkiSpYgzoUoUcvORR3n/kA2WXoRY4c+2SskuQJElSxXgOuiRJkiRJFWBAlyRJkiSpAgzokiRJkiRVgAFdkiRJkqQKMKBLkiRJklQBBnRJkiRJkirAgC5JkiRJUgWUeh/0iOgBzgEWARdm5lmTtj8Z+DTwLOB9mfnRhm23ApuBR4FHMvPIVtUtSZIkSdJcKy2gR8Qi4DzgOGAjsCYiLs/M9Q3NRoE+4BVTPM2xmbmp0EIlSZIkSWqBMg9xPwoYzsxbMvNh4FLgpMYGmXl3Zq4BtpZRoCRJkiRJrVLmIe7LgNsbljcCR89i/3Hg6ogYBz6RmZ+caYctW7awYcOG2VUptcjY2JgXhWgzY2Nj/k3SguEYK0ntaWxsDMAxYI6UGdA7trNufBb7vyAz74yIxwPXRMSPMvPr0+2wePFiVq5cOasipVbp6uriobKLUEt1dXX5N0mVtm7duqbbOsZKUnvq6uoCcAyYpanG2DIn7DYCBzUsHwjc2ezOmXln/fvdwFeoHTIvSZIkSdK8VGZAXwOsiIhDI2I34PXA5c3sGBGPiYg9Jx4DxwM/LKxSSZIkSZIKVtoh7pn5SEScBlxF7TZrF2fmzRFxan37BRGxP7AW2AvYFhF/BhwO7AN8JSKg9houyczBEl6GJEmSJElzotT7oGfmADAwad0FDY9/Ru3Q98nuB55ebHWSpLk0ODjIwMDAzA0LMDo6CkB3d3cp/ff29tLT01NK35Ikaf4oNaBLktQKIyMjQHkBXZIkqRkGdElSS/T09JQ2i9zX1wdAf39/Kf1LkiQ1w9suS5IkSZJUAQZ0SZIkSZIqwIAuSZIkSVIFGNAlSZIkSaoAA7okSZIkSRVgQJckSZIkqQIM6JIkSZIkVYABXZIkSZKkCjCgS5IkSZJUAQZ0SZIkSZIqYJeyC5AktU5/fz/Dw8Nll9FyQ0NDAPT19ZVcSWstX7687V6zJEnzmQFdktrI8PAwN958I+xddiUtVj9e7MY7biy3jla6t+wCJEnSbBnQJand7A3bjtlWdhUqWOd1nsUmSdJ84+gtSZIkSVIFGNAlSZIkSaoAA7okSZIkSRVgQJckSZIkqQIM6JIkSZIkVYABXZIkSZKkCjCgS5IkSZJUAQZ0SZIkSZIqwIAuSZIkSVIFGNAlSZIkSaqAXcouQJLUOqOjo3AvdF7n57ML3r0wusdo2VVIkqRZ8B2aJEmSJEkV4Ay6JLWR7u5ubnvwNrYds63sUlSwzus66e7uLrsMSZI0CwZ0qSJGR0f5xeZFnLl2SdmlqAVu27yIfUc9/FiSpIWiv7+f4eHhsstouaGhIQD6+vpKrqS1li9fXshrNqBLkiRJ0k4aHh7mxptvhL3LrqTF6idN33jHjeXW0Ur3FvfUBnSpIrq7u+m6/795/5EPlF2KWuDMtUvY3cOPJUlaWPbG08jaQJEX2/UicZIkSZIkVYABXZIkSZKkCpgxoEfEnzazTpIkSZIk7bhmZtBXbWfdm+e4DkmSJEmS2tqUF4mLiDcAJwOHRsTlDZv2BEaKLkySJEmSpHYy3VXcvwXcBewDfKxh/Wbg+0UWJUmSJElSu5kyoGfmbcBtwPNaV44kSZIkSe1pxvugR8SrgL8DHg901L/GM3OvgmuTJEmSJKltzBjQgQ8DL8vMDUUXI0mSJElSu2rmKu4/N5xLkiRJklSsZmbQ10bEZcC/AFsmVmbml3e284joAc4BFgEXZuZZk7Y/Gfg08CzgfZn50Wb3lSRJkiRpPmlmBn0vYAw4HnhZ/evEne04IhYB5wEnAIcDb4iIwyc1GwX6gI/uwL6SJEmSJM0bM86gZ+ZbCur7KGA4M28BiIhLgZOA9Q193w3cHRG/O9t9JUlTuBc6r2vm89kF5KH6991LraK17gWWlV2EJEmajWau4v4k4Hxgv8x8akQcAbw8M8/cyb6XAbc3LG8Eji5y3y1btrBhg6fTq5rGxsaaOqRFC8fY2FjL/yZ1d3cTh0ZL+6yC22+vDRkHPe6gkitpocfVft6t+h1zjJXU7sbGxsouQS1U1Pu4Zs5B/xTwXuATAJn5/Yi4BNjZgN6xnXXjRe67ePFiVq5c2WQXUmt1dXX9cpJP7aGrq6vlf5POOOOMlvZXFX19fQD09/eXXMn8sm7duqbbOsZKanddXV1wT9lVqFV29n3cVGNsMxN2XZn53UnrHtnhSn5lI9A4lXEgcGcL9pUkSZIkqXKamUHfFBGHUZ+hjojfA+6ag77XACsi4lDgDuD1wMkt2FeSJEmSpMppJqC/A/gk8OSIuAP4CXDKznacmY9ExGnAVdRulXZxZt4cEafWt18QEfsDa6ldSX5bRPwZcHhm3r+9fXe2JkmSJEmSytLMVdxvAV4aEY8BOjNz81x1npkDwMCkdRc0PP4ZtcPXm9pXkiRJkqT5asqAHhGnZObnIuJdk9YDkJlnF1ybJEmSJEltY7oZ9MfUv+/ZikIkSZIkSWpnUwb0zJy4rdoHWleOJEmSJEntacbbrEXE6ojYu2H5cRFxcaFVSZIkSZLUZpq5D/oRmXnvxEJm3gM8s7CKJEmSJElqQ80E9M6IeNzEQkR009zt2SRJkiRJUpOaCdofA74VEV+qL78G+FBxJUmSJEmS1H5mnEHPzM8ArwZ+DtwNvCozP1t0YZIkSZIktZMpA3pE7FX/3g38DLgE+Dzws/o6SZIkSZI0R6Y7xP0S4ERgHTDesL6jvvzEAuuSJEmSJKmtTBfQz6p/X5mZD7WiGEmSJEmS2tV056CfU//+rVYUIkmSJElSO5tuBn1rRHwaODAi+idvzMy+4sqSJEmSJKm9TBfQTwReCryY2nnokiRJkiSpINMF9Pdm5l9ExMGZubplFUmSJEmS1IamOwe9NyJ2BV7fqmIkSZIkSWpX082gDwKbgMdExP386vZqHcB4Zu7VgvokSZIkSWoLUwb0zHwv8N6I+GpmntTCmiRJC9Dg4CADAwOl9D00NARAX1851zft7e2lp6enlL4lSdL8Md0MOgCZeVJEPAFYkZnXRsQewC6Zubn48qT28tMHFnHm2iVll9FS9z3cAcBjdxsvuZLW+ukDi3hS2UW0kaVLl5ZdgiRJ0oxmDOgR8YfA24Bu4DDgQOAC4CXFlia1l+XLl5ddQilur89s7nfIipIraa0n0X4/856eHmeRJUmSpjFjQAfeARwFfAcgM4ci4vGFVqVfKvOQ0NHRUQC6u7tL6b/dDgkt69Dbsk287v7+/pIrkSRJkso13VXcJ2zJzIcnFiJiF2oXi9MCNzIywsjISNllSJIkSVJbaGYG/fqI+Gtgj4g4Dvhj4F+LLUsTyjwk1JlNSZIkSWqdZmbQ/xL4BfAD4I+AAeD9RRYlSZIkSVK7aeYq7tsiYjXwX79alR7iLi0g3v6qfa51IEmSpOqacQY9Io4BhoDzgH8AfhwRLyq4LkltYunSpd4CS5IkSaK5c9A/BhyfmQkQEU8CvgA8u8jCJLWOt7+SJEmSytfMOei7ToRzgMz8MbBrcSVJkiRJktR+mplBXxsRFwGfrS+fAqwrriRJkiRJktpPMwH97cA7gD6gA7geOL/IoiRJkiRJajdTBvSI2BfYNzPXA2fXv4iIpwJ7Ubv1miRJkiRJmgPTnYN+LrDvdtYvA84pphxJkiRJktrTdAH9aZl5/eSVmXkVcERxJUmSJEmS1H6mC+jTXandq7hLkiRJkjSHprtI3FBE9GbmQOPKiDgBuKXYsqqlv7+f4eHhsstouaGhIQD6+vpKrqT1li9f3pavW5IkSVJ5pgvo7wSuiIjX8qvbqh0JPA84sejCqmR4eJgbf7CebV3dZZfSUh2P1n491v33z0qupLU6x0bLLkGSJElSG5oyoGfmjyPiacDJwFPrq68H/igzH2pFcVWyraubhw5vq88l2tbu668ouwRJkiRJbWja+6Bn5hbg0y2qRZIkSVpQBgcHGRgYmLlhAUZHa0cFdne3/ijQ3t5eenp6Wt6vNN9NG9AlSZIkzU8jIyNAOQFd0o4xoEuSJEkF6enpKW0meeKCt/39/aX0L2n2Sg3oEdEDnAMsAi7MzLMmbe+ob+8FxoA3Z+YN9W23ApuBR4FHMvPIFpYuSZIkSdKcmjGgR8QLgDOAJ9TbdwDjmfnEnek4IhYB5wHHARuBNRFxeWaub2h2ArCi/nU0cH79+4RjM3PTztQhSZIkSVIVdDbR5iLgbOCFwHOo3WrtOXPQ91HAcGbekpkPA5cCJ01qcxLwmcwcz8xvA3tHxAFz0LckSZIkSZXSzCHu92XmlQX0vQy4vWF5I78+Oz5Vm2XAXcA4cHVEjAOfyMxPFlCjJEmSJEkt0UxA/4+I+AjwZWDLxMqJc8F3Qsd21o3Pos0LMvPOiHg8cE1E/Cgzvz5dh1u2bGHDhg2zLnRsbGzW+2h+Gxsb26HfFUlqVzs6xkoqzsR7WP9vtoaZob0UlReaCegTs9qNF2EbB168k31vBA5qWD4QuLPZNpk58f3uiPgKtUPmpw3oixcvZuXKlbMutKurC7h/1vtp/urq6tqh3xVJWkjWrVvXdNsdHWMlFaf2Hhb/b7ZIV1cX3FN2FWqVnc0LU42xMwb0zDx2h3ud3hpgRUQcCtwBvB44eVKby4HTIuJSah8U3JeZd0XEY4DOzNxcf3w88MGC6pQkSZIkqXDNXMX9scDpwIvqq64HPpiZ9+1Mx5n5SEScBlxF7TZrF2fmzRFxan37BcAAtVusDVO7zdpb6rvvB3wlIiZewyWZObgz9UxndHSUzrERdl9/RVFdqEI6x0YYHd2t7DIkSZIktZlmDnG/GPgh8Nr68puATwOv2tnOM3OAWghvXHdBw+Nx4B3b2e8W4Ok7278kSZIWvv7+foaHh8suo+WGhoYA6OvrK7mS1lq+fHnbvWYtHM0E9MMy89UNyx+IiJsKqqeSuru7+ck9D/PQ4SeWXYpaYPf1V9Dd3V12GZIkaY4MDw/z4x/ewMFLHi27lJbaa7x2veWHbl1TciWt89MHFpVdgrRTmgnoD0bECzPzGwAR8QLgwWLLkiRJkubOwUse5f1HPlB2GSrYmWuXlF2CtFOaCehvB1bXz0XvAEaBNxdZlCRJkiTNJ6Ojo3AvdF7XWXYpKtq9MLrHaCFP3cxV3G8Cnh4Re9WXvd+YJEmSJElzbMqAHhGnZObnIuJdk9YDkJlnF1ybJEmSJM0L3d3d3PbgbWw7ZlvZpahgndd1FnbNqulm0B9T/77ndraNF1CLJEmSJElta8qAnpmfqD+8NjO/2bitfqE4SZIkSZI0R5q5gsG5Ta6TJEmSJEk7aLpz0J8HPB/Yd9J56HsB3mBQkiRJkqQ5NN056LsBS+ptGs9Dvx/4vSKLkiRJkiSp3Ux3Dvr1wPUR8Y+ZeVsLa6qkzrFRdl9/RdlltFTH1gcBGN91j5Iraa3OsVFg/7LLkCRJktRmZrwPOjAWER8BngLsPrEyM19cWFUVs3z58rJLKMXQ0BAAKw5rt7C6f9v+zCVJkiSVp5mA/nngMuBE4FRgFfCLIouqmr6+vrJLKMXE6+7v7y+5EkmSJEla+Jq5ivvSzLwI2JqZ12fm7wPPLbguSZIkSZLaSjMz6Fvr3++KiN8F7gQOLK4kSZIkSZLaTzMB/cyIeCzwbmr3P98LeGehVUmSJEmS1GZmDOiZOXHp8vuAY4stR5MNDg4yMDBQSt8TF4kr6xz83t5eenp6SulbkiRJklptxoAeEYcCfwIc0tg+M19eXFmqgqVLl5ZdgiRJkiS1jWYOcf8X4CLgX4FthVaj39DT0+MssiRJkiS1gWYC+kOZ6X22JEmS5simTZv4wAc+wBlnnOERa5KkX2omoJ8TEacDVwNbJlZm5g2FVSVJkrSArV69mu9///usXr2ad73rXWWXI0mqiGYC+tOANwEv5leHuI/XlyVJkjQLmzZt4sorr2R8fJwrr7ySVatWOYsuSQKaC+ivBJ6YmQ8XXYwkSdJCt3r1asbHxwHYtm2bs+iSpF/qbKLN94C9C65DkiSpLVxzzTVs3boVgK1bt3L11VeXXJEkqSqamUHfD/hRRKzh189B9zZrkiRJs3TccccxMDDA1q1b2XXXXTn++OPLLkmSVBHNBPTTC69CkiSpTaxatYorr7wSgM7OTlatWlVyRZKkqpg2oEdEJ3BeZj61RfVIkiQtaPvssw8nnHACl19+OSeccIIXiJMk/dK056Bn5jbgexFxcIvqkSRJWvBWrVrFEUcc4ey5JOnXNHOI+wHAzRHxXeB/JlZ6DrokSdKO2WeffTj33HPLLkOSVDHNBPQPFF6FJEmSJEltbsbbrGXm9cCPgD3rXxvq6yRJkiRJ0hyZcQY9Il4LfAS4DugAzo2I92bmlwquTZIkSdppo6Oj/GLzIs5cu6TsUlSw2zYvYt/R0bLLkHZYM4e4vw94TmbeDRAR+wLXAgZ0SZIkSZLmSDMBvXMinNeN0MSh8ZIkSVIVdHd303X/f/P+Ix8ouxQV7My1S9i9u7vsMqQd1kxAH4yIq4Av1JdfBwwUV5IkSZIkSe1nypnwiFgMkJnvBT4BHAE8HfhkZv5Fa8qTJEmSJKk9TDeD/l/AsyLis5n5JuDLLapJkiRJkqS2M11A3y0iVgHPj4hXTd6YmQZ2SZIkSZLmyHQB/VTgjcDewMsmbRvHGXVJkiRJkubMlAE9M78BfCMi1mbmRS2sSZIkSZKktjPjVdwz86KIeD5wSGP7zPxMgXVJkiRJktRWZgzoEfFZ4DDgJuDR+upxYKcDekT0AOcAi4ALM/OsSds76tt7gTHgzZl5QzP7SpKk6uvv72d4eLiUvkdHRxkZGSml7zItXbqU7pLuE718+XL6+vpK6VuS5oNm7oN+JHB4Zo7PZccRsQg4DzgO2AisiYjLM3N9Q7MTgBX1r6OB84Gjm9xXkiRV3PDwMDf+YD3bulofGDu2PkjH1odb3m/ZNt99Hz+5p/Wvu3NstOV9StJ800xA/yGwP3DXHPd9FDCcmbcARMSlwElAY8g+CfhM/cOBb0fE3hFxALXD7WfaV5IkzQPburp56PATyy5DBdt9/RVllyBJlddMQN8HWB8R3wW2TKzMzJfvZN/LgNsbljdSmyWfqc2yJvf9DVu2bGHDhg07VKwkSZrajo6xY2NjBVSjqhobGyvlvdjY2BidLe9VZSnz90zto6jfs2YC+hlz3mtNx3bWTT6Mfqo2zez7GxYvXszKlSubKE2SJK1bt67ptjs6xnZ1dQH3z3o/zU9dXV2lvBfr6urioZb3qrKU+XvGPS3vViXZ2d+zqcbYZq7ifv0O9zq9jcBBDcsHAnc22Wa3JvaVJEmSJGnemDKgR8Rmtj8r3QGMZ+ZeO9n3GmBFRBwK3AG8Hjh5UpvLgdPq55gfDdyXmXdFxC+a2FeSJEmSpHljyoCemXsW2XFmPhIRpwFXUbtV2sWZeXNEnFrffgEwQO0Wa8PUbrP2lun2LbJeSZIkSZKK1Mw56IXJzAFqIbxx3QUNj8eBdzS7ryRJkiRJ81WpAV2SJLW30dFROsdGvAVXG+gcG2F0dLeyy5CkSvOOE5IkSZIkVYAz6JIkqTTd3d385J6HeejwE8suRQXbff0VdHd3l12GJFWaM+iSJEmSJFWAM+iSJKlUnWOjbXcOesfWBwEY33WPkitpnc6xUWD/ssuQpEozoEuSpNIsX7687BJKMTQ0BMCKw9opsO7ftj9vSWqWAV2SJJWmr6+v7BJKMfG6+/v7S65EklQlnoMuSZIkSVIFGNAlSZIkSaoAD3GXJEltaXBwkIGBgVL6njgHvYxD/Ht7e+np6Wl5v5KkmRnQJUmSWmzp0qVllyBJqiADuiRJaks9PT3OJLeRnz6wiDPXLim7jJa67+EOAB6723jJlbTOTx9YxJPKLkLaCQZ0SZIkLWjtenu32+unUux3yIqSK2mdJ9G+P28tDAZ0SZIkLWjezs/b+UnzhVdxlyRJkiSpAgzokiRJkiRVgAFdkiRJkqQK8Bx0SZIkqSCDg4MMDAyU0vdQ/SJxZZyD39vb610SpB1gQJckSZIWoKVLl5ZdgqRZMqBLkiRJBenp6XEmWVLTPAddkiRJkqQKMKBLkiRJklQBHuIuSZIkSXPhXui8rs3mQB+qf9+91Cpa615gWTFPbUCXJEmSpJ20fPnysksoxcTdAlYsW1FyJS20rLiftwFdkiRJknZSGbezq4KJ193f319yJQtDmx1/IUmSJElSNRnQJUmSJEmqAAO6JEmSJEkVYECXJEmSJKkCDOiSJEmSJFWAAV2SJEmSpAowoEuSJEmSVAEGdEmSJEmSKsCALkmSJElSBRjQJUmSJEmqAAO6JEmSJEkVYECXJEmSJKkCDOiSJEmSJFWAAV2SJEmSpAowoEuSJEmSVAEGdEmSJEmSKmCXMjqNiG7gMuAQ4FbgtZl5z3ba9QDnAIuACzPzrPr6M4A/BH5Rb/rXmTlQeOGSJEmSJBWkrBn0vwS+lpkrgK/Vl39NRCwCzgNOAA4H3hARhzc0+fvMfEb9y3AuSZIkSZrXygroJwGr649XA6/YTpujgOHMvCUzHwYure8nSZIkSdKCU8oh7sB+mXkXQGbeFRGP306bZcDtDcsbgaMblk+LiP8NrAXevb1D5CfbsmULGzZs2ImyJUnS9jjGSlJ7GhsbA3AMmCOFBfSIuBbYfzub3tfkU3RsZ914/fv5wN/Ul/8G+Bjw+zM94eLFi1m5cmWT3UuS1N7WrVvXdFvHWElqT11dXQCOAbM01RhbWEDPzJdOtS0ifh4RB9Rnzw8A7t5Os43AQQ3LBwJ31p/75w3P9SngirmpWpIkSZKkcpR1DvrlwKr641XAV7fTZg2wIiIOjYjdgNfX96Me6ie8EvhhgbVKkiRJklS4ss5BPwv4YkT8AfBT4DUAEfFb1G6n1puZj0TEacBV1G6zdnFm3lzf/8MR8Qxqh7jfCvxRi+uXJEmSJGlOlRLQM3MEeMl21t8J9DYsDwC/cQu1zHxToQVKkiRJktRiZR3iLkmSJEmSGhjQJUmSJEmqAAO6JEmSJEkVYECXJEmSJKkCDOiSJEmSJFWAAV2SJEmSpAowoEuSJEmSVAEGdEmSJEmSKsCALkmSJElSBRjQJUmSJEmqAAO6JEmSJEkVYECXJEmSJKkCDOiSJEmSJFWAAV2SJEmSpAowoEuSJEmSVAEGdEmSJEmSKsCALkmSJElSBRjQJUmSJEmqAAO6JEmSJEkVYECXJEmSJKkCDOiSJEmSJFWAAV2SJEmSpArYpewCJEmSJEk7bnBwkIGBgVL6HhoaAqCvr6/lfff29tLT09PyfotkQJckSZIk7ZClS5eWXcKCYkCXJEmSpHmsp6dnwc0ktyvPQZckSZIkqQIM6JIkSZIkVYABXZIkSZKkCjCgS5IkSZJUAQZ0SZIkSZIqwIAuSZIkSVIFGNAlSZIkSaoAA7okSZIkSRVgQJckSZIkqQIM6JIkSZIkVYABXZIkSZKkCtil7AJaaWxsbNO6detuK7sOSZLmiSc029AxVpKkWdnuGNsxPj7e6kIkSZIkSdIkHuIuSZIkSVIFGNAlSZIkSaoAA7okSZIkSRVgQJckSZIkqQIM6JIkSZIkVYABXZIkSZKkCmir+6CreRFxMXAicHdmPrXserRwRcStwGbgUeCRzDyy3Io030XEQcBngP2BbcAnM/OciHgNcAawEjgqM9eWV6XamWOsWsHxVUVwjC2eM+iayj8CPWUXobZxbGY+wzcPmiOPAO/OzJXAc4F3RMThwA+BVwFfL7M4CcdYtY7jq+aaY2zBDOjarsz8OjBadh2SNFuZeVdm3lB/vBnYACzLzA2ZmeVWJznGSpq/HGOLZ0CXVLZx4OqIWBcRbyu7GC0sEXEI8EzgOyWXIkmt5viqQjnGFsOALqlsL8jMZwEnUDtM6kVlF6SFISKWAP8M/Flm3l92PZLUYo6vKoxjbHEM6JJKlZl31r/fDXwFOKrcirQQRMSu1N44fD4zv1x2PZLUao6vKopjbLEM6JJKExGPiYg9Jx4Dx1O7yIi0wyKiA7gI2JCZZ5ddjyS1muOriuIYW7yO8fHxsmtQBUXEF4BjgH2AnwOnZ+ZFpRalBScinkjtU32o3fbxksz8UIklaQGIiBcC/wn8gNotYAD+GlgMnAvsC9wL3JSZv1NGjWpvjrEqmuOriuIYWzwDuiRJkiRJFeAh7pIkSZIkVYABXZIkSZKkCjCgS5IkSZJUAQZ0SZIkSZIqwIAuSZIkSVIFGNAlTSsiHtjB/Y6JiCtm0f6MiHjPjvQ1l3VIktQqjrGSJjOgS5IkSZJUAbuUXYCk+SEiOoAPAycA48CZmXnZVOsn7fsc4JPAq4GnAO8HdgNGgDdm5s/rTQ+PiOuAg4GPZ2Z/ff9TgL76Pt8B/jgzH42I84HnAHsAX8rM0+vte4CPA5uAG+b+X0OSpLnjGCtpgjPokpr1KuAZwNOBlwIfiYgDplkPQEQ8H7gAOCkzbwG+ATw3M58JXAr8eUMfTwZ+BzgKOD0ido2IlcDrgBdk5jOAR4E31tu/LzOPBI4A/ldEHBERuwOfAl4G/Daw/xz/O0iSNNccYyUBzqBLat4LgS9k5qPAzyPiemqfrE+1/n5gJbVP9Y/PzDvrz3MgcFn9DcZuwE8a+vi3zNwCbImIu4H9gJcAzwbWRATUPsm/u97+tRHxNmp/yw4ADqf2weNPMnMIICI+B7xtzv81JEmaO46xkgADuqTmdcxyPcBdwO7AM4GJNw/nAmdn5uURcQxwRkP7LQ2PH6X2N6oDWJ2Zf9X4xBFxKPAe4DmZeU9E/GO9L6gdBihJ0nzhGCsJ8BB3Sc37OvC6iFgUEfsCLwK+O816gHuB3wX+tv5GAeCxwB31x6ua6PdrwO9FxOMBIqI7Ip4A7AX8D3BfROxH7fw8gB8Bh0bEYfXlN+zIi5UkqYUcYyUBBnRJzfsK8H3ge8C/A3+emT+bZj0A9YvTvAw4LyKOpvZp/j9FxH9Su8DMtDJzPbUL3lwdEd8HrgEOyMzvATcCNwMXA9+st3+I2uF2/xYR3wBu2/mXLklSoRxjJQHQMT7uUSqSJEmSJJXNGXRJkiRJkirAgC5JkiRJUgUY0CVJkiRJqgADuiRJkiRJFWBAlyRJkiSpAgzokiRJkiRVgAFdkiRJkqQK+P+yPjwEkk6uqwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1008x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axes = plt.subplots(ncols=2, figsize=(14,5), sharey=True)\n",
    "\n",
    "# plot average of daily IC values\n",
    "sns.boxplot(x='lookahead', y='ic_by_day',data=lr_metrics, ax=axes[0])\n",
    "axes[0].set_title('IC by Day')\n",
    "\n",
    "# plot IC across all predictions\n",
    "sns.boxplot(x='lookahead', y='ic',data=lr_metrics, ax=axes[1])\n",
    "axes[1].set_title('IC Overall')\n",
    "axes[0].set_ylabel('Information Coefficient')\n",
    "axes[1].set_ylabel('')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Best Train/Test Period Lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For one- and five-day return forecasts, shorter train- and test-length yield better results in terms of daily avg IC:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:37.911949Z",
     "start_time": "2020-06-21T03:19:37.898655Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lookahead</th>\n",
       "      <th>train_length</th>\n",
       "      <th>test_length</th>\n",
       "      <th>ic_by_day</th>\n",
       "      <th>ic</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>252</td>\n",
       "      <td>21</td>\n",
       "      <td>0.072379</td>\n",
       "      <td>-0.008873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1134</td>\n",
       "      <td>21</td>\n",
       "      <td>0.054177</td>\n",
       "      <td>-0.004539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>252</td>\n",
       "      <td>63</td>\n",
       "      <td>0.036154</td>\n",
       "      <td>-0.016102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>5</td>\n",
       "      <td>252</td>\n",
       "      <td>21</td>\n",
       "      <td>0.188063</td>\n",
       "      <td>-0.023761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>1134</td>\n",
       "      <td>21</td>\n",
       "      <td>0.154049</td>\n",
       "      <td>-0.016191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>1134</td>\n",
       "      <td>63</td>\n",
       "      <td>0.076586</td>\n",
       "      <td>0.055433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>21</td>\n",
       "      <td>1134</td>\n",
       "      <td>21</td>\n",
       "      <td>0.144815</td>\n",
       "      <td>-0.012465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>21</td>\n",
       "      <td>252</td>\n",
       "      <td>21</td>\n",
       "      <td>0.127005</td>\n",
       "      <td>-0.078900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>21</td>\n",
       "      <td>1134</td>\n",
       "      <td>63</td>\n",
       "      <td>0.108506</td>\n",
       "      <td>0.096060</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    lookahead  train_length  test_length  ic_by_day        ic\n",
       "3           1           252           21   0.072379 -0.008873\n",
       "1           1          1134           21   0.054177 -0.004539\n",
       "2           1           252           63   0.036154 -0.016102\n",
       "7           5           252           21   0.188063 -0.023761\n",
       "5           5          1134           21   0.154049 -0.016191\n",
       "4           5          1134           63   0.076586  0.055433\n",
       "9          21          1134           21   0.144815 -0.012465\n",
       "11         21           252           21   0.127005 -0.078900\n",
       "8          21          1134           63   0.108506  0.096060"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(lr_metrics.groupby('lookahead', group_keys=False)\n",
    " .apply(lambda x: x.nlargest(3, 'ic_by_day')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:37.920850Z",
     "start_time": "2020-06-21T03:19:37.913179Z"
    }
   },
   "outputs": [],
   "source": [
    "lr_metrics.to_csv(results_path / 'lin_reg_metrics.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LightGBM Model Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The notebook example iterates over many configurations, optionally using random samples to speed up model selection using a diverse subset. The goal is to identify the most impactful parameters without trying every possible combination."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:37.935808Z",
     "start_time": "2020-06-21T03:19:37.921761Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_fi(model):\n",
    "    \"\"\"Return normalized feature importance as pd.Series\"\"\"\n",
    "    fi = model.feature_importance(importance_type='gain')\n",
    "    return (pd.Series(fi / fi.sum(),\n",
    "                      index=model.feature_name()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter Options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `base_params` are not affected by cross-validation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:37.943601Z",
     "start_time": "2020-06-21T03:19:37.936926Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "base_params = dict(boosting='gbdt',\n",
    "                   objective='regression',\n",
    "                   verbose=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We choose the following parameters and values to select our best model (see book chapter for detail):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:37.952308Z",
     "start_time": "2020-06-21T03:19:37.944671Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# constraints on structure (depth) of each tree\n",
    "max_depths = [2, 3, 5, 7]\n",
    "num_leaves_opts = [2 ** i for i in max_depths]\n",
    "min_data_in_leaf_opts = [250, 500, 1000]\n",
    "\n",
    "# weight of each new tree in the ensemble\n",
    "learning_rate_ops = [.01, .1, .3]\n",
    "\n",
    "# random feature selection\n",
    "feature_fraction_opts = [.3, .6, .95]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:37.960258Z",
     "start_time": "2020-06-21T03:19:37.953744Z"
    }
   },
   "outputs": [],
   "source": [
    "param_names = ['learning_rate', 'num_leaves',\n",
    "               'feature_fraction', 'min_data_in_leaf']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:37.969006Z",
     "start_time": "2020-06-21T03:19:37.961383Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Parameters: 108\n"
     ]
    }
   ],
   "source": [
    "cv_params = list(product(learning_rate_ops,\n",
    "                         num_leaves_opts,\n",
    "                         feature_fraction_opts,\n",
    "                         min_data_in_leaf_opts))\n",
    "n_params = len(cv_params)\n",
    "print(f'# Parameters: {n_params}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train/Test Period Lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:37.977749Z",
     "start_time": "2020-06-21T03:19:37.969903Z"
    }
   },
   "outputs": [],
   "source": [
    "lookaheads = [1, 5, 21]\n",
    "label_dict = dict(zip(lookaheads, labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only use test periods of 63 days length to save some model training and evaluation time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:37.986465Z",
     "start_time": "2020-06-21T03:19:37.978744Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train_lengths = [int(4.5 * 252), 252]\n",
    "test_lengths = [63]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:37.995277Z",
     "start_time": "2020-06-21T03:19:37.987379Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train configs: 6\n"
     ]
    }
   ],
   "source": [
    "test_params = list(product(lookaheads, train_lengths, test_lengths))\n",
    "n = len(test_params)\n",
    "test_param_sample = np.random.choice(list(range(n)), size=int(n), replace=False)\n",
    "test_params = [test_params[i] for i in test_param_sample]\n",
    "print('Train configs:', len(test_params))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Categorical Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We integer-encode categorical variables with values starting at zero, as expected by LightGBM (not necessary\n",
    "as long as the category codes have values less than $2^{32}$, but avoids a warning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:38.043537Z",
     "start_time": "2020-06-21T03:19:37.996178Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "categoricals = ['year', 'weekday', 'month']\n",
    "for feature in categoricals:\n",
    "    data[feature] = pd.factorize(data[feature], sort=True)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Loss Function: Information Coefficient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:38.046578Z",
     "start_time": "2020-06-21T03:19:38.044405Z"
    }
   },
   "outputs": [],
   "source": [
    "def ic_lgbm(preds, train_data):\n",
    "    \"\"\"Custom IC eval metric for lightgbm\"\"\"\n",
    "    is_higher_better = True\n",
    "    return 'ic', spearmanr(preds, train_data.get_label())[0], is_higher_better"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Cross-Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To explore the hyperparameter space, we specify values for key parameters that we would like to test in combination. The sklearn library supports `RandomizedSearchCV` to cross-validate a subset of parameter combinations that are sampled randomly from specified distributions. We will implement a custom version that allows us to monitor performance so we can abort the search process once we're satisfied with the result, rather than specifying a set number of iterations beforehand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:38.054547Z",
     "start_time": "2020-06-21T03:19:38.047670Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "lgb_store = Path(results_path / 'tuning_lgb.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:38.070171Z",
     "start_time": "2020-06-21T03:19:38.055637Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "labels = sorted(data.filter(like='fwd').columns)\n",
    "features = data.columns.difference(labels).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:38.078077Z",
     "start_time": "2020-06-21T03:19:38.071079Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "label_dict = dict(zip(lookaheads, labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:38.086476Z",
     "start_time": "2020-06-21T03:19:38.078921Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "num_iterations = [10, 25, 50, 75] + list(range(100, 501, 50))\n",
    "num_boost_round = num_iterations[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T03:19:38.094204Z",
     "start_time": "2020-06-21T03:19:38.087354Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "metric_cols = (param_names + ['t', 'daily_ic_mean', 'daily_ic_mean_n',\n",
    "                              'daily_ic_median', 'daily_ic_median_n'] +\n",
    "               [str(n) for n in num_iterations])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We iterate over our six CV configurations and collect the resulting metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T11:56:42.000694Z",
     "start_time": "2020-06-21T03:19:38.095078Z"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for lookahead, train_length, test_length in test_params:\n",
    "    # randomized grid search\n",
    "    cvp = np.random.choice(list(range(n_params)),\n",
    "                           size=int(n_params / 2),\n",
    "                           replace=False)\n",
    "    cv_params_ = [cv_params[i] for i in cvp]\n",
    "\n",
    "    # set up cross-validation\n",
    "    n_splits = int(2 * YEAR / test_length)\n",
    "    print(f'Lookahead: {lookahead:2.0f} | '\n",
    "          f'Train: {train_length:3.0f} | '\n",
    "          f'Test: {test_length:2.0f} | '\n",
    "          f'Params: {len(cv_params_):3.0f} | '\n",
    "          f'Train configs: {len(test_params)}')\n",
    "\n",
    "    # time-series cross-validation\n",
    "    cv = MultipleTimeSeriesCV(n_splits=n_splits,\n",
    "                              lookahead=lookahead,\n",
    "                              test_period_length=test_length,\n",
    "                              train_period_length=train_length)\n",
    "\n",
    "    label = label_dict[lookahead]\n",
    "    outcome_data = data.loc[:, features + [label]].dropna()\n",
    "    \n",
    "    # binary dataset\n",
    "    lgb_data = lgb.Dataset(data=outcome_data.drop(label, axis=1),\n",
    "                           label=outcome_data[label],\n",
    "                           categorical_feature=categoricals,\n",
    "                           free_raw_data=False)\n",
    "    T = 0\n",
    "    predictions, metrics, feature_importance, daily_ic = [], [], [], []\n",
    "    \n",
    "    # iterate over (shuffled) hyperparameter combinations\n",
    "    for p, param_vals in enumerate(cv_params_):\n",
    "        key = f'{lookahead}/{train_length}/{test_length}/' + '/'.join([str(p) for p in param_vals])\n",
    "        params = dict(zip(param_names, param_vals))\n",
    "        params.update(base_params)\n",
    "\n",
    "        start = time()\n",
    "        cv_preds, nrounds = [], []\n",
    "        ic_cv = defaultdict(list)\n",
    "        \n",
    "        # iterate over folds\n",
    "        for i, (train_idx, test_idx) in enumerate(cv.split(X=outcome_data)):\n",
    "            \n",
    "            # select train subset\n",
    "            lgb_train = lgb_data.subset(used_indices=train_idx.tolist(),\n",
    "                                       params=params).construct()\n",
    "            \n",
    "            # train model for num_boost_round\n",
    "            model = lgb.train(params=params,\n",
    "                              train_set=lgb_train,\n",
    "                              num_boost_round=num_boost_round,\n",
    "                              verbose_eval=False)\n",
    "            # log feature importance\n",
    "            if i == 0:\n",
    "                fi = get_fi(model).to_frame()\n",
    "            else:\n",
    "                fi[i] = get_fi(model)\n",
    "\n",
    "            # capture predictions\n",
    "            test_set = outcome_data.iloc[test_idx, :]\n",
    "            X_test = test_set.loc[:, model.feature_name()]\n",
    "            y_test = test_set.loc[:, label]\n",
    "            y_pred = {str(n): model.predict(X_test, num_iteration=n) for n in num_iterations}\n",
    "            \n",
    "            # record predictions for each fold\n",
    "            cv_preds.append(y_test.to_frame('y_test').assign(**y_pred).assign(i=i))\n",
    "        \n",
    "        # combine fold results\n",
    "        cv_preds = pd.concat(cv_preds).assign(**params)\n",
    "        predictions.append(cv_preds)\n",
    "        \n",
    "        # compute IC per day\n",
    "        by_day = cv_preds.groupby(level='date')\n",
    "        ic_by_day = pd.concat([by_day.apply(lambda x: spearmanr(x.y_test, x[str(n)])[0]).to_frame(n)\n",
    "                               for n in num_iterations], axis=1)\n",
    "        daily_ic_mean = ic_by_day.mean()\n",
    "        daily_ic_mean_n = daily_ic_mean.idxmax()\n",
    "        daily_ic_median = ic_by_day.median()\n",
    "        daily_ic_median_n = daily_ic_median.idxmax()\n",
    "        \n",
    "        # compute IC across all predictions\n",
    "        ic = [spearmanr(cv_preds.y_test, cv_preds[str(n)])[0] for n in num_iterations]\n",
    "        t = time() - start\n",
    "        T += t\n",
    "        \n",
    "        # collect metrics\n",
    "        metrics = pd.Series(list(param_vals) +\n",
    "                            [t, daily_ic_mean.max(), daily_ic_mean_n, daily_ic_median.max(), daily_ic_median_n] + ic,\n",
    "                            index=metric_cols)\n",
    "        msg = f'\\t{p:3.0f} | {format_time(T)} ({t:3.0f}) | {params[\"learning_rate\"]:5.2f} | '\n",
    "        msg += f'{params[\"num_leaves\"]:3.0f} | {params[\"feature_fraction\"]:3.0%} | {params[\"min_data_in_leaf\"]:4.0f} | '\n",
    "        msg += f' {max(ic):6.2%} | {ic_by_day.mean().max(): 6.2%} | {daily_ic_mean_n: 4.0f} | {ic_by_day.median().max(): 6.2%} | {daily_ic_median_n: 4.0f}'\n",
    "        print(msg)\n",
    "\n",
    "        # persist results for given CV run and hyperparameter combination\n",
    "        metrics.to_hdf(lgb_store, 'metrics/' + key)\n",
    "        ic_by_day.assign(**params).to_hdf(lgb_store, 'daily_ic/' + key)\n",
    "        fi.T.describe().T.assign(**params).to_hdf(lgb_store, 'fi/' + key)\n",
    "        cv_preds.to_hdf(lgb_store, 'predictions/' + key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CatBoost Model Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We repeat a similar process for CatBoost - see book and CatBoost [docs](https://catboost.ai/docs/concepts/about.html) for detail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter Options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T11:56:42.002366Z",
     "start_time": "2020-06-21T03:17:13.861Z"
    }
   },
   "outputs": [],
   "source": [
    "param_names = ['max_depth', 'min_child_samples']\n",
    "\n",
    "max_depth_opts = [3, 5, 7, 9]\n",
    "min_child_samples_opts = [20, 250, 500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T11:56:42.002915Z",
     "start_time": "2020-06-21T03:17:14.083Z"
    }
   },
   "outputs": [],
   "source": [
    "cv_params = list(product(max_depth_opts,\n",
    "                         min_child_samples_opts))\n",
    "n_params = len(cv_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train/Test Period Lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T11:56:42.003547Z",
     "start_time": "2020-06-21T03:17:14.456Z"
    }
   },
   "outputs": [],
   "source": [
    "lookaheads = [1, 5, 21]\n",
    "label_dict = dict(zip(lookaheads, labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T11:56:42.004201Z",
     "start_time": "2020-06-21T03:17:16.556Z"
    }
   },
   "outputs": [],
   "source": [
    "train_lengths = [int(4.5 * 252), 252]\n",
    "test_lengths = [63]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T11:56:42.004875Z",
     "start_time": "2020-06-21T03:17:17.116Z"
    }
   },
   "outputs": [],
   "source": [
    "test_params = list(product(lookaheads,\n",
    "                           train_lengths,\n",
    "                           test_lengths))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Loss Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T11:56:42.005692Z",
     "start_time": "2020-06-21T03:17:18.773Z"
    }
   },
   "outputs": [],
   "source": [
    "class CatBoostIC(object):\n",
    "    \"\"\"Custom IC eval metric for CatBoost\"\"\"\n",
    "\n",
    "    def is_max_optimal(self):\n",
    "        # Returns whether great values of metric are better\n",
    "        return True\n",
    "\n",
    "    def evaluate(self, approxes, target, weight):\n",
    "        target = np.array(target)\n",
    "        approxes = np.array(approxes).reshape(-1)\n",
    "        rho = spearmanr(approxes, target)[0]\n",
    "        return rho, 1\n",
    "\n",
    "    def get_final_error(self, error, weight):\n",
    "        # Returns final value of metric based on error and weight\n",
    "        return error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Cross-Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T11:56:42.006427Z",
     "start_time": "2020-06-21T03:17:29.495Z"
    }
   },
   "outputs": [],
   "source": [
    "cb_store = Path(results_path / 'tuning_catboost.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T11:56:42.007167Z",
     "start_time": "2020-06-21T03:17:29.677Z"
    }
   },
   "outputs": [],
   "source": [
    "num_iterations = [10, 25, 50, 75] + list(range(100, 1001, 100))\n",
    "num_boost_round = num_iterations[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T11:56:42.007861Z",
     "start_time": "2020-06-21T03:17:29.812Z"
    }
   },
   "outputs": [],
   "source": [
    "metric_cols = (param_names + ['t', 'daily_ic_mean', 'daily_ic_mean_n',\n",
    "                              'daily_ic_median', 'daily_ic_median_n'] +\n",
    "               [str(n) for n in num_iterations])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-21T11:56:42.008624Z",
     "start_time": "2020-06-21T03:17:30.033Z"
    }
   },
   "outputs": [],
   "source": [
    "for lookahead, train_length, test_length in test_params:\n",
    "    cvp = np.random.choice(list(range(n_params)),\n",
    "                           size=int(n_params / 1),\n",
    "                           replace=False)\n",
    "    cv_params_ = [cv_params[i] for i in cvp]\n",
    "\n",
    "    n_splits = int(2 * YEAR / test_length)\n",
    "    print(f'Lookahead: {lookahead:2.0f} | Train: {train_length:3.0f} | '\n",
    "          f'Test: {test_length:2.0f} | Params: {len(cv_params_):3.0f} | Train configs: {len(test_params)}')\n",
    "\n",
    "    cv = MultipleTimeSeriesCV(n_splits=n_splits,\n",
    "                              lookahead=lookahead,\n",
    "                              test_period_length=test_length,\n",
    "                              train_period_length=train_length)\n",
    "\n",
    "    label = label_dict[lookahead]\n",
    "    outcome_data = data.loc[:, features + [label]].dropna()\n",
    "    cat_cols_idx = [outcome_data.columns.get_loc(c) for c in categoricals]\n",
    "    catboost_data = Pool(label=outcome_data[label],\n",
    "                         data=outcome_data.drop(label, axis=1),\n",
    "                         cat_features=cat_cols_idx)\n",
    "    predictions, metrics, feature_importance, daily_ic = [], [], [], []\n",
    "    key = f'{lookahead}/{train_length}/{test_length}'\n",
    "    T = 0\n",
    "    for p, param_vals in enumerate(cv_params_):\n",
    "        params = dict(zip(param_names, param_vals))\n",
    "        # uncomment if running with GPU\n",
    "#         params['task_type'] = 'GPU'\n",
    "\n",
    "        start = time()\n",
    "        cv_preds, nrounds = [], []\n",
    "        ic_cv = defaultdict(list)\n",
    "        for i, (train_idx, test_idx) in enumerate(cv.split(X=outcome_data)):\n",
    "            train_set = catboost_data.slice(train_idx.tolist())\n",
    "\n",
    "            model = CatBoostRegressor(**params)\n",
    "            model.fit(X=train_set,\n",
    "                      verbose_eval=False)\n",
    "\n",
    "            test_set = outcome_data.iloc[test_idx, :]\n",
    "            X_test = test_set.loc[:, model.feature_names_]\n",
    "            y_test = test_set.loc[:, label]\n",
    "            y_pred = {str(n): model.predict(X_test, ntree_end=n)\n",
    "                      for n in num_iterations}\n",
    "            cv_preds.append(y_test.to_frame(\n",
    "                'y_test').assign(**y_pred).assign(i=i))\n",
    "\n",
    "        cv_preds = pd.concat(cv_preds).assign(**params)\n",
    "        predictions.append(cv_preds)\n",
    "        by_day = cv_preds.groupby(level='date')\n",
    "        ic_by_day = pd.concat([by_day.apply(lambda x: spearmanr(x.y_test, x[str(n)])[0]).to_frame(n)\n",
    "                               for n in num_iterations], axis=1)\n",
    "        daily_ic_mean = ic_by_day.mean()\n",
    "        daily_ic_mean_n = daily_ic_mean.idxmax()\n",
    "        daily_ic_median = ic_by_day.median()\n",
    "        daily_ic_median_n = daily_ic_median.idxmax()\n",
    "\n",
    "        ic = [spearmanr(cv_preds.y_test, cv_preds[str(n)])[0]\n",
    "              for n in num_iterations]\n",
    "        t = time() - start\n",
    "        T += t\n",
    "        metrics = pd.Series(list(param_vals) +\n",
    "                            [t, daily_ic_mean.max(), daily_ic_mean_n,\n",
    "                             daily_ic_median.max(), daily_ic_median_n] + ic,\n",
    "                            index=metric_cols)\n",
    "        msg = f'{p:3.0f} | {format_time(T)} ({t:3.0f}) | {params[\"max_depth\"]:3.0f} | {params[\"min_child_samples\"]:4.0f} | '\n",
    "        msg += f' {max(ic):6.2%} | {ic_by_day.mean().max(): 6.2%} | {daily_ic_mean_n: 4.0f} | {ic_by_day.median().max(): 6.2%} | {daily_ic_median_n: 4.0f}'\n",
    "        print(msg)\n",
    "        metrics.to_hdf(cb_store, 'metrics/' + key)\n",
    "        ic_by_day.assign(**params).to_hdf(cb_store, 'daily_ic/' + key)\n",
    "        cv_preds.to_hdf(cb_store, 'predictions/' + key)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "292.031px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}