marketing-analytics/channel-attribution-lstm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-touch Multi-channel Attribution Model Using LSTM with Attention\n",
    "\n",
    "This notebook demonstrates how multi-touch attribution can be performed by analyzing weights in attention layers or transformers. See references [1] and [2] for details.\n",
    "\n",
    "### Use Case\n",
    "We have a large number of customer interaction histories (journeys). Each history is a sequence of events (impressions) of different types. For example, each event can be linked to one of a few ad campaigns. Each history also has an outcome label such as a conversion, click, etc. Our goal is to estimate the contribution of each event type to the outcome. We also would like to have a method for evaluating and comparing different attribution models.\n",
    "\n",
    "### Prototype: Approach and Data\n",
    "\n",
    "We build a progression of attribution models starting with basic last-touch attribution and logistic regression, and then create an attention-based model. \n",
    "\n",
    "We use Critio dataset (see `datasets.md`) for prototyping purposes. Download the dataset to `data` folder. The dataset represents a sample of 30 days of Criteo live traffic data. Each line corresponds to one impression (a banner) that was displayed to a user. For each banner we have detailed information about the context, if it was clicked, if it led to a conversion and if it led to a conversion that was attributed to Criteo or not. Data has been sub-sampled and anonymized so as not to disclose proprietary elements. Here is a detailed description of the fields (they are tab-separated in the file):\n",
    "* `timestamp` - timestamp of the impression (starting from 0 for the first impression). The dataset is sorted according to timestamp.\n",
    "* `uid` - a unique user identifier\n",
    "* `campaign` - a unique identifier for the campaign\n",
    "* `conversion` - 1 if there was a conversion in the 30 days after the impression (independently of whether this impression was last click or not)\n",
    "* `conversion_timestamp` - the timestamp of the conversion or -1 if no conversion was observed\n",
    "* `conversion_id` - a unique identifier for each conversion (so that timelines can be reconstructed if needed). -1 if there was no conversion\n",
    "* `attribution` - 1 if the conversion was attributed to Criteo, 0 otherwise\n",
    "* `click` - 1 if the impression was clicked, 0 otherwise\n",
    "* `click_pos` - the position of the click before a conversion (0 for first-click)\n",
    "* `click_nb` - number of clicks. More than 1 if there were several clicks before a conversion\n",
    "* `cost` - the price paid by Criteo for this display (disclaimer: not the real price, only a transformed version of it)\n",
    "* `cpo` - the cost-per-order in case of attributed conversion (disclaimer: not the real price, only a transformed version of it)\n",
    "* `time_since_last_click` - the time since the last click (in s) for the given impression\n",
    "* `cat(1-9)` - contextual features associated to the display. Can be used to learn the click/conversion models. We do not disclose the meaning of these features, but it is not relevant for this study. Each column is a categorical variable. In the experiments, they are mapped to a fixed dimensionality space using the Hashing Trick (see paper for reference).\n",
    "\n",
    "Key figures: 2.4Gb uncompressed, 16.5M impressions, 45K conversions, 700 campaigns\n",
    "\n",
    "### Usage and Productization\n",
    "The dataset used in the prototype can be replaced with the actual campaign or click stream data. Campaigns can be replaced with channels or other qualifiers that need to be attributed with the contribution value. Then basic and advanced models can be compared to each other and the optimal approach can be chosen.\n",
    "\n",
    "### References\n",
    "1. N. Li, S. K. Arava, C. Dong, Z. Yan, and A. Pani -- Deep Neural Net with Attention for Multi-channel Multi-touch Attribution, 2018\n",
    "2. Ren K., Fang Y., Zhang W., Liu S., Li J., Zhang Y., Yu Y., and Wang J. -- Learning Multi-Touch Conversion Attribution with Dual-Attention Mechanisms for Online Advertising, 2018"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Imports and settings\n",
    "#\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.utils.extmath import softmax\n",
    "\n",
    "from keras.models import Sequential \n",
    "from keras.layers import Dense, LSTM, Input, Lambda, RepeatVector\n",
    "from keras.layers import Permute, Flatten, Activation, Multiply\n",
    "from keras import backend as K\n",
    "from keras.models import Model\n",
    "\n",
    "plt.style.use('ggplot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initial data preparation\n",
    "\n",
    "def add_derived_columns(df):\n",
    "    df_ext = df.copy()\n",
    "    df_ext['jid'] = df_ext['uid'].map(str) + '_' + df_ext['conversion_id'].map(str)\n",
    "    \n",
    "    min_max_scaler = MinMaxScaler()\n",
    "    for cname in ('timestamp', 'time_since_last_click'):\n",
    "        x = df_ext[cname].values.reshape(-1, 1) \n",
    "        df_ext[cname + '_norm'] = min_max_scaler.fit_transform(x)\n",
    "    \n",
    "    return df_ext\n",
    "\n",
    "def filter_journeys_by_length(df, min_touchpoints):\n",
    "    if min_touchpoints <= 1:\n",
    "        return df\n",
    "    else:\n",
    "        grouped = df.groupby(['jid'])['uid'].count().reset_index(name=\"count\")\n",
    "        return df[df['jid'].isin( grouped[grouped['count'] >= min_touchpoints]['jid'].values )]\n",
    "\n",
    "def sample_campaigns(df, n_campaigns):    \n",
    "    campaigns = np.random.choice( df['campaign'].unique(), n_campaigns, replace = False )\n",
    "    return df[ df['campaign'].isin(campaigns) ]\n",
    "\n",
    "def balance_conversions(df):\n",
    "    df_minority = df[df.conversion == 1]\n",
    "    df_majority = df[df.conversion == 0]\n",
    "    \n",
    "    df_majority_jids = np.array_split(df_majority['jid'].unique(), 100 * df_majority.shape[0]/df_minority.shape[0] )\n",
    "    \n",
    "    df_majority_sampled = pd.DataFrame(data=None, columns=df.columns)\n",
    "    for jid_chunk in df_majority_jids:\n",
    "        df_majority_sampled = pd.concat([df_majority_sampled, df_majority[df_majority.jid.isin(jid_chunk)]])\n",
    "        if df_majority_sampled.shape[0] > df_minority.shape[0]:\n",
    "            break\n",
    "    \n",
    "    return pd.concat([df_majority_sampled, df_minority]).sample(frac=1).reset_index(drop=True)\n",
    "\n",
    "def map_one_hot(df, column_names, result_column_name):\n",
    "    mapper = {} \n",
    "    for i, col_name in enumerate(column_names):\n",
    "        for val in df[col_name].unique():\n",
    "            mapper[str(val) + str(i)] = len(mapper)\n",
    "         \n",
    "    df_ext = df.copy()\n",
    "    \n",
    "    def one_hot(values):\n",
    "        v = np.zeros( len(mapper) )\n",
    "        for i, val in enumerate(values): \n",
    "            v[ mapper[str(val) + str(i)] ] = 1\n",
    "        return v    \n",
    "    \n",
    "    df_ext[result_column_name] = df_ext[column_names].values.tolist()\n",
    "    df_ext[result_column_name] = df_ext[result_column_name].map(one_hot)\n",
    "    \n",
    "    return df_ext\n",
    "    \n",
    "data_file = 'data/criteo_attribution_dataset.tsv.gz'\n",
    "df0 = pd.read_csv(data_file, sep='\\t', compression='gzip')\n",
    "\n",
    "n_campaigns = 400\n",
    "\n",
    "df1 = add_derived_columns(df0)\n",
    "df2 = sample_campaigns(df1, n_campaigns)\n",
    "df3 = filter_journeys_by_length(df2, 2)\n",
    "df4 = balance_conversions(df3)\n",
    "df5 = map_one_hot(df4, ['cat1', 'cat2', 'cat3', 'cat4', 'cat5', 'cat6', 'cat8'], 'cats')\n",
    "df6 = map_one_hot(df5, ['campaign'], 'campaigns').sort_values(by=['timestamp_norm'])\n",
    "\n",
    "print(df6.shape[0])\n",
    "print([df6[df6.conversion == 0].shape[0], df6[df6.conversion == 1].shape[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[145440]\n",
      "         jid\n",
      "count       \n",
      "2      58572\n",
      "3      27400\n",
      "4      15811\n",
      "5       9922\n",
      "6       6998\n",
      "7       4915\n",
      "8       3739\n",
      "9       3020\n",
      "10      2319\n",
      "11      1907\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAENCAYAAADOhVhvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XlYlOX6wPHvM6wuCMyAICKouOUekvuCSqXWqSzT6lTH\nzEpLy5bzS8uWU5nWOZZZmJVLamm2aHvZITP3FLdjLimaIooi4IIh6/v8/hgZRVkGBGbx/lxXVzPv\nvMv9DPjePO+zKa21RgghhKgEk6MDEEII4bokiQghhKg0SSJCCCEqTZKIEEKISpMkIoQQotIkiQgh\nhKg0SSJCCCEqTZKIEEKISpMkIoQQotIkiQghhKg0T0cHUBOOHDlSqeOCgoJIT0+v4mhqlpTBOUgZ\nnIc7lKMmyhAWFmbXflITEUIIUWkuVRM5duwYS5YsITs7myeffNLR4QghxBWvxmoiM2bMYOTIkZfc\n/Ldu3cpjjz3G2LFj+fLLL8s8R0hICKNHj67OMIUQQlRAjdVEYmNjGTBgAPHx8bZthmEwe/ZsJk6c\niMViYcKECcTExGAYBgsXLix2/OjRo/H396+pcIUQQtihxpJI69atSUtLK7YtKSmJ0NBQQkJCAOje\nvTsbN25k8ODBjB8/vqZCE0IIUUkObRPJzMzEYrHY3lssFvbu3Vvq/llZWSxatIgDBw6wdOlSBg8e\nXOJ+CQkJJCQkADBlyhSCgoIqFZ+np2elj3UWUgbnIGVwHu5QDmcqg0s1rPv5+fHggw+Wu19cXBxx\ncXG295XtCiddAZ2DlME5uEMZwD3KIV18zzGbzWRkZNjeZ2RkYDabq+TciYmJvPfee5U+Xm9Zz19f\nLSx/RyGEuII5NIlERUWRmppKWloaBQUFrF27lpiYmCo5d0xMDA899FClj9c7NvPXZ/OQJeiFEKJ0\nNfY4a9q0aezcuZOsrCxGjRrF0KFD6devHyNGjGDSpEkYhkHfvn1p1KhRlVwvMTGRTZs2VT6RhDdG\n//oj6kQ6mIOrJCYhhHA3NZZExo0bV+L26OhooqOjq/x6MTExl1WrUeGN0QApBySJCCFEKdx22pPL\nbRMhLBIAnXKgagISQgg35FK9syrismsitetgqt9AkogQQpTBbWsiVcErMkqSiBBClMFtk8hlP84C\nPBs3g2OH0fl5VRSVEEK4F3mcVQbPyCgwDEg9BBFRVRSZEEK4D7etiVQFz0hr4pBHWkIIUTK3TSJV\n8TjLo0E4eHlbu/kKIYS4hDzOKoPy8ISwCKmJCCFEKdy2JlJVVHik1ESEEKIUkkTKE94Ysk6hT59w\ndCRCCOF03DaJVEWbCIBq2Nj6QmojQghxCWkTKU9kFCiFTtqNan315Z9PCCHciNvWRKqKql0XGjVB\n7/nd0aEIIYTTkSRiB9WiHez/Q0auCyHERSSJ2EG1bAP5efDnHkeHIoQQTkWSiD2at7G2i8gjLSGE\nKMZtk0hV9c4CUHX8oGFj9B+SRIQQ4kLSO8tOqmVb9Kpl6Px8lJdXlZ1XCCFcmdvWRKqaatEW8vLg\nwF5HhyKEEE5Dkoi9WrQBkHYRIYS4gCQRO6m69aBhJPqP7Y4ORQghnIYkkQpQrTvC3h3onLOODkUI\nIZyCJJEKUO1ioKAAdm9zdChCCOEU3DaJVGUXX5vmrcG3Fvp/iVV7XiGEcFHSxbcClKcXtL4avX0T\nWmuUUlV6fiGEcDVuWxOpLqp9DJzMgEN/OjoUIYRwOEkiFaTadgJAb5dHWkIIIUmkgpR/IEQ2kyQi\nhBBIEqkU1S7GOjV81mlHhyKEEA4lSaQSVMcuoDV60xpHhyKEEA4lSaQyIppCeBP06v86OhIhhHAo\nSSKVoJRC9boWDiahk/c5OhwhhHAYl0siGzZsYObMmbz55pts2+a4keOqSyx4ekltRAhxRavRwYYz\nZsxg8+bN+Pv7M3XqVNv2rVu3MnfuXAzDoH///txyyy2lnqNz58507tyZM2fOsGDBAjp06FAToV9C\n1amLiu6OXv8resh9KG8fh8QhhBCOVKNJJDY2lgEDBhAfH2/bZhgGs2fPZuLEiVgsFiZMmEBMTAyG\nYbBw4cJix48ePRp/f38AlixZwvXXX1+T4V9C9boWveFX9Oa1qK59HRqLEEI4Qo0mkdatW5OWllZs\nW1JSEqGhoYSEhADQvXt3Nm7cyODBgxk/fvwl59Ba8/HHH9OxY0eaNm1a4nUSEhJISEgAYMqUKQQF\nBVUqXk9PzzKP1d1jyfi4Iab1KzDfeHulrlHdyiuDK5AyOAd3KAO4RzmcqQwOnzsrMzMTi8Vie2+x\nWNi7t/TVA3/44Qe2b99OdnY2R48e5brrrrtkn7i4OOLi4mzv09PTKxVbUFBQuccaXftS+OVHHN/5\nP1T9sEpdpzrZUwZnJ2VwDu5QBnCPctREGcLC7LufVSqJ5OXloZTCywFrjQ8aNIhBgwaVu19iYiKb\nNm3ioYceqtZ4VPf+6K8Wotf8jBp8T7VeSwghnI1dvbPmz59PUlISAJs3b+a+++7jvvvuIzHx8qf+\nMJvNZGRk2N5nZGRgNpsv+7wxMTHVnkAAVKAF2kaj1/6MLiys9usJIYQzsSuJrF69mkaNGgHw+eef\nM3bsWP7v//6PRYsWXXYAUVFRpKamkpaWRkFBAWvXrq2SKdyrZT2RUph6XgsnM2HH5hq5nhBCOAu7\nHmfl5ubi4+NDVlYWx44do2vXrkDF2xqmTZvGzp07ycrKYtSoUQwdOpR+/foxYsQIJk2ahGEY9O3b\n15awLkd1rCdSqvbXgJ8/xur/4tH+mpq5phBCOAG7kkhYWBirVq3i6NGjtG/fHoDTp0/j7e1doYuN\nGzeuxO3R0dFER0dX6FzORHl6orr1Rf/8DfpkBirAUv5BQgjhBux6nHX//fezbNkyduzYwbBhwwDY\ntm2bLaE4o5p8nAWg+gy0Tsq4bGmNXVMIIRxNaa21o4OobkeOHKnUcRXtRmfMfQu9cRWmyR9Y1x1x\nAtKd0TlIGZyHO5TDmbr42lUT+ec//8l3333HqVOnLisod6duuB0KC9DLljg6FCGEqBF2JZEhQ4aw\na9cuxowZw6uvvsrq1avJy8ur7tguS00/zgJQ9cNQXWLRv/6APn2iRq8thBCOYFfDepcuXejSpQtn\nzpxh7dq1LFu2jFmzZtG5c2d69+5N27ZtqzvOCqvR3lkXUDcMRa9fgf72U9Rd1T9ORQghHKlCI9br\n1q1LbGwsvr6+fP311/z222/s2rULk8nE/fff79QN7TVFhYSh+gxAr/ge3a0vqkkLR4ckhBDVxq4k\norVm27ZtrFy5ks2bN9OiRQtuueUWOnfujLe3N+vXr+ftt9/mgw8+qO547VZT056URN16L3rrbxjz\n3sY08U2Up8OnKBNCiGph193twQcfpF69evTu3Zu77777kmlJunbtyrJly6olwMpy1OMsAFWrNqa/\nj8KIn4T+aSlqkHPO8CuEEJfLriQyfvx4oqKiytznhRdeqJKA3IXq2AU6dUd/8wm6SyzKEuzokIQQ\nosrZ1TsrKiqKw4cP8/nnnzNr1iwADh8+zMGDB6s1OFdnuv1+QKO/W+zoUIQQolrYlUTWrVvH888/\nT2ZmJqtWrQIgJyeH+fPnV2twl8MRXXwvpizBqN4D0GsS0GmVG/AohBDOzK4k8umnn/Lcc8/x4IMP\nYjJZD4mMjOTAgQPVGdtlqamp4MujBg4BT0/0N1IbEUK4H7uSyKlTp4iMjCy2TSmFUqpagnInKsCM\n6nsD+rcV6CPJjg5HCCGqlF1JpGnTpqxcubLYtjVr1tCsWbNqCcrdqOtvA99aGB+/izZk4SohhPuw\nK4ncd999fPLJJ7zwwgvk5uYyadIkFi9ezD/+8Y/qjs8tKL96qGEjYc8O9H+/dnQ4QghRZezq4tuw\nYUOmTZvGpk2b6NSpExaLhU6dOuHr61vd8VWaIwcblkR174/etgH95QJ0m46o8CaODkkIIS6b3UOp\nfXx86N69e3XGUqUcOdiwJEopTPc8gvHiWIxZb2B6dirKq2KLegkhhLOxK4mkpaWxaNEiDhw4QE5O\nTrHP3n333WoJzB0pP39Mwx/FmP4S+ot5qDsecHRIQghxWexKIm+99RYhISHce++9+Pj4VHdMbk21\ni0H1/5t1Kd02V6PaOU9tSQghKsquJJKSksLLL79sGyMiLo+67R/o3f/DmPsWpheno+o5xyqIQghR\nUXZlhauuusqpBxa6GuXljemBf8LZbIyPHTuqXgghLoddNZHg4GAmTZpE586dCQgIKPbZsGHDqiUw\nd6caRqBuuhO9ZD5601pUJ9fptCCEEEXsqonk5ubSqVMnCgsLycjIKPafs3KGubPKo669BSKaYix6\nD/3XGUeHI4QQFVZuTcQwDHr37k3Lli3x8vKqiZiqhLN18S2J8vTE9I+xGJOeRH82GzX8MUeHJIQQ\nFVJuTcRkMvH666+7VAJxJSoiCnX9reg1P6O3bXB0OEIIUSF2N6zv2bOnumO5Yqm/3QnhjTHmvY3O\nOuXocIQQwm52N6xPnjyZmJgYLBZLsdl7pWH98ikvL0z3P4Ex6QmM+fGYHp4gMyQLIVyCXTWRvLw8\nrrnmGpRSZGZmukTDuqtR4Y1Rt9wDW9ejf/jc0eEIIYRd7KqJPPzww9UdhwDUtTdD8j700gUYnp6Y\nrhvs6JCEEKJMdiWRY8eOlfpZSEhIlQVzpVMmE4x4HAoL0Z/NxdAadd1gebQlhHBadiWRRx99tNTP\nFi+WZV+rkvLwgJFPAqA//xCS98M9j6B8azk2MCGEKIFdSeTiRHHy5Ek+++wzrrrqqmoJqjQpKSl8\n//33ZGVl0a5dO6677roavX5NUZ6e8OA/4cem6C8/Rh/6E9MTL6ECLI4OTQghiqnUjIoBAQEMHz6c\nhQsX2n3MjBkzGDlyJE8++WSx7Vu3buWxxx5j7NixfPnll2WeIzw8nAcffJDHH3+c3bt3VyZ0l6FM\nJkyDbsf0+L8g8zjGjMno/HxHhyWEEMXYvSjVxY4cOUJubq7d+8fGxjJgwADi4+Nt2wzDYPbs2Uyc\nOBGLxcKECROIiYnBMIxLEtTo0aPx9/cnMTGRn376id69e1c2dJeiruqAacQ4jHenoD9+F/4xVtpI\nhBBOw64k8vzzzxe7ceXm5nLo0CGGDBli94Vat25NWlpasW1JSUmEhobaGue7d+/Oxo0bGTx4MOPH\njy/xPEXTmUyePJmePXvafX1XpqK7o24chv52MTRpgeozwNEhCSEEYGcS6devX7H3vr6+REZG0qBB\ng8u6eGZmJhbL+ef8FouFvXv3lrr/jh07+O233ygoKODqq68udb+EhAQSEhIAmDJlCkFBQZWKz9PT\ns9LHVjV931hOHNhL4TeLsNw01O6ldZ2pDJUlZXAO7lAGcI9yOFMZ7EoisbGx1RyGfdq0aUObNm3K\n3S8uLo64uDjb+/T09EpdLygoqNLHVgcddzPGtBc4/uNXmLr1tesYZytDZUgZnIM7lAHcoxw1UYaw\nsDC79rMriRQUFLBkyRJWrlzJiRMnCAwMpHfv3tx66614ela6WQWz2Vxs1HtGRgZms7nS57tQYmIi\nmzZt4qGHHqqS8zmF1h0hNNy6tG7XWGkbEUI4nF0Z4KOPPmLfvn088MADBAcHc/z4cb744guys7MZ\nPnx4pS8eFRVFamoqaWlpmM1m1q5dW+aYlIpwhangK0ophep/I/rjmbBvNzSr2S7WQghxMbu6+K5f\nv57/+7//o0OHDoSFhdGhQweeeuop1q1bZ/eFpk2bxsSJEzly5AijRo1i+fLleHh4MGLECCZNmsTj\njz9Ot27daNSoUaULcyFXWJSqMlTXvlCrDnr5t44ORQgh7KuJaK0v+0Ljxo0rcXt0dDTR0dGXff6L\nuWNNBED51kL1jEP//A3Girao3tdbp0sRQggHsOvu061bN1577TW2bt1KSkoKW7du5d///jfdunWr\n7vgqzV1rIgBq4O3QvA3643cxXnsafTTF0SEJIa5QSttRzSgoKOCLL75g9erVtob1Hj16cNttt7nE\niodHjhyp1HHO3ItDa41evwL96Wzw8sb0zH9QAZd2SnDmMthLyuAc3KEM4B7lcLneWZ6engwbNkwW\noHIiSilUt77o8MYYrz2NMeNVTE9NQnn7ODo0IcQVpNQksnPnTlq3bg3A77//XvoJPD0JDg4uNmjQ\nGbhlF98SqEZNrKsivjsZ/eF0GPG4dQJHIYSoAaXebWbPns3UqVMBePfdd0s9gdaarKwsBg4cyF13\n3VX1EVaSuzasl0Rd3RV12z/Qn3+IPpGB6cF/ogKdK6kLIdxTqUmkKIEAxSZNLMnp06d57LHHnCqJ\nXGlM19+KEWBBL4jHeHkcplHjUS3KH90vhBCXo0r6htarV4+JEydWxamqjDv3ziqNqUsfTM9OhTp1\nMaa9gN6e6OiQhBBursoGGERFRVXVqapETEyM27eHlEQ1aITp/6ZAg0YY8ZPIWbPc0SEJIdyYjFJz\nQ8rPH9OTr0CTFpya9i905nFHhySEcFOlJpHExPOPQgoKCmokGFF1VO06mEY+CWj0j184OhwhhJsq\nNYm8/fbbttf3339/jQRTla7ENpGLKUt9avW7Ab3qJ3Smaw+uEkI4p1J7ZwUEBPDjjz8SHh5OYWFh\nqWNF2rZtW23BXY4rqYtvWercdi9nf/4W/eMXqLuuvDYiIUT1KjWJPPzww3z66ad8//33FBQUlDhW\nRCnFO++8U60BisvjUb8Bqnt/9Kpl6L43oBqEOzokIYQbKTWJtGzZkueeew6AsWPHFnu8JVyLGjgE\n/duvGC88Aq3aY+p3I6pjF0eHJYRwA3b1zipKIOnp6ezZs8flJy+70qjgUEwvxaNuvAPSUjHiJ2HM\neROdk+3o0IQQLs6uSZZOnjzJm2++yZ49e/Dz8yMrK4sWLVrw2GOPVdlytlXtSpk7y17KUh91053o\nG4aiv12M/u5TdNIuTE+8jAoKcXR4QggXZVdN5P333ycyMpK5c+fy/vvvM3fuXBo3bswHH3xQ3fFV\n2pU62LA8ysMD0813Yfrnq3DqBPqrhY4OSQjhwuxKIn/88Qf33nsvvr6+APj6+nL33XezZ8+eag1O\nVB/VvDUqdhD6t1/RRw87OhwhhIuyK4nUqVOHlJTiq+cdOXKE2rVrV0tQomao6weDlyf6u08dHYoQ\nwkXZ1SZy00038fLLL9OvXz+Cg4M5fvw4K1askEWqXJyqF2Ctjfz3a/QNQ1GhDR0dkhDCxdiVROLi\n4ggNDWX16tUkJycTGBjIo48+Srt27ao7PlHN1PWD0Su+x3jjOfCtBYaBuqo9KqYnNG+NMnk4OkQh\nhBOzewm8tm3bOu3odFF5ql4gathI9OZ11iRSUIBe+zN6xQ8Q3hjT8MdQkc41Q7MQwnnIOqoCU+8B\n0HuA7b3OzUFvXof+4kOMV59EDRqK+tsdKJNM+iyEKM5t7woyAWPlKR9fTN36YvrXO6jOfdDffoL+\nXhrfhRCXctuaiEzAePlUHT8YMQ60gf56EbpxC1TbaEeHJYRwIm5bExFVQymFuucRCIvAmDUVnZHm\n6JCEEE7ErprImTNn+Prrrzl48CA5OTnFPvvXv/5VLYEJ56F8fDE9PAHjlScw3v83pn9ORnm6bSVW\nCFEBdt0J3nrrLQoKCujWrRve3t7VHZNwQqp+GOqeR9Dv/xv93WLUzX93dEhCCCdgVxLZs2cPs2bN\nwsvLq7rjEU7MdE0vjO2b0N99hm59Nap5a0eHJIRwMLvaRCIiIsjIyKjuWIQLUHc9CEH1Md57HWP5\nt+jsM44OSQjhQHbVRNq2bcurr75KbGwsAQEBxT7r169ftQQmnJPyrY1p1HiMedPRi95Hf/EhatgD\nmHpf7+jQhBAOYFcS2b17NxaLhe3bt1/ymSSRK4+KaIrHc9PQB5MwvpiH/uhddL0AWS1RiCuQXUnk\nhRdeqO447JaTk8OLL77I7bffTqdOnRwdzhVNRTbD9MizGP95FuODf2N6ajKqSXNHhyWEqEF299M8\nc+YMmzZtIjMzE7PZTKdOnahbt67dF5oxYwabN2/G39+fqVOn2rZv3bqVuXPnYhgG/fv355Zbbinz\nPF999RXdunWz+7qieikfX0xjJ2JM/j+M+EmYXn0P5e3j6LCEEDXE7t5ZkydPpmHDhgQFBbF582Y+\n/PBDJkyYQIsWLey6UGxsLAMGDCA+Pt62zTAMZs+ezcSJE7FYLEyYMIGYmBgMw2DhwuIr7o0ePZqD\nBw8SHh5Ofn5+BYooqpuqF4jp3jEYbzyH3rgK1SPO0SEJIWqIXUnkww8/ZOTIkfTo0cO2be3atcyd\nO5fJkyfbdaHWrVuTllZ8tHNSUhKhoaGEhFjX+O7evTsbN25k8ODBjB8//pJz7Nixg9zcXFJSUvD2\n9ubqq6/GVMKkgAkJCSQkJAAwZcoUgoKC7IrxYp6enpU+1lnUVBl0z35kfDYHtfJHzDcNQylVZeeW\nn4NzcIcygHuUw5nKYFcSSU1NveQRUteuXS97jfXMzEwsFovtvcViYe/evaXuf+eddwKwYsUK/Pz8\nSkwgYF3/JC7u/F/D6enplYovKCio0sc6i5osg9FnIPqjGaSvXwVRrdAL4tFpRzA98QrKo/LrksjP\nwTm4QxnAPcpRE2UICwuzaz+7xomEhoaydu3aYtvWrVtnq0HUtNjY2HIb1WUW35qnusZC7Tron79B\nf/IBevV/Yc8O9KqfHB2aEKKa2FUTGT58OFOmTOGHH34gKCiI48ePk5qaWuIjp4owm83FBjFmZGRg\nNpsv65xFZBbfmqd8fFE9r0P/tNT6/rpb0AeS0F8vRHfujapdx8ERCiGqml01kZYtW/L2229z/fXX\n07RpUwYMGMDbb79Ny5YtL+viUVFRpKamkpaWRkFBAWvXrq2yG7/URBxDxQ4Eb29Ut36o24ZjGno/\nnDmN/uFz9O7/UfjqUxS++QK6oMDRoQohqoDSWuuauNC0adPYuXMnWVlZ+Pv7M3ToUPr168fmzZuZ\nN28ehmHQt29fbr311iq/9pEjRyp1nDw7rRz91xmoXcfWuG7MeRO9fgVoDX7+kHUKdd1gTLffZ9f5\n5OfgHNyhDOAe5XCmNpFSH2dNmjSJZ599FoDnn3++1N429k4FP27cuBK3R0dHEx0tCx25E1Wn+Pgh\nNfhe9PFjqPbXoPrfiP58LvqnpeioVqjo4h029F9Z1sWwhBAuodQk0qdPH9trV5zaJDExkU2bNvHQ\nQw85OpQrngq04PH0lPMbbr8f/edejA/fwtSoCSo4FMA6oeOnszG9/K5tmxDCuZWaRHr27Gl73bBh\nQ5o3v3Q6i6SkpOqJqgpIw7rzUl5emB76P4yXxmHMfgPTPyfDiXT0F/OgsBD9+2ZU30GODlMIYQe7\nGtZfeeWVErdPmjSpSoOpStKw7txUUAjq7tGwbzf628UYH80AZYJ6AehdWx0dnhDCTmV28TUMAwCt\nte2/IseOHcPjMgaQVTepiTg/U+feGL9vQn/7CQDqzgcheT96yzq0UYgyOe/vlxDCqswkUjRCHOCO\nO+4o9pnJZGLw4MHVE5W4Yqg7H0Lv+wP8A1CxA9EbV8OaBEjeD41lRmAhnF2ZSeSdd95Ba82LL75Y\nrBeWUop69erJeuvisqlatTG98BaYPKw1j6vaowG9axtKkogQTq/MNpHg4GDq16/PtGnTCAwMJDg4\nmODgYIKCgjCZTE49m660ibgO5e2D8rT+PaPqBULDSPSubQDowkIK09PKOlwI4UB2NaxPmjSJ/fv3\nF9u2f/9+p25Yj4mJke69Lkpd1QGSdqH/OoMx/SXSR92GTjlg+1zv243x/WeOC1AIYWNXEjl48OAl\nXXybNWvGwYMHqyUocWVTrTpAfh7Gq0/Crq3g6YXxxYcA6JxsjPdeRy9dgM7NdWygQgj7JmCsU6cO\np06dIiAgwLbt1KlT+PjICnaiGrRoAyYTHD+Gum8cdQrzOTPvHfTOrehtG+DEuekeMo5BWIRjYxXi\nCmdXTaRLly689dZbJCcnk5ubS3JyMu+8845TL1MrbSKuS9WqjbrzIUyPPIupW19qD7oNLPUx5r+D\n/uU7aHpu4s/0Y44NVAhhX03kjjvuYP78+TzzzDPk5+fj7e1NbGxssS7AzkbGibg2U+xA22vl7YMa\nfA961lQIsGAa8TjGxFHW+bgcF6IQAjuTiLe3NyNHjuT+++8nKysLPz+/Kl3+VIjyqGt6wZFDqHad\noH4D8PaB9KOODkuIK55dSaRITk4OOTk5nD171rbNUasbiiuLMplQg+8+vyEoBF3C4yxtFKLX/Iy6\npifKt3YNRijElcmuJJKSksL06dNL7I21ePHiKg9KiHIFh8LxEmoi/0tEz38HjiSjho2s+biEuMLY\n1bA+a9Ys2rRpw5w5c6hduzZz587l2muv5ZFHHqnu+CpNGtbdmwoKgfQ0Ll5TTf/2q/X/K75HZxx3\nRGhCXFHsHify97//nTp16qC1pnbt2tx9991OXQuRwYZuLigEcs/CmdO2TfpstrULcMcugEJ/s9Bx\n8QlxhbAriXh5eVFYWAiAn58f6enpaK05c+ZMtQYnRGlsi1Zd8EhLb1kH+XmYrr8VFTsIvfYXdOoh\nB0UoxJXBrjaRVq1asW7dOmJjY+natSuvvvoqXl5etGnTprrjE6JkQdYkotOPoc6NG9G//WqtoUS1\nQoWEoVf/hPH6BLhouV3V61pM199a4yEL4Y7sSiJPPPGE7fWdd95Jo0aNyMnJoXfv3tUWmBBlCqpv\n/f+5mog+dQJ2/Q81aIi1+7mfP+resbBlXbHDdPJ+9LKl6GtvlvVKhKgC5SYRwzB46aWXePbZZ/Hy\n8sJkMknyEA6nfHyhXoBt1LreuBK0geoSa9vHdE1PuKZnseOMDSvRH/wH9v0BzVtbt635GdUg3Faj\nEULYr9w2EZPJRFrapb1ghHC44FB0+jG0YaB//RGatEA1CC/zENUuBjw80VvXA6CPH0XPm47x2dya\niFgIt2NXw/qQIUP44IMPOH78OIZhFPvPWUkXX/enLCHWx1k7tsDRw6j+fyv/mFq1rQtfbVlvXfJ5\n+begNSSYcx87AAAgAElEQVTtRJ/IqIGohXAvdrWJFN2MV65ceclnztrNV+bOugIEh8DGVRjLlkCA\nGdWpu12HqY5d0R/NgH270av/C1GtrK83rUHF3VTNQQvhXuxKIu+88051xyFExQWHgjbgj+2oW+5G\neXrZdZjq2AX98bsYc96EnLOY7ngAY9476MTVUEoSKXpkptp1sg50LIXOyUYv/w517c0oL1k+Wri/\nMpPIyZMnCQgIIDg4uKbiEcJuKigEDeDljeozwP7j/AOt08nv2w3NrkI1bo6K6YH+8iN05nGUuYTf\n962/oRfORDeMxPTMf1DeJa+lo3/6Cv3NIlRoQ4i2r2YkhCsrs03kscceK/b+P//5T7UGI0SFnBtw\nqLr1RdWtV6FDVccuAJjO1TxUjLUXl9609pJ9tdYYP3wOdevB4YPoTz4o8Zw656y1jQXQB/eXuI8Q\n7qbMmsjFPbJ27NhRrcEIURHKHIwa+SSqzdUVPzZ2EPgFwNVdre9DwiCiKXrdcnRIGCiTtZZSqzbs\n/h8c2Iu6+2HISEP/8DlGy3aYuvQpdk696if4Kwtq10En76uSMgrh7MpMIrJmiHB2F9/I7aV8a6F6\n9C++rXMf9OdzMd5+2bohOBTTqPHWWoi/GdW9H5g80Ht3oudOw8g6her/N5RS6Px89E9LoWU7lKU+\nensiWmv5NyTcXplJpLCwkN9//9323jCMYu8B2rZtWz2RCVHDVNxNqKvaQ6EBp09gfPQuxuSnoKAA\nNeQ+W0O5aexEjDnT0ItnoffuRLXpCCkH4GQmpvseQ6cehrU/w8lMCLSgC/LJ3bIeGjVzbAGFqAZl\nJhF/f3/effdd2/u6desWe6+Ukp5bwm0oDw+IiLK9NzVpgTFrKqSmoPpcf36/2nUxPfyMdfqULxeg\nN59rR4lqBVd1RHn7Whv8k/dZk8iK7zm5eDamF6ajwhvXaJmEqG5lJpH4+PiaisMuO3bsYPHixYSH\nh9OjRw+ZAFJUK1UvAI8nXkYX5F/SfViZTKiBt6H7DIC8HOvGuvWsj7YaNQFlQh9MQnXojN64GgCd\nckCSiHA7FVoe93LMmDGDzZs34+/vz9SpU23bt27dyty5czEMg/79+3PLLbeUeg6lFL6+vuTn52Ox\nWGoibCHKHH+iateB2nWKb/PxhdCG1skeM9Jg/x/WD44kV2eYQjhEjSWR2NhYBgwYUKx2YxgGs2fP\nZuLEiVgsFiZMmEBMTAyGYbBwYfEFhUaPHk2rVq145plnOHnyJPPnz+fRRx+tqfCFqBAVGYXe/T/0\npjXW937+aEkiwg3VWBJp3bo1aWlpxbYlJSURGhpKSIh1BHD37t3ZuHEjgwcPZvz48aWeq27duuTn\n51drvEJclogoWL8CveIHiGyGd1gjcpN2OToqIapcjSWRkmRmZhZ7LGWxWNi7d2+p+//2229s27aN\nv/76iwEDSh+hnJCQQEJCAgBTpkwhKCioUvF5enpW+lhnIWVwjLz20Zz4dDYcP0rdex9B5eeSu34F\nFr+61sddLsgVfw4lcYdyOFMZHJpEKqpLly506dKl3P3i4uKIi4uzvU9PT6/U9YKCgip9rLOQMjiG\nrnf+j6Psqzril3EMtCb9922oyKji+xYWgslU7pgSnZ+P8rJvfrDqUNGfgzYKQZVfrprmir9PF6uJ\nMoSFhdm1n11TwVcXs9lMRsb56bczMjIwm81Vcm6ZCl44kqpVG0LDrWucBIXgGdEE4JJ2EV1YiDF+\nJPqnL8s8n979P4xHh6G3b6q2mKuStVwPWB/nCbfm0CQSFRVFamoqaWlpFBQUsHbt2iqbvj0mJoaH\nHnqoSs4lRGWYRj2NaeSTAHg0aAQenpf20Dp6GE5moFf/t9SF3/TpE9bxKgUF6HXLqzvsqnE0BU6k\nw4HSH08L91Bjj7OmTZvGzp07ycrKYtSoUQwdOpR+/foxYsQIJk2ahGEY9O3bl0aNGlXJ9RITE9m0\naZMkEuEwqmHk+deentZuvxfXRA4mWV8cTYHDB+GicSTaKMSY9QZk/wVXdUBv24DOzUX5lDyLsLPQ\nB61zh+mMtHL2FK6uxpLIuHHjStweHR1NdHR0lV9PFqUSzkaFRaD/3FN8Y/I+8PSCwkJ04upLBiPq\nZUth1zbUvWNQwaEYUyfC74nQqUfNBV4ZRRNQph9zbByi2jn0cVZ1kjYR4XTCGkH6MXRujm2TTt4H\nkVHQqh06cU2xR1r6+FH0N59AdHdUz2uhRRuoF2AbAe/MbLMYn0i3dhwQbsttk4i0iQhno8IirC9S\nDwHW1RJJ/hMV0RQV0wOOHbZO5Mi5NUwWvQ8mD0x3PIBSCmXyQHXqjt6+EZ1z1kGlKF9RuahVGwwD\nTsra9e7MbZOIEE7nXBKxtYukHYHcsxDZDHV1NzCZrEv0AmxZB9sTUbfchQo8311YxfSEvDz09sSa\njt5+58ql2l1jfS/tIm7NpcaJVIQ0rAunE9wAPD3P1zbONT6riCiUnz+0ao9e/i2FO7ZAWiqEN0H1\nvbH4OZpdBf5m9IZVcE0v22Zj6UfoHZvLj0EpTIPvQbXuWFWluoStXNHd0Bt+RaenoVpU2+WEg7lt\nTUQeZwlnozw8oFUH9PoV6LxcSN5vbVRvYO2RaLphKLRoC/UC4Kr2mEY+YT3mwnOYPFBdY2HbBnTa\nEQB0agr6h8/AKLQeW9Z/KX+it22o3oIWlatoxUmpibg1t62JCOGMTANuw/jPM+g1CdbG5/DG1u6/\ngGrRFo8W5S/ypuJuQv/8DfrHJah7x6CXfQFeXpjG/QtVL6DMYwufexh9MrNKylIafTDJWi7fWhBg\nhgzpoeXO3LYmIr2zhFNq0QaiWqF/XALJ+1ARUeUfcxEVYEb16G9dD37fbvT6Faie15WbQADwD4RT\n1ZdEtNaQvP98uSz10elSE3FnbptE5HGWcEZKKUwDb4fM49YBhJEVTyIA6vpbwTAwpv/L+v66wfYd\n52+GUycqdU27pB+Ds+fLpSwh1rIKt+W2SUQIp9U+Bs6NZr94MkZ7qeBQ1DW9IPsvVOc+KEuwfQcG\nWJNIaVOsXEzv3Ire+pv9gZ0bgW8rlyUYMo9bJ2MUbkmSiBA1TCmFach90Kq9LZlU6jw3DrNO8HjD\nUPsP8g+E/DxrbcEOxjeLML6YZ/fp9c6t4FMLws6VK6g+FBZCNbfDCMdx24Z16eIrnJlqG41H28ub\n7keFhuPxzH8qdpB/oPX/p05A7brl75+eBtlZdtVcdEEBevM6VMfOtinrlSUEXXQes521JeFS3DaJ\nyNxZQlxKBZitN/WTmbauxaXRBfnWRnitIesUBJeTBHb/D/7Ksg6ILGKpbz1XZhqKNpcXvHBK8jhL\niCvJuZqItqeHVma6NYEAZJTfOK4TV1unOikaHwLWNhGw1kSEW5IkIsSVxP/com/29NC6cJBgOWM9\ndEE+est6VIcuKC9v23bl5W1NXDLg0G1JEhHiSuJbC3x84WT5SURfMI17ueuC7NoG2WeKP8oqYqkv\n64q4MbdtE5GGdSEupZQqdcCh/ivL2mU4ONS6ISMNlAl8fYs9jtIZx8HLE1Uv8Py2jauhVh0oYU4u\nZamPLmOFQ/3HdvjrjPVNoybnrw/otFTbXGPFNG+D8qt3fr/UQxAYZB0l70K0UQiHDlS6q7czcNsk\nIg3rQpTCPxBdwuMsvXg2+o//4fHaHOuGjDQItEBdv2I1CSP+FQiw4PHo89bjtEZvT0R1uMbWK6uY\noBDYvBZdkI/yLP65TvkT4z/Pnt8QEYXHc2+ev9a7k0tMIqpHf9Twx6znyMvFeOVxVNzNqMH32Pst\nOAW9YRV69huYXopHldPRwVm5bRIRQpRM+ZvRyfsv2a73/wGZ6eiTGagAizVxBNWH2n7W6d3BuqBW\nykHrYlNaW2s2mcfhzGmIalXyBRs1tY4VOZIMF03zovdbV3o0PfYCestv6DX/Refnoby8rdc6nIzq\nOwjV63rbMcbiWeg/L6jZpBywTo+//4/L+2IcYf9uwPo9uGoSkTYRIa40/oGXNKzrs9nWRbEADp5L\nMBlpKHN9VFB9yEizjhU59CdoA85kWXtvARRN/R7ZrMTLqcim1muc26+Y5H3Wx2BtolFtOlqTTcpB\n62fnrqXaRKMaNTn/X/M2kJpiWyHStopi8j67R+I7C1syTy7hu3ERkkSEuNL4myH3bPHVEQ/9aXup\nk/dZx4icyLTWRCz1ITcHnXXq/A0bbDc+nbwPTKbSR98HhVq7/pZwo9TJ+yGiqbVGc66Woi88L1xS\ne1GRTa2JrOgxV9GNOPsvl1rT3doeYo1dSxIRQriMgEu7+epk65xX+Plbp3I/kWG9UVvO1USAwrRU\na62jjp91FcZz82Tpg/ugQSOUt0+Jl1MmE0REXVIT0QUFcOjPC+bZqm8dRX/uvBzcB37+5+MtcnGy\nOZhk3Q9c6y/61MOQl2eNPXm/y84vJklEiCuMsk19ckEPrYP7IcBsXfEweb/tL3plqQ/m80lEJ++D\nJi2gQSN08n7r46ODSaU+yrJdM6IppBxAF15wozx6CArybUlBKQWRUbZHPDp5n3XpYKWKnywwyHrj\nPbgPnZ9vbTfp0gc8PEp+ZOakipKg6t4f8nLh2BEHR1Q5kkSEuNKcG3Coi9VE9llv5hFR1kbzA+dq\nA0Eh1kdaQMHhZDiSbF3ON6Kp9a/+U5nWKVHKWxclIso68WPqofPXPNf2cmH3VhURBYcPWNtozl3r\nYtZHX02tMR9JhsICaNoKwiJc67FQ8j7w9kZ17g2U0mbkAtw2iciiVEKUIqB4TUTn5kBqijU5nLuh\n6y3rQCkItKBq14VadcjduBoMw9omERFlnVL+fxuB843npSmqqRS7yR9Mss74Wz/s/LbIKCgoQG9c\nef5aJZ0vIgqOJKP37bJdX0VEWWsnLtK4bl0Bsom1Lcnb29ZBwdW4bRKRRamEKEXtutY10IumZ085\nYO0FFRll7Y4L8OceCLCcH9dhCaZg707r68hm55PCyp+syaZR2UmEkAbWkfIXdC221n6aWNtMzrEl\nsZU/2a5VEhXZDAoL0WuXW3t3BTew7nvmNJxIt/+7cBBtGJBsbQ9SHh4Q3sS1alEXcNskIoQo2flR\n69bHWRf2glK160D9Btb352bgLfa6jp91SvdGja3J42AShIajfHzLvqbJAxo1sT2ysfZM+vPSx1VF\nPbkOJp2/VkkiziWtg0m23l2qaJsr3IzTUiH37Pn2oIgoOLTfmlxcjCQRIa5EF45aL+rdFGgBsN3Y\ni3plWV+HWF8U3bB9a0NI2Ln9y6mFFJ3DdqMshKOHrY3JF3ffNZnO12qKuv6WJCjEth6K7frhTUCZ\nXKJtoahnmy2JRjSFs9lw/KgDo6ocSSJCXIkCzLbHWfrgfoiMOn/DLmroNl9aE7mwF5aKOPe6nJ5Z\nNpFRkJsDx1LP90wq4diiR1pl9fgq6sl14fWVjw+ENXKJJELyfvD0hLAIoJQ2Ixch054IcQVS/oHo\nrb9R+PwjcPQwqv35eeZURJR14aoLayKWYOu2C2sOkU1hw68l9qAq8ZrnzmtMe8HaU8vbG0IbXrqj\n7a/zss+rIpqid20rdn0V0RS9cZW1XKVI9/SksKDArpirzYl0aNgY5XnuFhzWCDw90Yvep/CbT8o9\n3N4ymEaMQzVufpnBlk2SiBBXINWtP2SdRmsDFd4Y1a3v+Q+bt0FdNxjVscv5ba07UvumO8hpd35J\nX9W5D5w6CVEt7btoWASq/9/QJzOsxze7ytqofHFsHTpD3M2odmUvH6x6Xgde3rbHagAqdhDk56N1\n6W0Lnt4+FObl2hdzdQlrhKlzH9tb5emFGnyP3fN/2V2GUgaAViWlXaU/3GU4cqRyg3iCgoJIT3f+\nnh5lkTI4BymD83CHctREGcLCwsrfCReriRiGweLFizl79ixNmzYlNjbW0SEJIcQVrcaSyIwZM9i8\neTP+/v5MnTrVtn3r1q3MnTsXwzDo378/t9xyS6nnSExMJCMjAz8/PywWS02ELYQQogw1lkRiY2MZ\nMGAA8fHxtm2GYTB79mwmTpyIxWJhwoQJxMTEYBgGCxcuLHb86NGjOXLkCC1btuTaa69l6tSptGvX\nrqbCF0IIUYIaSyKtW7cmLa34OstJSUmEhoYSEmLtg969e3c2btzI4MGDGT9+/CXnMJvNeJ7rzVBq\n/3EhhBA1xqFtIpmZmcUeS1ksFvbuLX0t5i5dujBnzhx2795N69atS90vISGBhIQEAKZMmUJQUFCl\n4vP09Kz0sc5CyuAcpAzOwx3K4UxlcKmGdR8fH0aPHl3ufnFxccTFxdneV7YXg/TicA5SBufgDmUA\n9yiHM/XOcuiIdbPZTEZGhu19RkYGZrO5jCPsJ7P4CiFE9XNoEomKiiI1NZW0tDQKCgpYu3YtMTEx\n5R9oB5nFVwghql+NDTacNm0aO3fuJCsrC39/f4YOHUq/fv3YvHkz8+bNwzAM+vbty6233lol10tM\nTGTTpk2SSIQQojppUaqnn37a0SFcNimDc5AyOA93KIczlUFm8RVCCFFpkkSEEEJUmseLL774oqOD\ncGZNm9q34I4zkzI4BymD83CHcjhLGa6IWXyFEEJUD3mcJYQQotJcasR6TarI7MLOIj09nfj4eE6e\nPIlSiri4OAYNGsSZM2d48803OX78OMHBwTz++OPUrVvX0eGWyjAMxo8fj9lsZvz48S4XP8Bff/3F\nzJkzOXToEEopRo8eTVhYmEuV49tvv2X58uUopWjUqBEPP/wweXl5Tl2GkmYLL+v3Z+nSpSxfvhyT\nycR9991Hx44dHRk+UHIZFixYwKZNm/D09CQkJISHH36YOnXqAE5QBkd3D3NGhYWFesyYMfro0aM6\nPz9fP/XUU/rQoUOODqtcmZmZet++fVprrbOzs/Wjjz6qDx06pBcsWKCXLl2qtdZ66dKlesGCBY4M\ns1zffPONnjZtmp48ebLWWrtc/Fpr/fbbb+uEhASttdb5+fn6zJkzLlWOjIwM/fDDD+vc3FyttdZT\np07Vv/zyi9OXYceOHXrfvn36iSeesG0rLeZDhw7pp556Sufl5eljx47pMWPG6MLCQofEfaGSyrB1\n61ZdUFCgtbaWx5nKII+zSnDh7MKenp622YWdXWBgoK2xrVatWjRs2JDMzEw2btxInz7WpTj79Onj\n1GXJyMhg8+bN9O/f37bNleIHyM7OZteuXfTr1w+wTpZXp04dlyuHYRjk5eVRWFhIXl4egYGBTl+G\n1q1bX1IzKi3mjRs30r17d7y8vKhfvz6hoaEkJSXVeMwXK6kMHTp0wOPcUsItWrQgMzMTcI4yyOOs\nElR0dmFnlJaWxp9//kmzZs04deoUgYGBAAQEBHDq1CkHR1e6Dz/8kLvvvpuzZ8/atrlS/GD97uvV\nq8eMGTM4ePAgTZs2Zfjw4S5VDrPZzN/+9jdGjx6Nt7c3HTp0oEOHDi5VhiKlxZyZmUnz5s1t+5nN\nZtvN2ZktX76c7t27A85RBqmJuKGcnBymTp3K8OHDqV27drHPlFJOuxbLpk2b8Pf3L7ProjPHX6Sw\nsJA///yT6667jtdffx0fHx++/PLLYvs4eznOnDnDxo0biY+P57333iMnJ4eVK1cW28fZy1ASV4z5\nQkuWLMHDw4NevXo5OhQbqYmUoDpnF65uBQUFTJ06lV69etGlSxcA/P39OXHiBIGBgZw4cYJ69eo5\nOMqS/fHHHyQmJrJlyxby8vI4e/Ys06dPd5n4i1gsFiwWi+0vxK5du/Lll1+6VDm2b99O/fr1bTF2\n6dKFPXv2uFQZipQW88X/zjMzM5363/mKFSvYtGkTzz//vC0ROkMZpCZSguqcXbg6aa2ZOXMmDRs2\n5MYbb7Rtj4mJ4ddffwXg119/5ZprrnFUiGW66667mDlzJvHx8YwbN462bdvy6KOPukz8RQICArBY\nLBw5cgSw3pDDw8NdqhxBQUHs3buX3NxctNZs376dhg0bulQZipQWc0xMDGvXriU/P5+0tDRSU1Np\n1qyZI0Mt1datW/nqq694+umn8fHxsW13hjLIYMNSVNfswtVp9+7dPP/880RERNj+Urnzzjtp3rw5\nb775Junp6U7ZLbMkO3bs4JtvvmH8+PFkZWW5XPwHDhxg5syZFBQUUL9+fR5++GG01i5Vjk8//ZS1\na9fi4eFB48aNGTVqFDk5OU5dhpJmC7/mmmtKjXnJkiX88ssvmEwmhg8fztVXX+3gEpRchqVLl1JQ\nUGCLu3nz5jz44IOA48sgSUQIIUSlyeMsIYQQlSZJRAghRKVJEhFCCFFpkkSEEEJUmiQRIYQQlSZJ\nRNSIFStW8Nxzzznk2vHx8XzyyScVOua5557jzz//rKaISvfpp58yffr0Gr9ukU8++YT777+fBx54\nwGExpKWlMXToUAoLC6v83O+//z6ff/55lZ/3hx9+4KOPPqry87oCSSIu4sUXX+Tnn392dBhOryqS\nVWJiIr6+vjRp0qSKonIN6enpfPPNN7zxxht88MEHl3y+Y8cORo0a5YDIqs6DDz7IkCFD7Nq3In98\n9O/fn9WrV7vEXGJVTZLIFaw6/tJzB//973/p3bu3o8O4bBX9+aanp+Pn54e/v381ReS+vL296dix\no21k/JVE5s5yQQkJCXz11VecOXOGVq1a8cADD2A2m0lLS2PMmDEsWrTINm30iy++SK9evejfvz8r\nVqzg559/JioqipUrV3LdddcRGhrKzz//TPPmzfnll1+oXbs2I0eOtI16zc7OZt68eWzZsgWlFH37\n9mXo0KEYhsEDDzzAv/71LyIiIgDrbKmPPPIIM2bMKHdOpcOHDzNnzhz2799PvXr1GDZsmG1m0vj4\neHx8fDh+/Di7du0iPDycRx99lNDQUAC2bdvGnDlzOHnyJL169eLQoUP07t2bli1b8sEHH1BQUMA9\n99yDh4cHH374IWCdUHDy5Mklnu9CBQUF/P7777bRwGB9xJSSkoK3tzcbNmwgKCiIRx55hKioKACG\nDh3K9OnTbeeLj4/HYrFwxx13sGPHDt5++20GDhzIN998g8lkYuTIkXh6ejJv3jxOnz7N3/72t2Iz\nIuTn5/Pmm2+yZcsWGjRowOjRo2ncuDFgnRtpzpw57Nq1C19fX2644QYGDRpki/PQoUN4eXmxadMm\n7r333mJT6hf9POfMmcOWLVvw8fGhf//+DB48mN9//53XXnvN9t117dqVRx55xHZcTk4Or776qu1z\ngLfeegs/Pz8+/vhj1q1bB0C3bt34+9//jpeXl+337eWXX7ad58LvKi8vj08++YT169fz119/ERER\nUawWuWrVKhYvXkxeXh433HCD7TsqKqfJZCrxO0pJSWHWrFkcOHAAs9nMXXfdZZu2qKSfzQ033MBX\nX32FyWTizjvvpG/fviQkJLB69WoAvvvuO9q0acP48eP58ssv+eGHHzh79iyBgYGMHDmSdu3aAdCm\nTRuWL1/OTTfdVNqvvVuSmoiL+f3331m0aBGPP/4477//PsHBwbz11lt2H793715CQkL44IMPbP8o\nk5KSCAsLY/bs2dx8883MnDmTookM4uPj8fDwYPr06bz++uts27aNn3/+GU9PT3r06FFsZtc1a9bQ\ntm3bchNITk4Or7zyCj179mTWrFmMGzeO2bNnk5KSYttn7dq13H777cydO5fQ0FDbY4XTp0/zxhtv\ncNdddzFnzhzCwsLYs2cPAOHh4TzwwAO0aNGCBQsW2BJIWee7WGpqKiaTqdhSAGCdYbh79+58+OGH\nxMTEMGfOHDu/cTh58iT5+fnMnDmToUOH8t5777Fq1SqmTJnCSy+9xBdffEFaWppt/8TERLp168ac\nOXPo0aMH//73vykoKMAwDF577TUaN27Me++9x/PPP8/333/P1q1bix3btWtX5s6dW+JMr3PmzCE7\nO5t33nmHF198kZUrV7JixQrat2/PM888Q2BgIAsWLCiWQAB8fX2Lfb5gwQLMZjNLlixh7969vP76\n6/z73/8mKSmJL774wq7vZf78+ezfv59XXnmFuXPncvfddxebYXf37t289dZbPPfcc3z++efFfj9K\n+44KCgp47bXXaN++PbNmzWLEiBFMnz7dNo9ZST+b7OxsZs6cyahRo5g9ezZnzpwhLi6Onj17cvPN\nN7NgwQLGjx/PkSNHWLZsGZMnT2b+/Pk8++yzBAcH287VsGFDDhw4YFfZ3YkkERezatUq+vbtS9Om\nTfHy8uKuu+5iz549xW5CZQkMDGTgwIF4eHjg7e0NWCfbi4uLw2Qy0adPH06cOMGpU6c4efIkW7Zs\nYfjw4fj6+uLv788NN9zA2rVrAesCP2vWrLElnJUrV9r1GGjz5s0EBwfTt29fPDw8aNKkCV26dLH9\nNQvQuXNnmjVrhoeHBz179rT949yyZQvh4eF06dIFDw8PBg4cSEBAQLnXLO18F/vrr7/w9fW9ZHur\nVq2Ijo7GZDLRu3fvCt0sPDw8uPXWW22JNysri0GDBlGrVi0aNWpEeHh4sfM1bdqUrl274unpyY03\n3kh+fj579+5l3759nD59miFDhtiWSe3fv7/t5wHWBYs6d+6MyWSy/XyLGIbBmjVruOuuu6hVqxb1\n69fnxhtvvGSK94pYvXo1t912G/7+/tSrV48hQ4awatWqco8zDINffvmF4cOHYzabMZlMtGzZEi8v\nL9s+t99+O97e3jRu3JjIyEgOHjxY7ne0d+9ecnJyuOWWW/D09KRt27ZER0fbahUX8/DwsH2f0dHR\n+Pr6lppwTCYT+fn5pKSk2OZEu7A2W6tWLbKzs+396tyGPM5yMSdOnCjW4Ovr60vdunXtngI6KCjo\nkm0X3oSLZgjNycnhzJkzFBYWFnu0o7W2/ZXevHlzfHx82LFjB4GBgRw9etSu2Y6PHz/O3r17GT58\nuG1bYWFhsQR0cUw5OTmAtfwX1hKUUnaVu7TzXaxu3bolfnZhO4G3tzf5+fkUFhbaHhuWxc/PD5PJ\nZLrvC3AAAAWESURBVDu2pPNdeM0Ly1dUKzpx4gRgLf+F35thGFx11VUlHnux06dPU1hYWOx3IDg4\n+LIWMcrMzCz217i958vKyiI/P7/ER4pFyvqZlfUdBQUF2b7v8mLy8/Mr9jMs63cjNDSU4cOH89ln\nn5GSkkKHDh249957bb9/Z8+evWT9niuBJBEXExgYSHp6uu190c3ebDbb/oLOzc21/TKfPHmy0tey\nWCx4enoye/bsUm+Wffr0YdWqVQQEBNC1a9dL/vot7bytW7euVC+qgICAYjcErXWVruQWGhpqO6e9\n6zL4+PiQm5tre3/y5Mkyb+bluXB9CMMwyMjIIDAwEA8PD+rXr1/pLsD16tXDw8OD9PR0wsPDAWtj\nur3lLGkxJ7PZzPHjx2nUqNEl5/Px8SEvL8+274W/i35+fnh5eXH06FFbW0ZFlPYdFcVgGIYtkaSn\np9OgQYMKX6Ok8vbs2ZOePXuSnZ3N+++/z8cff8zYsWMBaztfZcri6uRxlovp0aMHv/zyCwcOHCA/\nP59FixbRrFkz2wJCZrOZVatWYRgGy5cv59ixY5W+VmBgIB06dGD+/PlkZ2djGAZHjx5l586dtn16\n9erFhg0bWLVqld09mjp16kRqaiorV660PcdOSkoq9sy7NNHR0SQnJ7NhwwYKCwtZtmxZsZtTUZIp\nKCioeIGxroferl27YmUsT+PGjVm9ejWGYbB169YKHVuS/fv389tvv1FYWMj333+Pl5cXzZs3p1mz\nZtSqVYsvv/ySvLw8DMMgOTnZ7jW1TSYT3bp1Y9GiRZw9e5bjx4/z7bff2r1Knr+/P1lZWcUe2fTo\n0YMlS5Zw+vRpTp8+zeeff247X2RkJIcOHeLAgQPk5eXx6aefFoulb9++zJ8/n8zMTAzDYM+ePeTn\n51/Wd1RUO/76668pKChgx44dbNq0iR49eth13ovLe+G/nyNHjvD777+Tn5+Pt7c33t7exRLNzp07\n6dixY4Wv4+qkJuJClFK0b9+eYcOGMXXqVM6cOUPLli0ZN26cbZ+HHnqIWbNmsWjRIvr160eLFi0u\n65pjxozh448/5oknnuDs2bOEhIRw88032z4PCgqiadOmHD16tNhjlbLUqlWLiRMnMm/ePObNm4fW\nmsjISP7xj3+Ue2y9evV44oknmDt3LvHx8fTq1cvWPgTQtm1bWwO7yWRi9uzZFS7ztddey48//kjP\nnj3t2n/48OHEx8ezbNkyrrnmmsteqKlooaH4+HhCQ0N58skn8fS0/lN9+umnmT9/Po888ggFBQWE\nhYUxbNgwu889YsQI5syZw5gxY/D29qZ///707dvXrmMbNmxIjx49GDNmDIZh8MYbb3DrrbeSnZ3N\nU089BVhXcSzqsBEWFsaQIUN4+eWX8fb25s477yQhIcF2vnvvvZeFCxcyYcIEcnJyaNy4Mc8++2yV\nfEezZs1i6dKlmM1mxowZQ8OGDe3+jor069ePN954g+HDh9O6dWuGDRvGxx9/zOHDh/Hw8KBly5a2\nR715eXls2bKFKVOmVPg6rk7WE3ERTz/9NLfddhudO3d2dCiXmDFjBmazmTvuuKPGr20YBqNHj2bs\n2LG0bdu2ys773HPPMWLEiCtuwKEr+PTTTzl69CiPPvqoo0Ox+eGHH8jIyODuu+92dCg1TmoiLuDQ\noUOkpKQ45Q0tLS2NDRs28Prrr9fYNbdu3Urz5s3x9vbm66+/Rmt92TWui104tkGI8gwcONDRITiM\nJBEn99FHH7Fq1Sr+/ve/F+sF4ww++eQTvvvuOwYPHkz9+vVr7Lp79uxh+vTpFBQUEB4ezj//+U+7\nGvSFEFVPHmcJIYSoNOmdJYT4//bqWAAAAABgkL/1HHaXRLBJBIBNIgBsEgFgkwgAm0QA2AIB2eIY\nYfkYogAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x15b5214a8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Data exploration\n",
    "\n",
    "def journey_lenght_histogram(df):\n",
    "    counts = df.groupby(['jid'])['uid'].count().reset_index(name=\"count\").groupby(['count']).count()\n",
    "    return counts.index, counts.values / df.shape[0]\n",
    "\n",
    "hist_x, hist_y = journey_lenght_histogram(df4)\n",
    "\n",
    "plt.plot(range(len(hist_all)), hist_all, label='all journeys')\n",
    "plt.yscale('log')\n",
    "plt.xlim(0, 120)\n",
    "plt.xlabel('Journey length (number of touchpoints)')\n",
    "plt.ylabel('Fraction of journeys')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Last Touch Attribution\n",
    "\n",
    "In this section, we develop a baseline last-touch attribution model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {},
   "outputs": [],
   "source": [
    "def last_touch_attribution(df):\n",
    "    \n",
    "    def count_by_campaign(df):\n",
    "        counters = np.zeros(n_campaigns)\n",
    "        for campaign_one_hot in df['campaigns'].values:\n",
    "            campaign_id = np.argmax(campaign_one_hot)\n",
    "            counters[campaign_id] = counters[campaign_id] + 1\n",
    "        return counters\n",
    "        \n",
    "    campaign_impressions = count_by_campaign(df)\n",
    "    \n",
    "    df_converted = df[df['conversion'] == 1]\n",
    "    idx = df_converted.groupby(['jid'])['timestamp_norm'].transform(max) == df_converted['timestamp_norm']\n",
    "    campaign_conversions = count_by_campaign(df_converted[idx])\n",
    "        \n",
    "    return campaign_conversions / campaign_impressions\n",
    "    \n",
    "lta = last_touch_attribution(df6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4UAAAENCAYAAAC4rMA8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XtwVGWe//FPp9vQhEBCd0NiIMgQcRjwAiFcjIhiYnZ0\nXSpDqeCVDKKyXBfdKckUDiqXySxREEEQiUHK0c1qzQirDhsyXlCjRRQChaAkijjIJaRzIWACJN2/\nPyz7Z08InFxOdzr9flVR9jnnefr5Pt2Pnf72ec5zLF6v1ysAAAAAQFiKCHYAAAAAAIDgISkEAAAA\ngDBGUggAAAAAYYykEAAAAADCGEkhAAAAAIQxkkIAAAAACGMkhQAAAAAQxgKWFJaWlmrevHmaM2eO\n3nzzzRbLlZeXa8qUKfr0009bXRcAAAAA0DoBSQo9Ho/y8vL0+9//XitWrNDHH3+sw4cPn7fcn//8\nZ11zzTWtrgsAAAAAaL2AJIXl5eWKj49XXFycbDabUlNTVVJS0qzc3/72N40ZM0a9evVqdV0AAAAA\nQOvZAtFIVVWVnE6nb9vpdKqsrKxZmR07dmjRokVau3Ztq+r+pKioSEVFRZKknJwcnT17tiO7YSqb\nzabGxsZgh4EujDEGMzG+YDbGGMzGGIPZgjHGIiMjDZULSFJoxMaNG3XPPfcoIqLtJy/T09OVnp7u\n266srOyI0ALC5XKFVLwIPYwxmInxBbMxxmA2xhjMFowxlpCQYKhcQJJCh8Mht9vt23a73XI4HH5l\nvv76az377LOSpJMnT2rXrl2KiIgwVBcAAAAA0DYBSQqTkpJ09OhRVVRUyOFwqLi4WHPnzvUrs2bN\nGr/HI0eO1OjRo9XU1HTRugAAAACAtglIUmi1WjVt2jQtXbpUHo9HEyZMUGJiogoLCyVJGRkZra7b\nFl6vVw0NDfJ4PLJYLG16DrMcP35cZ86caVUdr9eriIgI2e32TtcfAAAAAKHB4vV6vcEOwixHjhzx\n266vr9cll1wim63TXErp09YLTxsbG3Xu3Dl1797dhKjQlXCtBMzE+ILZGGMwG2MMZuvM1xQG7Ob1\nnYHH4+mUCWF72Gw2eTyeYIcBAAAAIESFVVLYVadYdtV+AQAAADBfWCWFAAAAAAB/XWsuZSs1PTix\nQ5/P+uKWi5YZPHiwysrKfNsrVqzQO++8I4vFov3792vIkCGSpHvuuUdZWVmSpJtuuknDhg3Tc889\n16HxAgAAAEBYJ4Wdwfz58zV//nxJ0q9+9Stt27bN7/j+/ftltVpVXFys+vp6FpQBEFQt/Zh2vIXy\nRn4sAwAAwcX00U7uzTff1O23367U1NRmCSMAAAAAtBdJYSf31ltvaeLEicrMzNTmzZuDHQ4AAACA\nLoaksBP7/PPPFR8fr0svvVTjx49XaWmpTp48GeywAAAAAHQhJIWd2ObNm/Xll19qzJgxGjdunOrq\n6vS3v/0t2GEBAAAA6EJYaKaT8ng8evvtt/X++++rT58+kqTt27dr7dq1mjx5cpCjAwAAANBVhHVS\nGIxV8err6zVy5Ejf9kMPPaSHH364WbmPP/5YiYmJvoRQklJTUzVnzhydOHHCbz8AAAAAtFVYJ4XB\ncPjw4fPut9ls2r9/v2/7+uuv1/XXX9+szO7du02NDwAAAEB44ZpCAAAAAAhjJIUAAAAAEMbCKin0\ner3BDsEUXbVfAAAAAMwXVklhRESEGhsbgx1Gh2psbFRERFi9jQAAAAA6UFgtNGO329XQ0KAzZ87I\nYrEEOxw/3bp105kzZ1pVx+v1KiIiQna73aSoAAAAAHR1YZUUWiwWde/ePdhhnJfL5VJlZWWwwwAA\nAAAQZgKWFJaWlio/P18ej0dpaWnKzMz0O15SUqKCggJZLBZZrVZlZWVpyJAhkqRZs2bJbrcrIiJC\nVqtVOTk5gQobAAAAALq0gCSFHo9HeXl5WrhwoZxOp7Kzs5WSkqL+/fv7ylx11VVKSUmRxWLRoUOH\ntGLFCq1cudJ3fNGiRerVq1cgwgUAAACAsBGQFUrKy8sVHx+vuLg42Ww2paamqqSkxK+M3W73XefX\nGa/5AwAAAICuKCBnCquqquR0On3bTqdTZWVlzcrt2LFDr776qmpra5Wdne13bPHixYqIiNDNN9+s\n9PT087ZTVFSkoqIiSVJOTo5cLlcH9sJcNpstpOJF6GGMoSMcb2V5xhw6Cp9hMBtjDGbrzGOsUy00\nM3r0aI0ePVr79u1TQUGBHn/8cUk/JoQOh0O1tbVasmSJEhISNHTo0Gb109PT/RLGUFq4hYVmYDbG\nGIKBMYeOwmcYzMYYg9mCMcYSEhIMlQvI9FGHwyG32+3bdrvdcjgcLZYfOnSojh8/rpMnT/rqS1JM\nTIxGjRql8vJycwMGAAAAgDARkKQwKSlJR48eVUVFhRobG1VcXKyUlBS/MseOHZPX65UkffPNNzp3\n7px69uyphoYG1dfXS5IaGhq0Z88eDRgwIBBhAwAAAECXF5Dpo1arVdOmTdPSpUvl8Xg0YcIEJSYm\nqrCwUJKUkZGhTz/9VNu3b5fValVkZKTmz58vi8Wi2tpa5ebmSpKampo0btw4DR8+PBBhAwAAAECX\nZ/H+dHquCzpy5EiwQzCMeewwG2MMHaHpwYmtKm99cYtJkSDc8BkGszHGYLawv6YQAAAAANA5kRQC\nAAAAQBgjKQQAAACAMEZSCAAAAABhjKQQAAAAAMIYSSEAAAAAhDGSQgAAAAAIYySFAAAAABDGbMEO\nAACAzqzpwYmtrmN9cYsJkQAAYA7OFAIAAABAGDN8pnD37t369ttv1dDQ4Ld/8uTJHR4UAAAAACAw\nDCWFeXl5+uSTTzRs2DB169bN7JgAAEAIa+2UW6bbAkBwGUoKP/roIy1fvlwul8vseAAAAIKCZBZA\nuDJ0TWGvXr3Uo0cPs2MBAAAAAASYoTOFt912m1atWqXf/OY3iomJ8TsWFxdnSmDoWPz6CQAAAOB8\nDCWFGzZskCTt3Lmz2bGCgoKOjQgAAAAAEDCGkkISPwAAAADomlp18/rKykpVVVXJ4XCw6AwAAAAA\ndAGGksLq6mqtXLlSBw4cUM+ePVVXV6crrrhC8+bNk8PhMNRQaWmp8vPz5fF4lJaWpszMTL/jJSUl\nKigokMVikdVqVVZWloYMGWKoLgAAAACgbQwlhS+++KIuu+wyZWdny263q6GhQa+99ppefPFFPfbY\nYxet7/F4lJeXp4ULF8rpdCo7O1spKSnq37+/r8xVV12llJQUWSwWHTp0SCtWrNDKlSsN1QUAAAAA\ntI2hW1J89dVXuv/++2W32yVJdrtd9957rw4cOGCokfLycsXHxysuLk42m02pqakqKSnxK2O322Wx\nWCRJZ86c8T02UhcAAAAA0DaGzhT26NFDhw8f1sCBA337jhw5oqioKEONVFVVyel0+radTqfKysqa\nlduxY4deffVV1dbWKjs7u1V1JamoqEhFRUWSpJycnJC67tFms5ka7/FWlg+l1w7GmD3GEB7C8bOk\ntX2Wuka/28OMcRKIz7BwHN/4//g7CbN15jFmKCmcOHGiFi9erJtuukl9+vTRiRMn9P7772vy5Mkd\nGszo0aM1evRo7du3TwUFBXr88cdbVT89PV3p6em+7crKyg6Nz0wul6tTxduZYkHH6GxjDOEhXMdc\nuPa7rYy8Xp3xM6yzxYP26YxjDF1LMMZYQkKCoXKGpo+mp6dr/vz5qqur0+eff666ujrNnTvXLwG7\nEIfDIbfb7dt2u90XXKBm6NChOn78uE6ePNnqugAAAAAA4wzfkuLKK6/UlVde2aZGkpKSdPToUVVU\nVMjhcKi4uFhz5871K3Ps2DHFxcXJYrHom2++0blz59SzZ0/16NHjonUBAAAAAG3TYlL4l7/8RZMm\nTZJ04ZvXG5lCarVaNW3aNC1dulQej0cTJkxQYmKiCgsLJUkZGRn69NNPtX37dlmtVkVGRmr+/Pm+\n21Ocry4AAKGg6cGJrSpvfXGLSZEAAHB+LSaF/zxls72Sk5OVnJzsty8jI8P3ODMzs8X7D56vLgAA\nAACg/VpMCh988EHf45kzZwYkGAAAAABAYBm6pvDw4cOKjo5WbGysGhoatGXLFlksFk2cOFHdunUz\nO0YAAAAAgEkMrT767LPP6ocffpAkbdq0Sfv371dZWZnWr19vanAAAAAAAHMZOlNYUVGhhIQEeb1e\n7dixQ88884wiIyM1e/Zss+MDAACASVgICYBkMCmMjIxUfX29Dh8+LJfLpV69eqmpqUnnzp0zOz4A\nAAAAgIkMJYXXXXednnrqKdXX1+vXv/61JOngwYPq27evqcEBAAAAAMxlKCnMysrS7t27ZbVafTew\nt1gsmjp1qqnBAcHElBoAAACEA0NJoSRdc801vsfHjx9Xz549lZSUZEpQAAAAAIDAMLT66MqVK/XV\nV19Jkt577z098sgjevTRR/Xuu++aGhwAAAAAwFyGksK9e/f6zgq+9dZbevzxx7Vs2TK9+eabpgYH\nAAAAADCXoemjjY2Nstlsqqqq0qlTpzRkyBBJUm1tranBAQAAAADMZSgpHDhwoP7617/qxIkTSk5O\nliRVVVWpe/fupgYHAAAAADCXoemjM2bM0HfffaezZ89qypQpkqQDBw5o3LhxpgYHAAAAADCXoTOF\n8fHxmjdvnt++sWPHauzYsaYEBQAAALSE20YBHctQUuj1evX3v/9dxcXFOnnypHJzc7Vv3z7V1NQo\nNTXV7BgBAAAAACYxNH20oKBA7733ntLS0lRZWSlJcjqd2rx5s6nBAQAAAADMZSgp/OCDD/TYY4/p\nuuuuk8VikST17dtXFRUVpgYHAAAAADCXoemjHo9Hdrvdb19DQ0OzfQB+xLUOAAAACBWGksLhw4dr\n06ZNmjp1qqQfrzEsKCjQyJEjDTdUWlqq/Px8eTwepaWlKTMz0+/4hx9+qM2bN8vr9ap79+6aPn26\nBg4cKEmaNWuW7Ha7IiIiZLValZOTY7jdroREAwAAAEBHM5QUTp06VWvWrFFWVpYaGxt1//336+qr\nr9bs2bMNNeLxeJSXl6eFCxfK6XQqOztbKSkp6t+/v69M37599cQTTyg6Olq7du3S+vXrtWzZMt/x\nRYsWqVevXq3sHgAAAADgQi6aFHq9XtXV1emRRx7RqVOndOLECblcLsXGxhpupLy8XPHx8YqLi5Mk\npaamqqSkxC8p/OUvf+l7PHjwYLnd7tb0AwACjrP3AACgK7joQjMWi0X/+Z//KYvFopiYGF1++eWt\nSgglqaqqSk6n07ftdDpVVVXVYvl3331XI0aM8Nu3ePFiPfbYYyoqKmpV2wAAAACAlhmaPjpw4EAd\nPXpU/fr1Mzse7d27V++9956eeuop377FixfL4XCotrZWS5YsUUJCgoYOHdqsblFRkS9pzMnJkcvl\nMj3ejmKz2S4a7/FWPufPn689dcNVV3u9jYwxtE5nfJ/NRp+N6eyfB2Yzo8+B+Azjvbq4ztLnUB1j\nCG+deYwZSgqHDRumZcuW6YYbbmjWkZtuuumi9R0Oh990ULfbLYfD0azcoUOH9MILLyg7O1s9e/b0\nqy9JMTExGjVqlMrLy8+bFKanpys9Pd23/dM9FUOBy+Xq8Hjb83yh9Np1Fp399TZjjKF1wvH1D8c+\nS53/86CzMdLnzvgZ1tniCYRQ7XOojjF0LcEYYwkJCYbKGUoKv/rqK/Xt21f79+9vdsxIUpiUlKSj\nR4+qoqJCDodDxcXFmjt3rl+ZyspK5ebmavbs2X7BNzQ0+FYkbWho0J49e3T77bcbCRsAAAAAcBGG\nksJFixa1qxGr1app06Zp6dKl8ng8mjBhghITE1VYWChJysjI0BtvvKFTp05pw4YNvjo5OTmqra1V\nbm6uJKmpqUnjxo3T8OHD2xUPAAAAAOBHhpJCSTp9+rR27typ6upq9e7dW8nJyerRo4fhhpKTk5Wc\nnOy3LyMjw/d4xowZmjFjRrN6cXFxWr58ueF2AABA+7R2ZV2J1XXDESswA12HoaRw7969ys3NVUJC\nglwul9xut/Ly8vToo4/qqquuMjtGAEAXQKIBAOGFz/3QYSgpzMvL00MPPaTU1FTfvk8++UR5eXla\nuXKlacEBAAAAAMx10fsUSlJ1dbXGjh3rt2/06NGqqakxJSgAAAAAQGAYSgrHjx+vrVu3+u0rLCzU\n+PHjTQkKAAAAABAYhqaPHjx4UNu2bdOWLVvkcDhUVVWl2tpaDR482G9l0ieffNK0QAEAADorFl0B\nEMoMJYVpaWlKS0szOxYAQADw5RUA2obPT3RVhpLCG2+80eQwAAAAAADBYPg+hfv379fBgwfV0NDg\nt3/SpEkdHhQAAB2JX/cBAGiZoaTwpZde0ieffKIhQ4YoMjLSt99isZgWGDoP7jEDAMFBMgsACARD\nSeGHH36op59+Wg6Hw+x4AAAAYBA/3ALoCIZuSeFyuXTJJZeYHQsAAAAAIMAMnSmcMWOGXnjhBV13\n3XWKiYnxOzZ06FBTAgMAAAA6WktnV4+3UJ4zqwgHhpLCb775Rrt27dL+/fv9rimUpLVr15oSGAAA\nAADAfIaSwtdee02PPfaYrr76arPjAQAAAAAEkKGksFu3bkwTBQAAnRqLrgBA2xhaaGby5MnauHGj\nampq5PF4/P4BAAAAAEKXoTOFP103uG3btmbHCgoKOjYiAAAAAEDAGEoKV69ebXYcAAAAAIAgMJQU\n9unTp90NlZaWKj8/Xx6PR2lpacrMzPQ7/uGHH2rz5s3yer3q3r27pk+froEDBxqqC3Qlrb0mhuth\nAAAA/PF9qnVaTApfeOEFPfzww5Kk5557ThaL5bzlZs+efdFGPB6P8vLytHDhQjmdTmVnZyslJUX9\n+/f3lenbt6+eeOIJRUdHa9euXVq/fr2WLVtmqC4AoOvjDzwAAOZoMSns27ev73F8fHy7GikvL1d8\nfLzi4uIkSampqSopKfFL7H75y1/6Hg8ePFhut9twXQDhjWQBAACg7VpMCn/zm9/4Ht9xxx3taqSq\nqkpOp9O37XQ6VVZW1mL5d999VyNGjGhTXQAAAKCr4QdQmMnQNYWBtHfvXr333nt66qmnWl23qKhI\nRUVFkqScnBy5XK6ODs80NpvtovEeb+Vz/vz5Aln3n+uHqlB5vY2+1kbGWKhq1+v9m9RW1Y37a3GH\ntBtMoTK2g9l2V4i7PYLZ55aY8Xfyn9vmvQpc3fYKlc+DztLnYAnmd8jO+Hp15u9iAUkKHQ6Hbzqo\nJLndbjkcjmblDh06pBdeeEHZ2dnq2bNnq+pKUnp6utLT033blZWVHdUF07lcrg6Ptz3P195YQum1\n7yjBer2N1jVjjIWqzv5edTah+llC3IFldtxmfYbxXoVG3faiz6EjWHEHot1gfBdLSEgwVC4gSWFS\nUpKOHj2qiooKORwOFRcXa+7cuX5lKisrlZubq9mzZ/sFb6QuAIQbphEBAICOctGk0OPx6PXXX9ek\nSZN0ySWXtKkRq9WqadOmaenSpfJ4PJowYYISExNVWFgoScrIyNAbb7yhU6dOacOGDb46OTk5LdYF\nAAAAALTfRZPCiIgIFRYWtnuxmeTkZCUnJ/vty8jI8D2eMWOGZsyYYbguAAAAgK6LWTGBE2Gk0Pjx\n47Vt2zazYwEAAAAABJihawrLy8u1detWbdmyRU6n0+9G9k8++aRpwaFr4FceAAAAoPMylBSmpaUp\nLS3N7FgAAAAAAAFmKCm88cYbTQ4DAAAAAIKvtbPcpNCf6WYoKfR6vfr73/+ujz/+WHV1dcrNzdW+\nfftUU1Oj1NTW3fgZgHku9CHW0k1cQ/1DDAAAAO1jaKGZgoICvffee0pPT/fdcNHpdGrz5s2mBgcA\nAAAAMJehM4UffPCB/vSnP6lXr16++wj27dtXFRUVpgYHsEgNAABA8PBdLDwYSgo9Ho/sdrvfvoaG\nhmb7AABA58AXOQCAUYamj44YMUKbNm3SuXPnJP14jWFBQYFGjhxpanAAAAAAAHMZOlN4//33a82a\nNcrKylJjY6Puv/9+XX311Zo9e7bZ8QEAAABoB2YO4GIMJYVRUVH63e9+p9raWp04cUIul0uxsbFm\nxwYAAAAAMJmhpFCSTp8+rT179qi6ulq9e/fWiBEjFB0dbWZsAAAAAACTGUoK9+7dq9zcXCUkJMjl\ncsntdisvL0+PPvqorrrqKrNjBBACmJrSOuF4Y1wAANA5GUoK8/Ly9NBDD/ndqP6TTz5RXl6eVq5c\naVpwAAAAAABzGVp9tLq6WmPHjvXbN3r0aNXU1JgSFAAAAAAgMAwlhePHj9fWrVv99hUWFmr8+PGm\nBAUAAAAACAxD00cPHjyobdu2acuWLXI4HKqqqlJtba0GDx6sRYsW+co9+eSTpgUKAAAAAOh4hpLC\ntLQ0paWlmR0LgDDFIjUAAADBYygpvPHGG9vdUGlpqfLz8+XxeJSWlqbMzEy/499//72ef/55HTx4\nUFOmTNHEif//S+KsWbNkt9sVEREhq9WqnJycdscDAACA4ODHQKBzMXyfwvbweDzKy8vTwoUL5XQ6\nlZ2drZSUFPXv399XJjo6Wr/97W9VUlJy3udYtGiRevXqFYhwAQAAACBsGFpopr3Ky8sVHx+vuLg4\n2Ww2paamNkv+YmJidPnll8tqtQYiJAAAAACAAnSmsKqqSk6n07ftdDpVVlbWqudYvHixIiIidPPN\nNys9Pb2jQwQAAACAsBSQpLC9Fi9eLIfDodraWi1ZskQJCQkaOnRos3JFRUUqKiqSJOXk5MjlcgU6\n1Daz2WwXjfd4K5/z588XyLrBbLsj3/NQeb078r1qj2C+z6HyeneF/6/oc+duuyv0uSVm/J3857ZD\n8e9VqPY5HD/32ysU+xwu4/Of67fEyOdYsBhKChsbG/X+++/r22+/VUNDg9+x2bNnX7S+w+GQ2+32\nbbvdbjkcDsNB/lQ2JiZGo0aNUnl5+XmTwvT0dL+ziJWVlYbbCDaXy9Xh8bbn+dobS7DaDuZ7Hqp9\nDtZrFqp9Ju7A1Q1m2+EYd2fvsxl/J4223dnqtlco9rmzj08z6rZXqPa5K8dt1ufYhSQkJBgqZ+ia\nwtWrV+vtt9+W3W5XXFyc3z8jkpKSdPToUVVUVKixsVHFxcVKSUkxVLehoUH19fW+x3v27NGAAQMM\n1QUAAAAAXJihM4W7d+/W6tWr1aNHjzY1YrVaNW3aNC1dulQej0cTJkxQYmKiCgsLJUkZGRmqqanR\nggULVF9fL4vFonfeeUfPPPOM6urqlJubK0lqamrSuHHjNHz48DbFAQAAAADwZygpdLlcOnfuXLsa\nSk5OVnJyst++jIwM3+PY2FitW7euWb2oqCgtX768XW0DAAAAAM7PUFI4fvx4LV++XLfccotiY2P9\njl155ZWmBAYAAMJPSzc1b2nhB25qDgDtZygp3Lp1qyTptdde89tvsVi0evXqjo8KAAAA6EJa+sHj\nQvjRA4FiKCl87rnnFBERkPvcAwAAAAAC6KKZnsfj0X333dfuawoBAAAAAJ3PRZPCiIgIJSQkqK6u\nLhDxAAAAAAACyND00XHjxulPf/qTbrnlFjmdTlksFt8xFpoBAAAAgNBlKCn86X6Cr7/+ut9+FpoB\nAAAAgNBmKClcs2aN2XEAAAAAAIKAJUUBAAAAIIwZOlP47//+7y0eW7t2bYcFAwAAAAAILENJ4Zw5\nc/y2q6ur9c477+i6664zJaiurKUblx5voTw3LQUAAABgJkNJ4dChQ5vtGzZsmJYuXapbb721w4MC\nEBwt/WjREn60AAAACH1tvqbQZrOpoqKiI2MBAAAAAASYoTOFBQUFfttnzpzRrl27NGLECFOCAgAA\nAAAEhqGk0O12+21369ZNt912m8aPH29KUAAAAACAwDCUFN59992KjY1ttr+mpua8+wEAAAAAocHQ\nNYXz5s077/758+d3aDAAAAAAgMAydKbQ6/U22/fDDz8oIqLN69QAAABArPwMIPgumBT+dNP6s2fP\nNruB/alTp1p1n8LS0lLl5+fL4/EoLS1NmZmZfse///57Pf/88zp48KCmTJmiiRMnGq4LAAAAAGib\nCyaFc+bMkdfr1R//+MdmN7CPjY1VQkKCoUY8Ho/y8vK0cOFCOZ1OZWdnKyUlRf379/eViY6O1m9/\n+1uVlJS0ui4AAAAAoG0umBT+dNP6vLw8devWrc2NlJeXKz4+XnFxcZKk1NRUlZSU+CV2MTExiomJ\n0c6dO1tdFwAAAADQNoYuCoyIiNBrr72m2bNna+rUqZKk3bt3a+vWrYYaqaqqktPp9G07nU5VVVWZ\nXhcAAAAAcGGGFprZuHGjqqurNXfuXC1btkySlJiYqJdfflm//vWvTQ2wNYqKilRUVCRJysnJkcvl\nCnJEzR1vZfmf9yFU6gaz7Y58z0Pl9e4K71V73+dQjDsc36tw6XMw26bPrReKcYd7n4PZNu+VuXWD\n2XZHxt0Sm83WKfMTyWBSWFJSolWrVslut8tisUiSHA6H4TN2DodDbrfbt+12u+VwODq8bnp6utLT\n033blZWVhtrozNrTh2DVDWbbwXzPQ7XPoRh3OPY5mG3T59Bpmz6HTtv0OXB1g9l2OMYdjn02Wt/l\ncgX8u6rRNWAMTR+12WzyeDx++06ePKmePXsaaiQpKUlHjx5VRUWFGhsbVVxcrJSUFNPrAgAAAAAu\nzNCZwrFjx2r16tXKysqSJFVXV2vjxo1KTU011IjVatW0adO0dOlSeTweTZgwQYmJiSosLJQkZWRk\nqKamRgsWLFB9fb0sFoveeecdPfPMM4qKijpvXQAAAABA+xlKCu+++2698sorevTRR3X27FnNnTtX\naWlpuv322w03lJycrOTkZL99GRkZvsexsbFat26d4boAAAAAgPYzlBTabDZlZWUpKyvLN230p2sL\nAQAAAAChy9A1hT/Xq1cvWSwWfffdd3rmmWfMiAkAAAAAECAXPFN45swZ/fWvf9W3336rSy+9VHfc\ncYfq6uq0adMm7dmzRzfccEOg4gQAAAAAmOCCSWFeXp4OHjyoa665RqWlpfruu+905MgR3XDDDXr4\n4YfVq1fGWcKDAAAQP0lEQVSvQMUJAAAAADDBBZPC3bt367/+678UExOjW265RTNnztQTTzyhX/3q\nV4GKDwAAAABgogteU9jQ0KCYmBhJktPplN1uJyEEAAAAgC7kgmcKm5qatHfvXr99/7x95ZVXdnxU\nAAAAAICAuGBSGBMTo7Vr1/q2o6Oj/bYtFotWr15tXnQAAAAAAFNdMClcs2ZNoOIAAAAAAASBoZvX\nA6Go6cGJra5jfXGLCZEAAAAAnVerb14PAAAAAOg6SAoBAAAAIIyRFAIAAABAGCMpBAAAAIAwRlII\nAAAAAGGMpBAAAAAAwhhJIQAAAACEMZJCAAAAAAhjAbt5fWlpqfLz8+XxeJSWlqbMzEy/416vV/n5\n+dq1a5e6deummTNnatCgQZKkWbNmyW63KyIiQlarVTk5OYEKGwAAAAC6tIAkhR6PR3l5eVq4cKGc\nTqeys7OVkpKi/v37+8rs2rVLx44d06pVq1RWVqYNGzZo2bJlvuOLFi1Sr169AhEuAAAAAISNgEwf\nLS8vV3x8vOLi4mSz2ZSamqqSkhK/Mp999pnGjx8vi8WiK664QqdPn1Z1dXUgwgMAAACAsBWQM4VV\nVVVyOp2+bafTqbKysmZlXC6XX5mqqir17t1bkrR48WJFRETo5ptvVnp6eiDCBgAAAIAuL2DXFLbH\n4sWL5XA4VFtbqyVLlighIUFDhw5tVq6oqEhFRUWSpJycHL8ks7M43sryP+9DqNQNZtvhGHe49zmY\nbfNemVs3mG2HY9zh2Odgtk2f2143mG3zXplbN5htd2TcLbHZbJ0yP5EClBQ6HA653W7fttvtlsPh\naFamsrLyvGV++m9MTIxGjRql8vLy8yaF6enpfmcRf/58oao9fQhW3WC2HY5x0+fQaTsc4w7HPgez\nbfocOm3T58DVDWbb4Rh3OPbZaH2XyxXw/CQhIcFQuYBcU5iUlKSjR4+qoqJCjY2NKi4uVkpKil+Z\nlJQUbd++XV6vVwcOHFBUVJR69+6thoYG1dfXS5IaGhq0Z88eDRgwIBBhAwAAAECXF5AzhVarVdOm\nTdPSpUvl8Xg0YcIEJSYmqrCwUJKUkZGhESNGaOfOnZo7d64iIyM1c+ZMSVJtba1yc3MlSU1NTRo3\nbpyGDx8eiLABAAAAoMsL2DWFycnJSk5O9tuXkZHhe2yxWDR9+vRm9eLi4rR8+XLT4wMAAACAcBSQ\n6aMAAAAAgM6JpBAAAAAAwhhJIQAAAACEMZJCAAAAAAhjJIUAAAAAEMZICgEAAAAgjJEUAgAAAEAY\nIykEAAAAgDBGUggAAAAAYYykEAAAAADCGEkhAAAAAIQxkkIAAAAACGMkhQAAAAAQxkgKAQAAACCM\nkRQCAAAAQBgjKQQAAACAMEZSCAAAAABhjKQQAAAAAMKYLVANlZaWKj8/Xx6PR2lpacrMzPQ77vV6\nlZ+fr127dqlbt26aOXOmBg0aZKguAAAAAKBtAnKm0OPxKC8vT7///e+1YsUKffzxxzp8+LBfmV27\ndunYsWNatWqVHnroIW3YsMFwXQAAAABA2wQkKSwvL1d8fLzi4uJks9mUmpqqkpISvzKfffaZxo8f\nL4vFoiuuuEKnT59WdXW1oboAAAAAgLYJSFJYVVUlp9Pp23Y6naqqqmpWxuVyNStjpC4AAAAAoG0s\nXq/Xa3Yjn376qUpLSzVjxgxJ0vbt21VWVqYHHnjAVyYnJ0eZmZkaMmSIJOmpp57SPffcoxMnTly0\n7k+KiopUVFTkez4AAAAAwIUF5Eyhw+GQ2+32bbvdbjkcjmZlKisrm5UxUvcn6enpysnJCcmEcMGC\nBcEOAV0cYwxmYnzBbIwxmI0xBrN15jEWkKQwKSlJR48eVUVFhRobG1VcXKyUlBS/MikpKdq+fbu8\nXq8OHDigqKgo9e7d21BdAAAAAEDbBOSWFFarVdOmTdPSpUvl8Xg0YcIEJSYmqrCwUJKUkZGhESNG\naOfOnZo7d64iIyM1c+bMC9YFAAAAALRfwO5TmJycrOTkZL99GRkZvscWi0XTp083XLerSU9PD3YI\n6OIYYzAT4wtmY4zBbIwxmK0zj7GALDQDAAAAAOicAnJNIQAAAACgcwrY9FGcX2lpqfLz8+XxeJSW\nlqbMzMxgh4QQ9/zzz2vnzp2KiYnR008/LUk6deqUVqxYoRMnTqhPnz6aP3++oqOjgxwpQlVlZaXW\nrFmjmpoaWSwWpaen69Zbb2WcoUOcPXtWixYtUmNjo5qamjR27FjdeeedjC90OI/HowULFsjhcGjB\nggWMMXSoWbNmyW63KyIiQlarVTk5OZ16jDF9NIg8Ho/mzZunhQsXyul0Kjs7W/PmzVP//v2DHRpC\n2L59+2S327VmzRpfUvjKK68oOjpamZmZevPNN3Xq1Cnde++9QY4Uoaq6ulrV1dUaNGiQ6uvrtWDB\nAv3ud7/T+++/zzhDu3m9Xp05c0Z2u12NjY36wx/+oKysLO3YsYPxhQ711ltv6euvv/Z9jvG3Eh1p\n1qxZ+uMf/6hevXr59nXmMcb00SAqLy9XfHy84uLiZLPZlJqaqpKSkmCHhRA3dOjQZr86lZSU6IYb\nbpAk3XDDDYwztEvv3r01aNAgSVL37t3Vr18/VVVVMc7QISwWi+x2uySpqalJTU1NslgsjC90KLfb\nrZ07dyotLc23jzEGs3XmMcb00SCqqqqS0+n0bTudTpWVlQUxInRVtbW16t27tyQpNjZWtbW1QY4I\nXUVFRYUOHjyoyy+/nHGGDuPxePTYY4/p2LFj+pd/+RcNHjyY8YUOtXHjRt17772qr6/37WOMoaMt\nXrxYERERuvnmm5Went6pxxhJIRBmLBaLLBZLsMNAF9DQ0KCnn35aWVlZioqK8jvGOEN7REREaPny\n5Tp9+rRyc3P13Xff+R1nfKE9Pv/8c8XExGjQoEH64osvzluGMYb2Wrx4sRwOh2pra7VkyRIlJCT4\nHe9sY4ykMIgcDofcbrdv2+12y+FwBDEidFUxMTGqrq5W7969VV1d7Te/HWiLxsZGPf3007r++us1\nZswYSYwzdLwePXpo2LBhKi0tZXyhw3z11Vf67LPPtGvXLp09e1b19fVatWoVYwwd6qfv9DExMRo1\napTKy8s79RjjmsIgSkpK0tGjR1VRUaHGxkYVFxcrJSUl2GGhC0pJSdEHH3wgSfrggw80atSoIEeE\nUOb1erVu3Tr169dPt912m28/4wwd4eTJkzp9+rSkH1ci3bNnj/r168f4Qoe5++67tW7dOq1Zs0b/\n8R//oSuvvFJz585ljKHDNDQ0+KYmNzQ0aM+ePRowYECnHmOsPhpkO3fu1MsvvyyPx6MJEyZo0qRJ\nwQ4JIW7lypXat2+f6urqFBMTozvvvFOjRo3SihUrVFlZ2emWQEbo+fLLL/WHP/xBAwYM8E19ueuu\nuzR48GDGGdrt0KFDWrNmjTwej7xer6699lrdfvvtqqurY3yhw33xxRf63//9Xy1YsIAxhg5z/Phx\n5ebmSvpxwaxx48Zp0qRJnXqMkRQCAAAAQBhj+igAAAAAhDGSQgAAAAAIYySFAAAAABDGSAoBAAAA\nIIyRFAIAAABAGCMpBAAgACorK3XffffJ4/EEOxQAAPxwSwoAQMj76KOP9NZbb+n7779X9+7dNXDg\nQE2aNElDhgwJdmgBsWbNGjmdTk2ZMkUVFRWaPXu2unXrJkmy2+1KSkrSrbfeqquvvjrIkQIAOiNb\nsAMAAKA93nrrLb355pt68MEHdc0118hms2n37t367LPPwiYpPJ+NGzfKarWqpqZGxcXFWr58uR54\n4AHdeOONwQ4NANDJkBQCAELWDz/8oIKCAs2cOVNjxozx7R85cqRGjhwpSSovL1d+fr6+//57RUZG\nasyYMZo6dapsth//BN5555164IEH9Pbbb6umpka33nqrbrzxRq1evVr/+Mc/dM0112ju3Lmy2Wz6\n4osv9NxzzykjI0Nvv/227Ha7pkyZouuvv16StHPnTv33f/+3jh8/rqioKE2YMEF33nmnJPnO4L32\n2muyWq2qqKjQmjVrdPDgQQ0ePFiXXnqpfvjhB82dO9dXdubMmSooKNDZs2f1r//6r5o0aVKrX6PY\n2Fjdeuutamxs1J///GeNHz9eERFcPQIA+P/4qwAACFkHDhzQuXPnNHr06BbLREREaOrUqcrLy9OS\nJUu0d+9e/d///Z9fmd27dysnJ0dLly7Vli1btH79es2ZM0dr167VP/7xD3300Ue+sjU1Naqrq9O6\ndes0a9YsrV+/XkeOHJEkdevWTbNnz1Z+fr4WLFigbdu2aceOHeeN69lnn1VSUpJeeukl3XHHHfrw\nww+blfnyyy/17LPP6vHHH9cbb7yhw4cPt+VlkiSNGTNGtbW1vlgBAPgJSSEAIGTV1dWpZ8+eslqt\nLZYZNGiQrrjiClmtVvXt21fp6enat2+fX5mJEycqKipKiYmJSkxM1NVXX624uDhFRUVpxIgR+vbb\nb/3KT548WZdccomGDh2qESNGqLi4WJI0bNgwDRgwQBEREbrssst03XXXNWtL+nHRma+//lqTJ0+W\nzWbTkCFDfGc2f+6OO+5QZGSkBg4cqMsuu0yHDh1qw6v0o969e0uSTp061ebnAAB0TUwfBQCErJ49\ne6qurk5NTU0tJoZHjhzRpk2b9PXXX+vs2bNqamrSoEGD/MrExsb6HkdGRjbbrqmp8W336NFDdrvd\nt92nTx9VV1dLksrKyvTqq6/qu+++U2NjoxobGzV27NhmMVVVVSk6Otq3GIwkuVwuVVZWthhXt27d\n1NDQcMHX40KqqqokSdHR0W1+DgBA18SZQgBAyLriiit0ySWXqKSkpMUyGzZsUEJCglatWqWXX35Z\nd911l9qz8Pbp06f9krPKykrfWbhVq1Zp5MiRWrt2rV5++WXdfPPN522rd+/eOnXqlM6cOeP3PGba\nsWOHYmJilJCQYGo7AIDQQ1IIAAhZUVFRuvPOO5WXl6cdO3bozJkzamxs1K5du/TKK69Ikurr6xUV\nFSW73a7vv/9ehYWF7W73f/7nf9TY2Kj9+/dr586duvbaa31tRUdHKzIyUuXl5X7XIv5cnz59lJSU\npNdff12NjY06cOCAPv/883bHdT41NTXaunWr3njjDd11110sMgMAaIbpowCAkPZv//Zvio2N1V/+\n8hc999xzstvtGjRokG+lzvvuu0/r16/X5s2b9Ytf/EKpqanau3dvm9uLjY1VdHS0Hn74YUVGRurB\nBx9Uv379JEnTp0/Xpk2b9NJLL2no0KG69tprdfr06fM+z5w5c/T8889r2rRpuvzyy5WamtqhN7bP\nysqS9OO006SkJD3yyCMaPnx4hz0/AKDr4Ob1AAAY9NMtKdatW9fhz71ixQr169fPdwsLAAAChTkk\nAAAEQXl5uY4dOyaPx6PS0lJ99tlnGjVqVLDDAgCEIaaPAgAQBDU1NXr66adVV1cnp9Op6dOn6xe/\n+EWwwwIAhCGmjwIAAABAGGP6KAAAAACEMZJCAAAAAAhjJIUAAAAAEMZICgEAAAAgjJEUAgAAAEAY\nIykEAAAAgDD2/wCK41Ckrsd5YQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20f770a20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualization of the attribution scores\n",
    "\n",
    "campaign_idx = range(150, 200)\n",
    "\n",
    "fig = plt.figure(figsize=(15,4))\n",
    "ax = fig.add_subplot(111)\n",
    "plt.bar( range(len(lta[campaign_idx])), lta[campaign_idx], label='LTA' )\n",
    "plt.xlabel('Campaign ID')\n",
    "plt.ylabel('Return per impression')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "\n",
    "In this section, we create a basic logistic regression (linear) models and use its coefficients as attribution weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "def features_for_logistic_regression(df):\n",
    "\n",
    "    def pairwise_max(series):\n",
    "        return np.max(series.tolist(), axis = 0).tolist()\n",
    "    \n",
    "    aggregation = {\n",
    "        'campaigns': pairwise_max,\n",
    "        'cats': pairwise_max,\n",
    "        'click': 'sum',\n",
    "        'cost': 'sum',\n",
    "        'conversion': 'max'\n",
    "    }\n",
    "    \n",
    "    df_agg = df.groupby(['jid']).agg(aggregation)\n",
    "    \n",
    "    df_agg['features'] = df_agg[['campaigns', 'cats', 'click', 'cost']].values.tolist()\n",
    "    \n",
    "    return (\n",
    "        np.stack(df_agg['features'].map(lambda x: np.hstack(x)).values),\n",
    "        df_agg['conversion'].values\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(145440, 1561)\n"
     ]
    }
   ],
   "source": [
    "x, y = features_for_logistic_regression(df6)\n",
    "print(np.shape(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.20, random_state = 1)\n",
    "x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size = 0.20, random_state = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8653396589658966\n"
     ]
    }
   ],
   "source": [
    "# Quick sanity check\n",
    "logisticRegr = LogisticRegression()\n",
    "logisticRegr.fit(x_train, y_train)\n",
    "score = logisticRegr.score(x_test, y_test)\n",
    "print(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 93081 samples, validate on 23271 samples\n",
      "Epoch 1/10\n",
      "93081/93081 [==============================] - 3s 30us/step - loss: 0.5277 - acc: 0.7693 - val_loss: 0.4710 - val_acc: 0.8031\n",
      "Epoch 2/10\n",
      "93081/93081 [==============================] - 1s 12us/step - loss: 0.4442 - acc: 0.8172 - val_loss: 0.4363 - val_acc: 0.8172\n",
      "Epoch 3/10\n",
      "93081/93081 [==============================] - 1s 12us/step - loss: 0.4183 - acc: 0.8293 - val_loss: 0.4180 - val_acc: 0.8290\n",
      "Epoch 4/10\n",
      "93081/93081 [==============================] - 1s 13us/step - loss: 0.4032 - acc: 0.8367 - val_loss: 0.4066 - val_acc: 0.8343\n",
      "Epoch 5/10\n",
      "93081/93081 [==============================] - 1s 14us/step - loss: 0.3931 - acc: 0.8416 - val_loss: 0.3984 - val_acc: 0.8383\n",
      "Epoch 6/10\n",
      "93081/93081 [==============================] - 1s 13us/step - loss: 0.3853 - acc: 0.8443 - val_loss: 0.3931 - val_acc: 0.8410\n",
      "Epoch 7/10\n",
      "93081/93081 [==============================] - 1s 13us/step - loss: 0.3795 - acc: 0.8471 - val_loss: 0.3875 - val_acc: 0.8432\n",
      "Epoch 8/10\n",
      "93081/93081 [==============================] - 1s 13us/step - loss: 0.3747 - acc: 0.8489 - val_loss: 0.3830 - val_acc: 0.8444\n",
      "Epoch 9/10\n",
      "93081/93081 [==============================] - 1s 13us/step - loss: 0.3706 - acc: 0.8500 - val_loss: 0.3799 - val_acc: 0.8448\n",
      "Epoch 10/10\n",
      "93081/93081 [==============================] - 1s 14us/step - loss: 0.3674 - acc: 0.8511 - val_loss: 0.3772 - val_acc: 0.8474\n",
      "Test score: 0.3732171668471283\n",
      "Test accuracy: 0.8457783278327833\n"
     ]
    }
   ],
   "source": [
    "m = np.shape(x)[1]\n",
    "    \n",
    "model = Sequential()  \n",
    "model.add(Dense(1, input_dim=m, activation='sigmoid', name = 'contributions')) \n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy']) \n",
    "history = model.fit(x_train, y_train, batch_size=128, epochs=10, verbose=1, validation_data=(x_val, y_val)) \n",
    "score = model.evaluate(x_test, y_test, verbose=0) \n",
    "print('Test score:', score[0]) \n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4sAAAENCAYAAACmZfCPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X9YlXWe//HXAQJC5Mc5KA5GOZBerloGohkU6sCwk9t6\nOV6l2U/XzBpU3OnHpTVNbdNo7KqroYylEuo0a2xd5TjNLItMm03irBhgmU5Ca5mJ8uMcCFFUOHz/\n6Op8O90Ct8LhHOD5+Ovc9/35nM/7Pucj8PL+ZWlvb28XAAAAAADf4eftAgAAAAAAvoewCAAAAAAw\nICwCAAAAAAwIiwAAAAAAA8IiAAAAAMCAsAgAAAAAMCAsAgAAAAAMCIsAAAAAAAPCIgAAAADAgLAI\nAAAAADAI8HYB3nDy5Elvl2BaVFSU6urqvF0G+jHmGDyNOQZPYn7B05hj8DRvzLGYmBhT7TiyCAAA\nAAAwICwCAAAAAAwIiwAAAAAAA8IiAAAAAMCAsAgAAAAAMCAsAgAAAAAMCIsAAAAAAAPCIgAAAADA\ngLAIAAAAADAI8HYBAAD0VW0Pz7is9v6bd3moEgAAeh5HFgEAAAAABoRFAAAAAIABYREAAAAAYEBY\nBAAAAAAYEBYBAAAAAAaERQAAAACAAWERAAAAAGBAWAQAAAAAGBAWAQAAAAAGhEUAAAAAgEGAtwsA\nAPQPbQ/PuOT60x2099+8y3PFAACAbuPIIgAAAADAgLAIAAAAADAgLAIAAAAADAiLAAAAAAADwiIA\nAAAAwICwCAAAAAAwICwCAAAAAAwIiwAAAAAAA8IiAAAAAMCAsAgAAAAAMCAsAgAAAAAMCIsAAAAA\nAAPCIgAAAADAgLAIAAAAADAgLAIAAAAADAiLAAAAAACDgN4aqKKiQvn5+XI6nUpLS9PMmTPdtre3\ntys/P1/l5eUKCgpSZmam4uLiTPX9wx/+oN/+9rfasmWLwsLCemuXAAAAAKDf6pUji06nU3l5eXr6\n6ae1du1a7d27VydOnHBrU15erlOnTiknJ0cLFy7Uli1bTPWtq6vTRx99pKioqN7YFQAAAAAYEHol\nLFZVVWnYsGGKjo5WQECAkpOTVVpa6tbmwIEDSk1NlcVi0ahRo9Tc3CyHw9Fl323btunee++VxWLp\njV0BAAAAgAGhV05DtdvtstlsrmWbzabKykpDm+8eHbTZbLLb7Z32LS0tldVq1YgRIzodv7i4WMXF\nxZKk7OzsPnUUMiAgoE/Vi76HOYaecvoy2/eHeTcQ99nX8DMMnsYcg6f58hzrtWsWe9r58+f19ttv\n65lnnumybXp6utLT013LdXV1niytR0VFRfWpetH3MMfgLQNx3g3EffY0fobB05hj8DRvzLGYmBhT\n7UyHxYMHD+rzzz9XS0uL2/o5c+Z02ddqtaq+vt61XF9fL6vVamjz3Q/p2zZtbW2X7Hv69GnV1NTo\nySefdK1ftmyZXnzxRUVERJjdLQAAAADAJZgKi3l5edq3b5/Gjh2roKCgyx4kPj5e1dXVqqmpkdVq\nVUlJibKystzaJCUlqbCwUCkpKaqsrFRISIgiIyMVFhZ2yb6xsbGum+BI0qJFi/Tiiy9yN1QAAAAA\n6AGmwuIHH3ygVatWXfG5tP7+/po/f75WrFghp9OpadOmKTY2VkVFRZKkjIwMJSQkqKysTFlZWQoM\nDFRmZmanfQEAAAAAnmMqLIaFhWnQoEHdGigxMVGJiYlu6zIyMlyvLRaLFixYYLrv9+Xm5narPgAA\nAADA/2cqLN5xxx3KycnRT3/6U4WHh7tti46O9khhAAAAAADvMRUWv702sKyszLCtoKCgZysCAAAA\nAHidqbBIIAQAAACAgeWynrNYV1cnu90uq9Xqsw+OBAAAAAB0n6mw6HA4tG7dOh09elSDBw9WU1OT\nRo0apaVLlxqelwgAAAAA6Pv8zDTavHmzrrvuOuXn52vTpk3Kz8/XiBEjtHnzZk/XBwAAAADwAlNh\n8dNPP9UDDzyg4OBgSVJwcLDuu+8+HT161KPFAQAAAAC8w1RYHDRokE6cOOG27uTJkwoJCfFIUQAA\nAAAA7zJ1zeKMGTP0wgsv6Ec/+pGGDBmi2tpavffee5ozZ46n6wMAAAAAeIGpsJienq5hw4bpgw8+\n0PHjxxUZGamsrCzdcMMNnq4PAAAAfUjbwzMuu4//5l0eqARAd5l+dMa4ceM0btw4T9YCAAAAYAC4\n3P9U4D8UvKPDsPjWW29p1qxZkqSCgoIO34BTUQEAAACg/+kwLNbX11/yNfof/mcHAAAAwPd1GBYf\nfvhh1+vMzMxeKQYAAAAA4BtMXbN44sQJhYaGKiIiQi0tLdq1a5csFotmzJihoKAgT9cIAAAAAOhl\npp6z+NJLL+ns2bOSpO3bt+vIkSOqrKzUpk2bPFocAAAAAMA7TB1ZrKmpUUxMjNrb27V//379+7//\nuwIDA7V48WJP1wcAAOBVXNsPYKAyFRYDAwN17tw5nThxQlFRUQoLC1NbW5suXrzo6foAAAAAAF5g\nKiympKToV7/6lc6dO6ef/OQnkqRjx45p6NChHi0OAAD4Fo6yDQx8zwAkk2Fx3rx5OnjwoPz9/TVu\n3DhJksVi0YMPPujR4gAAQP9BAAGAvsVUWJSk8ePHu16fPn1agwcPVnx8vEeKAgAAAAB4l6m7oa5b\nt06ffvqpJOl//ud/9Nhjj+nxxx/Xu+++69HiAAAAAADeYSosHjp0yHUU8Z133tEvf/lLrVy5Ujt3\n7vRocQAAAAAA7zB1Gmpra6sCAgJkt9t15swZjR49WpLU2Njo0eIAAAAAAN5hKiyOGDFCb7/9tmpr\na5WYmChJstvtuvrqqz1aHAAAAADAO0ydhvroo4/q+PHjunDhgu6++25J0tGjR3Xrrbd6tDgAAAAA\ngHeYOrI4bNgwLV261G3d5MmTNXnyZI8UBQAAAADwLlNhsb29XX/+859VUlKir7/+WqtXr9bhw4fV\n0NCg5ORkT9cIAAAAAOhlpsJiQUGBPv74Y02fPl2bN2+WJNlsNm3bto2w6AN4yDHM6GienO6gPfME\nAABgYDN1zeKePXu0bNkypaSkyGKxSJKGDh2qmpoajxYHAAAAAPAOU2HR6XQqODjYbV1LS4thHQAA\nAACgfzAVFm+66SZt375dFy9elPTNNYwFBQWaMGGCR4sDAAAAAHiHqbD44IMPyuFwaN68eTp79qwe\neOAB1dbW6t577/V0fQAAAAAAL+jyBjft7e1qamrSY489pjNnzqi2tlZRUVGKiIjojfoAAAAAAF7Q\n5ZFFi8WiJ554QhaLReHh4br++usJigAAAADQz5l6dMaIESNUXV2t4cOHX/FAFRUVys/Pl9PpVFpa\nmmbOnOm2vb29Xfn5+SovL1dQUJAyMzMVFxfXad/XX39dBw4ccAXZzMxMWa3WK64RAAAAAPANU2Fx\n7NixWrlypaZMmaKoqCi3bT/60Y+67O90OpWXl6dnnnlGNptNTz31lJKSknTNNde42pSXl+vUqVPK\nyclRZWWltmzZopUrV3bad8aMGbr77rslSX/605/05ptvauHChZez/wAAAACASzAVFj/99FMNHTpU\nR44cMWwzExarqqo0bNgwRUdHS5KSk5NVWlrqFhYPHDig1NRUWSwWjRo1Ss3NzXI4HKqtre2wb0hI\niKv/+fPnXc+ABAAAAAB0j6mw+Nxzz3VrELvdLpvN5lq22WyqrKw0tPnuUUubzSa73d5l3x07duj9\n999XSEhIh3UWFxeruLhYkpSdnW04OurLAgICuqz39GW+5/ffr7v90TfwPcPTBuIcY5+79t199sTn\nZeb3ZHfxPXetO9/z9/v7mt6YYwPRQPx31RFfnmOmwqIkNTc3q6ysTA6HQ5GRkUpMTNSgQYM8WZsp\nc+fO1dy5c/X222+rsLBQs2fPNrRJT09Xenq6a7murq43S+yWqKioHq+3u+/Xlz4/XDm+Z3jaQJxj\n7HPP9/XE78nu8rV6ekN//tvCF+fYQNSfvwNvzLGYmBhT7Uw9Z/HQoUNatGiR/uu//ktVVVUqLCzU\nokWL9PHHH5saxGq1qr6+3rVcX19vuBGN1Wp1+5C+bWOmryTddttt+t///V9T9QAAAAAAOmfqyGJe\nXp4WLlyo5ORk17p9+/YpLy9P69at67J/fHy8qqurVVNTI6vVqpKSEmVlZbm1SUpKUmFhoVJSUlRZ\nWamQkBBFRkYqLCysw77V1dX6wQ9+IEkqLS01nZABAAAAAJ0zFRYdDocmT57stm7SpEl65ZVXTA3i\n7++v+fPna8WKFXI6nZo2bZpiY2NVVFQkScrIyFBCQoLKysqUlZWlwMBAZWZmdtpXkn73u9+purpa\nFotFUVFR3AkVAAAAAHqIqbCYmpqqwsJCTZ8+3bWuqKhIqamppgdKTExUYmKi27qMjAzXa4vFogUL\nFpjuK0lPPPGE6fEBAAAAAOaZCovHjh3T7t27tWvXLlmtVtntdjU2NmrkyJFudyB9/vnnPVYogIGp\n7eEZl9Xef/MuD1UCAAAwsJgKi2lpaUpLS/N0LQAAL7rcYC4RzgEA6M9MhcWpU6d6uAwAAAAAgC8x\n/ZzFI0eO6NixY2ppaXFbP2vWrB4vCgCA/o5TrAEAvs5UWHz11Ve1b98+jR49WoGBga71FovFY4UB\nAAAAfQGn8aO/MhUW//KXv2jNmjWyWq2ergcAAAAA4AP8zDSKiorSVVdd5elaAAAAAAA+wtSRxUcf\nfVSvvPKKUlJSFB4e7rZtzJgxHikMAAAAAOA9psLi//3f/6m8vFxHjhxxu2ZRkjZu3OiRwgAAAAAA\n3mMqLO7YsUPLli3TjTfe6Ol6BqyOLow+3UF7X7komrv5ob9ibgMAgIHO1DWLQUFBnG4KAAAAAAOI\nqSOLc+bM0datW3XnnXcqLCzMbZufn6m8CQAAAHSJMzsA32EqLH57XeLu3bsN2woKCnq2IgAAAACA\n15kKixs2bPB0HQAAAAAAH2IqLA4ZMsTTdQAAAAAAfEiHYfGVV17RI488Iklav369LBbLJdstXrzY\nM5UBAAAAALymw7A4dOhQ1+thw4b1SjEAAAAAAN/QYVj86U9/6np911139UoxGFj66t3O+mrdAAAA\nwOXguRcAAAAAAAPCIgAAAADAwNTdUAEAAABfx6UiQM/qMiw6nU698cYbmjVrlq666qreqAkAANP4\n4xC+jPkJoC/r8jRUPz8/FRUVyd/fvzfqAQAAAAD4AFPXLKampmr37t2ergUAAAAA4CNMXbNYVVWl\nwsJC7dq1SzabTRaLxbXt+eef91hxAAAAAADvMBUW09LSlJaW5ulaAAAAAAA+wlRYnDp1qofLAAAA\nAAD4ElNhsb29XX/+85+1d+9eNTU1afXq1Tp8+LAaGhqUnJzs6RoBAAAAAL3MVFgsKCjQxx9/rOnT\np2vz5s2SJJvNpm3bthEWAfgsblkPADCro98Zpztoz+8MDASm7oa6Z88eLVu2TCkpKa6b2wwdOlQ1\nNTUeLQ4AAAAA4B2mwqLT6VRwcLDbupaWFsM6AAAAAED/YCosJiQkaPv27bp48aKkb65hLCgo0IQJ\nEzxaHAAAAADAO0yFxQceeEAOh0Pz5s3T2bNn9cADD6i2tlb33nuvp+sDAAAAAHiBqRvchISE6Mkn\nn1RjY6Nqa2sVFRWliIgIT9cGAAAAAPASU2FRkpqbm/XRRx/J4XAoMjJSCQkJCg0NNT1QRUWF8vPz\n5XQ6lZaWppkzZ7ptb29vV35+vsrLyxUUFKTMzEzFxcV12ve3v/2tPvzwQwUEBCg6OlqZmZkaNGiQ\n6ZoAAAAAAJdmKiweOnRIq1evVkxMjKKiolRfX6+8vDw9/vjjuuGGG7rs73Q6lZeXp2eeeUY2m01P\nPfWUkpKSdM0117jalJeX69SpU8rJyVFlZaW2bNmilStXdtr3xhtv1D333CN/f3+99tprevvtt3Xf\nffdd+acBAD6AR34AAABfYCos5uXlaeHChW7PVNy3b5/y8vK0bt26LvtXVVVp2LBhio6OliQlJyer\ntLTULSweOHBAqampslgsGjVqlJqbm+VwOFRbW9th3/Hjx7v6jxo1Sn/961/N7TUAAAAAoFOmbnDj\ncDg0efJkt3WTJk1SQ0ODqUHsdrtsNptr2WazyW63G9pERUUZ2pjpK0nvvvuubrrpJlP1AAAAAAA6\nZ+rIYmpqqgoLCzV9+nTXuqKiIqWmpnqssMvx1ltvyd/fX7fddtsltxcXF6u4uFiSlJ2d7RZKfcXp\ny2z/3X3oTl9vjt3dur2FunvXQJyf3hr7csf9/th9pW5f+a661fenyZ20NIp+u+QyR+uYr/27CggI\nMNWuO5+Zr+1zb+jNfe5u/77Y9/v9B6K++m/DE8z+HPMGU2Hx2LFj2r17t3bt2iWr1Sq73a7GxkaN\nHDlSzz33nKvd888/f8n+VqtV9fX1ruX6+npZrVZDm7q6OkObtra2Tvu+9957+vDDD/Xss8/KYrFc\ncvz09HSlp6e7lr87Tl/VnX3o7v57a+y++r1Rd+8aiPOzr47trboH4ufVn/c5KirKI/s3EL+r7hiI\nf1t4c58Hov78eXnq51hnYmJiTLUzFRbT0tKUlpZ2xcXEx8erurpaNTU1slqtKikpUVZWllubpKQk\nFRYWKiUlRZWVlQoJCVFkZKTCwsI67FtRUaHf//73ev755xUUFHTF9QEAAAAA3JkKi1OnTu3WIP7+\n/po/f75WrFghp9OpadOmKTY2VkVFRZKkjIwMJSQkqKysTFlZWQoMDFRmZmanfaVvbrzT2tqqF154\nQZI0cuRILVy4sFu1AgAAAAAu4zmL3ZWYmKjExES3dRkZGa7XFotFCxYsMN1XktavX9+zRQIAAAAA\nJPViWASAgeJyn5Mo8axEAADgewiL6JP4Yxy4tMv9tzHQ/13wswQA+h5+1/UeU89ZBAAAAAAMLKaO\nLLa2tuq9997T559/rpaWFrdtixcv9khhAAAAAADvMRUWN2zYoC+++EITJkxQeHi4p2sC+i1OmwAA\nAEBfYSosHjx4UBs2bNCgQYM8XQ/QKwhtAAAAQOdMXbMYFRWlixcveroWAAAAAICPMHVkMTU1VatW\nrdLtt9+uiIgIt23jxo3zSGEAAAAAAO8xFRYLCwslSTt27HBbb7FYtGHDhp6vCgAAAECXuLQGnmQq\nLK5fv15+fjxlAwAAAMA3CKr9X5cJ0Ol06v777+eaRQAAAAAYQLoMi35+foqJiVFTU1Nv1AMAAAAA\n8AGmTkO99dZb9a//+q+6/fbbZbPZZLFYXNu4wQ3QezjdAwAAAL3FVFgsKiqSJL3xxhtu67nBDQAA\nAAD0T6bCYm5urqfrAAAAAACPutyztKSBfaYWtzgFAAAAABiYOrL4s5/9rMNtGzdu7LFiAAAAAMBX\nDbT7R5gKi0uWLHFbdjgc+tOf/qSUlBSPFAUAAAAA8C5TYXHMmDGGdWPHjtWKFSs0ffr0Hi8KAAB4\nzkD7n/GBiu8ZQHeZCouX7BgQoJqamp6sBQAAAEAv4WYv6IqpsFhQUOC2fP78eZWXlyshIcEjRQEA\nAAAAvMtUWKyvr3dbDgoK0h133KHU1FSPFAUAAAAA8C5TYfGee+5RRESEYX1DQ8Ml1wMAcDm4tgoA\nAN9jKiwuXbpU27ZtM6z/+c9/rvz8/B4vCkD/wfUQAAAAfZOfmUbt7e2GdWfPnpWfn6nuAAAAAIA+\nptMjiz/72c8kSRcuXHC9/taZM2d4ziIAAAAA9FOdhsUlS5aovb1dL774opYsWeK2LSIiQjExMR4t\nDgAAAADgHZ2GxTFjxkiS8vLyFBQU1CsFAQAAAAC8z9QNbvz8/LRjxw7t3btXTU1N2rZtmw4ePKjq\n6mr95Cc/8XSNALyMO1UCQO/jBmEAvM3UHWq2bt2qL7/8UllZWbJYLJKk2NhYFRUVebQ4AAAAAIB3\nmDqyWFpaqpycHAUHB7vCotVqld1u92hxAAAAAADvMHVkMSAgQE6n023d119/rcGDB3ukKAAAAACA\nd5kKi5MnT9aGDRtUU1MjSXI4HMrLy1NycrJHiwMAAAAAeIepsHjPPfdo6NChevzxx3X27FllZWUp\nMjJSd955p6frAwAAAAB4galrFgMCAjRv3jzNmzfPdfrpt9cuAgAAeFpHdwY93Ukf7gyKvoK7jsNX\nmQqL3xUWFiZJOn78uN5880099thjpvpVVFQoPz9fTqdTaWlpmjlzptv29vZ25efnq7y8XEFBQcrM\nzFRcXFynffft26c33nhDX331lVauXKn4+PjL3R0AAAAAwCV0ehrq+fPn9frrrys7O1vbtm3T2bNn\ndfr0aa1atUq/+MUvXMGxK06nU3l5eXr66ae1du1a7d27VydOnHBrU15erlOnTiknJ0cLFy7Uli1b\nuuwbGxurJ554Qn/3d393JfsOAAAAAOhAp0cW8/LydOzYMY0fP14VFRU6fvy4Tp48qSlTpuiRRx4x\nHRarqqo0bNgwRUdHS5KSk5NVWlqqa665xtXmwIEDSk1NlcVi0ahRo9Tc3CyHw6Ha2toO+363PwAA\nAACg53QaFg8ePKh/+7d/U3h4uG6//XZlZmbqX/7lXy77SJ7dbpfNZnMt22w2VVZWGtpERUW5tbHb\n7ab6AgAAAAB6VqdhsaWlReHh4ZK+CWnBwcF98pTP4uJiFRcXS5Kys7PdQqmv6OwC/Uv57j50p683\nx+7Nvt4cm+/q8vXFugfiPntzbPb5yvt6c2y+K8/29ebYA7HugbjP3hy7P+xzRwICAnwyn0hdhMW2\ntjYdOnTIbd33l8eNG9flIFarVfX19a7l+vp6Wa1WQ5u6ujpDm7a2ti77diU9PV3p6emu5e+O01d1\nZx+6u//eGnsg1j0Q99mbY7PPfWds9rn3+npz7IFY90DcZ2+OzT73nbH78z5HRUX1ej6JiYkx1a7T\nsBgeHq6NGze6lkNDQ92WLRaLNmzY0OUg8fHxqq6uVk1NjaxWq0pKSpSVleXWJikpSYWFhUpJSVFl\nZaVCQkIUGRmpsLCwLvsCAAAAAHpWp2ExNze3Rwbx9/fX/PnztWLFCjmdTk2bNk2xsbEqKiqSJGVk\nZCghIUFlZWXKyspSYGCgMjMzO+0rSfv379err76qr7/+WtnZ2RoxYoR+8Ytf9EjNAAAAADCQXfZz\nFq9UYmKiEhMT3dZlZGS4XlssFi1YsMB0X0maNGmSJk2a1LOFAgAAAAA6f84iAAAAAGBgIiwCAAAA\nAAwIiwAAAAAAA8IiAAAAAMCAsAgAAAAAMCAsAgAAAAAMCIsAAAAAAAPCIgAAAADAgLAIAAAAADAg\nLAIAAAAADAiLAAAAAAADwiIAAAAAwICwCAAAAAAwICwCAAAAAAwIiwAAAAAAA8IiAAAAAMCAsAgA\nAAAAMCAsAgAAAAAMCIsAAAAAAAPCIgAAAADAgLAIAAAAADAgLAIAAAAADAiLAAAAAAADwiIAAAAA\nwICwCAAAAAAwICwCAAAAAAwIiwAAAAAAA8IiAAAAAMCAsAgAAAAAMCAsAgAAAAAMCIsAAAAAAAPC\nIgAAAADAgLAIAAAAADAgLAIAAAAADAiLAAAAAACDgN4aqKKiQvn5+XI6nUpLS9PMmTPdtre3tys/\nP1/l5eUKCgpSZmam4uLiOu175swZrV27VrW1tRoyZIh+/vOfKzQ0tLd2CQAAAAD6rV45suh0OpWX\nl6enn35aa9eu1d69e3XixAm3NuXl5Tp16pRycnK0cOFCbdmypcu+O3fu1A033KCcnBzdcMMN2rlz\nZ2/sDgAAAAD0e70SFquqqjRs2DBFR0crICBAycnJKi0tdWtz4MABpaamymKxaNSoUWpubpbD4ei0\nb2lpqaZMmSJJmjJliuE9AQAAAABXplfCot1ul81mcy3bbDbZ7XZDm6ioKEObzvo2NjYqMjJSkhQR\nEaHGxkZP7gYAAAAADBiW9vb2dk8P8te//lUVFRV69NFHJUnvv/++Kisr9dBDD7naZGdna+bMmRo9\nerQk6Ve/+pXuvfde1dbWdth33rx52rp1q+s9/umf/kn5+fmG8YuLi1VcXOwaBwAAAADQuV45smi1\nWlVfX+9arq+vl9VqNbSpq6sztOmsb3h4uBwOhyTJ4XAoLCzskuOnp6crOzu7TwbF5cuXe7sE9HPM\nMXgacwyexPyCpzHH4Gm+PMd6JSzGx8erurpaNTU1am1tVUlJiZKSktzaJCUl6f3331d7e7uOHj2q\nkJAQRUZGdto3KSlJe/bskSTt2bNHEydO7I3dAQAAAIB+r1ceneHv76/58+drxYoVcjqdmjZtmmJj\nY1VUVCRJysjIUEJCgsrKypSVlaXAwEBlZmZ22leSZs6cqbVr1+rdd991PToDAAAAANB9vXLNIq5c\ncXGx0tPTvV0G+jHmGDyNOQZPYn7B05hj8DRfnmOERQAAAACAQa9cswgAAAAA6Ft65ZpFXJmKigrl\n5+fL6XQqLS1NM2fO9HZJ6ON+85vfqKysTOHh4VqzZo0k6cyZM1q7dq1qa2td1/6GhoZ6uVL0RXV1\ndcrNzVVDQ4MsFovS09M1ffp05hh6zIULF/Tcc8+ptbVVbW1tmjx5smbPns0cQ49yOp1avny5rFar\nli9fzvxCj1u0aJGCg4Pl5+cnf39/ZWdn++w84zRUH+V0OrV06VI988wzstlseuqpp7R06VJdc801\n3i4Nfdjhw4cVHBys3NxcV1h87bXXFBoaqpkzZ2rnzp06c+aM7rvvPi9Xir7I4XDI4XAoLi5O586d\n0/Lly/Xkk0/qvffeY46hR7S3t+v8+fMKDg5Wa2urnn32Wc2bN0/79+9njqHHvPPOO/rss89cP8f4\nPYmetmjRIr344otuj/3z1XnGaag+qqqqSsOGDVN0dLQCAgKUnJys0tJSb5eFPm7MmDGG/6UqLS3V\nlClTJElTpkxhnuGKRUZGKi4uTpJ09dVXa/jw4bLb7cwx9BiLxaLg4GBJUltbm9ra2mSxWJhj6DH1\n9fUqKytTWlqaax3zC73BV+cZp6H6KLvdLpvN5lq22WyqrKz0YkXorxobGxUZGSlJioiIUGNjo5cr\nQn9QU1PHOIxAAAAIQ0lEQVSjY8eO6frrr2eOoUc5nU4tW7ZMp06d0t///d9r5MiRzDH0mK1bt+q+\n++7TuXPnXOuYX/CEF154QX5+fvrxj3+s9PR0n51nhEUALhaLRRaLxdtloI9raWnRmjVrNG/ePIWE\nhLhtY46hu/z8/LRq1So1Nzdr9erVOn78uNt25hiu1Icffqjw8HDFxcXpk08+uWQb5hd6wgsvvCCr\n1arGxkb9+te/VkxMjNt2X5pnhEUfZbVaVV9f71qur6+X1Wr1YkXor8LDw+VwOBQZGSmHw+F2/jxw\nuVpbW7VmzRrddtttuvnmmyUxx+AZgwYN0tixY1VRUcEcQ4/49NNPdeDAAZWXl+vChQs6d+6ccnJy\nmF/ocd/+TR8eHq6JEyeqqqrKZ+cZ1yz6qPj4eFVXV6umpkatra0qKSlRUlKSt8tCP5SUlKQ9e/ZI\nkvbs2aOJEyd6uSL0Ve3t7Xr55Zc1fPhw3XHHHa71zDH0lK+//lrNzc2Svrkz6kcffaThw4czx9Aj\n7rnnHr388svKzc3VP//zP2vcuHHKyspifqFHtbS0uE5zbmlp0UcffaRrr73WZ+cZd0P1YWVlZdq2\nbZucTqemTZumWbNmebsk9HHr1q3T4cOH1dTUpPDwcM2ePVsTJ07U2rVrVVdX51O3akbf87e//U3P\nPvusrr32WtfpM3PnztXIkSOZY+gRX3zxhXJzc+V0OtXe3q5bbrlFd955p5qamphj6FGffPKJ/vCH\nP2j58uXML/So06dPa/Xq1ZK+uVHXrbfeqlmzZvnsPCMsAgAAAAAMOA0VAAAAAGBAWAQAAAAAGBAW\nAQAAAAAGhEUAAAAAgAFhEQAAAABgQFgEAMCL6urqdP/998vpdHq7FAAA3PDoDABAv/XBBx/onXfe\n0VdffaWrr75aI0aM0KxZszR69Ghvl9YrcnNzZbPZdPfdd6umpkaLFy9WUFCQJCk4OFjx8fGaPn26\nbrzxRi9XCgDwRQHeLgAAAE945513tHPnTj388MMaP368AgICdPDgQR04cGDAhMVL2bp1q/z9/dXQ\n0KCSkhKtWrVKDz30kKZOnert0gAAPoawCADod86ePauCggJlZmbq5ptvdq2fMGGCJkyYIEmqqqpS\nfn6+vvrqKwUGBurmm2/Wgw8+qICAb341zp49Ww899JD++Mc/qqGhQdOnT9fUqVO1YcMGffnllxo/\nfryysrIUEBCgTz75ROvXr1dGRob++Mc/Kjg4WHfffbduu+02SVJZWZlef/11nT59WiEhIZo2bZpm\nz54tSa4jfjt27JC/v79qamqUm5urY8eOaeTIkfrBD36gs2fPKisry9U2MzNTBQUFunDhgv7hH/5B\ns2bNuuzPKCIiQtOnT1dra6t+97vfKTU1VX5+XJ0CAPj/+K0AAOh3jh49qosXL2rSpEkdtvHz89OD\nDz6ovLw8/frXv9ahQ4f03//9325tDh48qOzsbK1YsUK7du3Spk2btGTJEm3cuFFffvmlPvjgA1fb\nhoYGNTU16eWXX9aiRYu0adMmnTx5UpIUFBSkxYsXKz8/X8uXL9fu3bu1f//+S9b10ksvKT4+Xq++\n+qruuusu/eUvfzG0+dvf/qaXXnpJv/zlL/Xmm2/qxIkTV/IxSZJuvvlmNTY2umoFAOBbhEUAQL/T\n1NSkwYMHy9/fv8M2cXFxGjVqlPz9/TV06FClp6fr8OHDbm1mzJihkJAQxcbGKjY2VjfeeKOio6MV\nEhKihIQEff75527t58yZo6uuukpjxoxRQkKCSkpKJEljx47VtddeKz8/P1133XVKSUkxjCV9c7Ob\nzz77THPmzFFAQIBGjx7tOhL6XXfddZcCAwM1YsQIXXfddfriiy+u4FP6RmRkpCTpzJkzV/weAID+\nidNQAQD9zuDBg9XU1KS2trYOA+PJkye1fft2ffbZZ7pw4YLa2toUFxfn1iYiIsL1OjAw0LDc0NDg\nWh40aJCCg4Ndy0OGDJHD4ZAkVVZW6j/+4z90/Phxtba2qrW1VZMnTzbUZLfbFRoa6roJjSRFRUWp\nrq6uw7qCgoLU0tLS6efRGbvdLkkKDQ294vcAAPRPHFkEAPQ7o0aN0lVXXaXS0tIO22zZskUxMTHK\nycnRtm3bNHfuXHXnBuHNzc1uoa2urs511C4nJ0cTJkzQxo0btW3bNv34xz++5FiRkZE6c+aMzp8/\n7/Y+nrR//36Fh4crJibGo+MAAPoewiIAoN8JCQnR7NmzlZeXp/379+v8+fNqbW1VeXm5XnvtNUnS\nuXPnFBISouDgYH311VcqKirq9rj/+Z//qdbWVh05ckRlZWW65ZZbXGOFhoYqMDBQVVVVbtc6fteQ\nIUMUHx+vN954Q62trTp69Kg+/PDDbtd1KQ0NDSosLNSbb76puXPncnMbAIABp6ECAPqlf/zHf1RE\nRITeeustrV+/XsHBwYqLi3PdOfT+++/Xpk2b9Pvf/14//OEPlZycrEOHDl3xeBEREQoNDdUjjzyi\nwMBAPfzwwxo+fLgkacGCBdq+fbteffVVjRkzRrfccouam5sv+T5LlizRb37zG82fP1/XX3+9kpOT\n5XQ6r7iu75s3b56kb05fjY+P12OPPaabbrqpx94fANB/WNq7c84NAABwPTrj5Zdf7vH3Xrt2rYYP\nH+561AYAAL2Fc04AAPAhVVVVOnXqlJxOpyoqKnTgwAFNnDjR22UBAAYgTkMFAMCHNDQ0aM2aNWpq\napLNZtOCBQv0wx/+0NtlAQAGIE5DBQAAAAAYcBoqAAAAAMCAsAgAAAAAMCAsAgAAAAAMCIsAAAAA\nAAPCIgAAAADAgLAIAAAAADD4f87YWp8mC6roAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e00d2eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualization of the attribution scores\n",
    "keras_logreg = model.get_layer('contributions').get_weights()[0].flatten()[0:n_campaigns]\n",
    "keras_logreg = softmax([keras_logreg]).flatten()\n",
    "\n",
    "fig = plt.figure(figsize=(15,4))\n",
    "ax = fig.add_subplot(111)\n",
    "plt.bar(range(len(keras_logreg[campaign_idx])), keras_logreg[campaign_idx] )\n",
    "plt.xlabel('Campaign ID')\n",
    "plt.ylabel('Return per impression')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic LSTM\n",
    "\n",
    "In this section, we create feature engineering procedures for sequential models and test basic LSTM.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(145440, 15, 1563)\n"
     ]
    }
   ],
   "source": [
    "def features_for_lstm(df, max_touchpoints):\n",
    "    \n",
    "    df_proj = df[['jid', 'campaigns', 'cats', 'click', 'cost', 'time_since_last_click_norm', 'timestamp_norm', 'conversion']]\n",
    "    \n",
    "    x2d = df_proj.values\n",
    "    \n",
    "    x3d_list = np.split(x2d[:, 1:], np.cumsum(np.unique(x2d[:, 0], return_counts=True)[1])[:-1])\n",
    "    \n",
    "    x3d = []\n",
    "    y = []\n",
    "    for xi in x3d_list:\n",
    "        journey_matrix = np.apply_along_axis(np.hstack, 1, xi)\n",
    "        journey_matrix = journey_matrix[ journey_matrix[:, 5].argsort() ] # sort impressions by timestamp\n",
    "        n_touchpoints = len(journey_matrix)\n",
    "        padded_journey = []\n",
    "        if(n_touchpoints >= max_touchpoints):\n",
    "            padded_journey = journey_matrix[0:max_touchpoints]\n",
    "        else:\n",
    "            padded_journey = np.pad(journey_matrix, ((0, max_touchpoints - n_touchpoints), (0, 0)), 'constant', constant_values=(0))\n",
    "            \n",
    "        x3d.append(padded_journey[:, 0:-1])\n",
    "        y.append(np.max(padded_journey[:, -1]))\n",
    "        \n",
    "    return np.stack(x3d), y\n",
    "\n",
    "x, y = features_for_lstm(df6, max_touchpoints = 15)\n",
    "print(np.shape(x))\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.20, random_state = 1)\n",
    "x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size = 0.20, random_state = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 93081 samples, validate on 23271 samples\n",
      "Epoch 1/5\n",
      "93081/93081 [==============================] - 135s 1ms/step - loss: 0.3033 - acc: 0.8677 - val_loss: 0.2547 - val_acc: 0.8899\n",
      "Epoch 2/5\n",
      "93081/93081 [==============================] - 152s 2ms/step - loss: 0.2603 - acc: 0.8879 - val_loss: 0.2322 - val_acc: 0.9013\n",
      "Epoch 3/5\n",
      "93081/93081 [==============================] - 160s 2ms/step - loss: 0.2415 - acc: 0.8979 - val_loss: 0.2274 - val_acc: 0.9069\n",
      "Epoch 4/5\n",
      "93081/93081 [==============================] - 156s 2ms/step - loss: 0.2293 - acc: 0.9047 - val_loss: 0.2091 - val_acc: 0.9129\n",
      "Epoch 5/5\n",
      "93081/93081 [==============================] - 155s 2ms/step - loss: 0.2248 - acc: 0.9067 - val_loss: 0.2136 - val_acc: 0.9112\n",
      "Test score: 0.21668663059081575\n",
      "Test accuracy: 0.9088627612761276\n"
     ]
    }
   ],
   "source": [
    "n_steps, n_features = np.shape(x)[1:3]\n",
    "    \n",
    "model = Sequential() \n",
    "model.add(LSTM(64, dropout=0.2, recurrent_dropout=0.2, input_shape=(n_steps, n_features)))\n",
    "model.add(Dense(1, activation='sigmoid')) \n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy']) \n",
    "history = model.fit(x_train, y_train, batch_size=64, epochs=5, verbose=1, validation_data=(x_val, y_val)) \n",
    "score = model.evaluate(x_test, y_test, verbose=0) \n",
    "print('Test score:', score[0]) \n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LSTM with Attention\n",
    "\n",
    "In this section, we fit the LSTM with Attention model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 93081 samples, validate on 23271 samples\n",
      "Epoch 1/5\n",
      "93081/93081 [==============================] - 110s 1ms/step - loss: 0.2293 - acc: 0.9018 - val_loss: 0.2162 - val_acc: 0.9110\n",
      "Epoch 2/5\n",
      "93081/93081 [==============================] - 109s 1ms/step - loss: 0.2001 - acc: 0.9152 - val_loss: 0.2038 - val_acc: 0.9123\n",
      "Epoch 3/5\n",
      "93081/93081 [==============================] - 96s 1ms/step - loss: 0.1906 - acc: 0.9199 - val_loss: 0.2002 - val_acc: 0.9166\n",
      "Epoch 4/5\n",
      "93081/93081 [==============================] - 85s 910us/step - loss: 0.1876 - acc: 0.9213 - val_loss: 0.1949 - val_acc: 0.9168\n",
      "Epoch 5/5\n",
      "93081/93081 [==============================] - 98s 1ms/step - loss: 0.1846 - acc: 0.9222 - val_loss: 0.1924 - val_acc: 0.9190\n",
      "Test score: 0.19408383931476947\n",
      "Test accuracy: 0.9186262376237624\n"
     ]
    }
   ],
   "source": [
    "n_steps, n_features = np.shape(x)[1:3]\n",
    "\n",
    "hidden_units = 64\n",
    "\n",
    "main_input = Input(shape=(n_steps, n_features))\n",
    "    \n",
    "embeddings = Dense(128, activation='linear', input_shape=(n_steps, n_features))(main_input)\n",
    "\n",
    "activations = LSTM(hidden_units, dropout=0.2, recurrent_dropout=0.2, return_sequences=True)(embeddings)\n",
    "\n",
    "attention = Dense(1, activation='tanh')(activations)\n",
    "attention = Flatten()(attention)\n",
    "attention = Activation('softmax', name = 'attention_weigths')(attention)\n",
    "attention = RepeatVector(hidden_units * 1)(attention)\n",
    "attention = Permute([2, 1])(attention)\n",
    "\n",
    "weighted_activations = Multiply()([activations, attention])\n",
    "weighted_activations = Lambda(lambda xin: K.sum(xin, axis=-2), output_shape=(hidden_units,))(weighted_activations)\n",
    "\n",
    "main_output = Dense(1, activation='sigmoid')(weighted_activations)\n",
    "\n",
    "model = Model(inputs=main_input, outputs=main_output)\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy']) \n",
    "history = model.fit(x_train, y_train, batch_size=64, epochs=5, verbose=1, validation_data=(x_val, y_val)) \n",
    "score = model.evaluate(x_test, y_test, verbose=0) \n",
    "print('Test score:', score[0]) \n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis of LSTM-A Model\n",
    "\n",
    "In this section, we compare the previously developed attribution models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_campaign_id(x_journey_step):\n",
    "    return np.argmax(x_journey_step[0:n_campaigns])\n",
    "\n",
    "attention_model = Model(inputs=model.input, outputs=model.get_layer('attention_weigths').output)\n",
    "\n",
    "a = attention_model.predict(x_train)\n",
    "\n",
    "attributions = np.zeros(n_campaigns)\n",
    "campaign_freq = np.ones(n_campaigns)\n",
    "for i, journey in enumerate(a):\n",
    "    for step, step_contribution in enumerate(journey):\n",
    "        if(np.sum(x_train[i][step]) > 0):\n",
    "            campaign_id = get_campaign_id(x_train[i][step])\n",
    "            attributions[campaign_id] = attributions[campaign_id] + step_contribution\n",
    "            campaign_freq[campaign_id] = campaign_freq[campaign_id] + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4sAAAENCAYAAACmZfCPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X9U1HW+x/HXwIhIKDADwQVxE8xjblbimEWmuYx0VszY\nMn+01pqV7cW02546i/eY7blmi6vGruWPbSXK471dbWvp53FdaoNb7CYG5JaV4GrtKorAwPoDsHHm\n/uFp1tkv6iDMD+D5+Mf5fr+fz/fz/sx8GHzz/Xy/H5Pb7XYLAAAAAIBzhAU7AAAAAABA6CFZBAAA\nAAAYkCwCAAAAAAxIFgEAAAAABiSLAAAAAAADkkUAAAAAgAHJIgAAAADAgGQRAAAAAGBAsggAAAAA\nMCBZBAAAAAAYmIMdQDAcPnw42CH4LD4+Xo2NjcEOA30YYwz+xhiDPzG+4G+MMfhbMMZYcnKyT+W4\nsggAAAAAMAjYlcWamhoVFxfL5XIpKytLubm5XscPHTqkDRs26MCBA5ozZ45mzJgh6exVwMLCQk+5\nhoYGzZo1Szk5Odq+fbveffddDRkyRJI0d+5cZWRkBKpLAAAAANBnBSRZdLlcKioq0rJly2S1WrV0\n6VLZbDYNHTrUUyY6Olr33XefKisrveomJydr9erVnvM89NBDuv766z3Hc3JyPIklAAAAAKBnBCRZ\nrKurU1JSkhITEyVJmZmZqqys9EoWY2JiFBMTo6qqqvOe5y9/+YuSkpKUkJDQo/G53W61t7fL5XLJ\nZDL16Lm76+jRo+ro6Aha+263W2FhYYqMjAy59wYAAACA/wQkWWxubpbVavVsW61W1dbWdvk8H374\noW666SavfTt27FB5ebnS0tJ07733Kjo6usvnbW9v14ABA2Q2h97zfsxms8LDw4Mag9PpVHt7uwYN\nGhTUOAAAAAAETuhlR+fhdDr18ccf6+677/bsy87O1syZMyVJ27Zt05YtW5SXl2eoW1paqtLSUklS\nQUGB4uPjvY4fPXpUAwcO9GP03RPsJNZsNstkMhneN/QNZrOZzxZ+xRiDPzG+4G+MMfhbKI+xgGQh\nFotFTU1Nnu2mpiZZLJYunaO6ulrDhw9XbGysZ9+5r7OysrRq1apO69rtdtntds/2vz6atqOjI+hX\n787HbDbL6XQGOwx1dHTw2Og+ikeCw98YY/Anxhf8jTEGf+v3S2ekp6ervr5eDQ0NcjqdqqiokM1m\n69I5OpuC6nA4PK937dql1NTUHokXAAAAAPq7gFxZDA8P14IFC7Ry5Uq5XC5NmTJFqamp2rlzp6Sz\n00lbWlqUn5+vtrY2mUwmvfPOO3rmmWcUFRWl9vZ27dmzRwsXLvQ679atW3Xw4EGZTCYlJCQYjl+q\nMw/27NNVw3/zxkXLXHnllYb7OOvq6rR06VK1tLTo9OnTmjBhgqZNm6aVK1dKkg4ePKikpCRFRkbq\nqquu0pw5c3TXXXdp9erVnum6n376qW699VY98cQT+vGPf9xp2zt27ND999+vsrIyjRgxopu9BQAA\nANAXBOxmuIyMDMMaiNnZ2Z7XsbGx2rRpU6d1IyMj9cILLxj2L168uGeDDDHLly/XwoULNXXqVEnS\n559/rquuukq33HKLJGnmzJl64okndO2110qSKioqNGrUKL355pueZPH111/X6NGjL9hOSUmJrr/+\nepWUlOixxx7zX4cAAL1eV/+g6ssfTAEAoSkg01BxaRoaGrzmE1911VUXrZOSkqKOjg4dO3ZMbrdb\nf/zjH/W9733vvOVPnjypyspKrVmzRq+//nqPxA0AAACg9yNZDGEPPvig7rzzTs2bN0/PP/+8Wltb\nfaqXk5Ojt956S7t379aYMWMUERFx3rK///3vdcsttyg9PV1xcXHas2dPT4UPAAAAoBcjWQxhs2fP\n1gcffKDp06frT3/6k2677TZ1dHRctN5tt92mt956SyUlJcrNzb1g2ZKSEt1+++2SpNtvv10lJSU9\nEjsAAACA3o1kMcQlJSVpzpw5Ki4ultls1pdffnnROpdffrnMZrPKy8s1ceJEz/5Dhw5p6tSpmjp1\nqrZs2SKHw6EPP/xQjz32mCZMmKCNGzfqzTfflNvt9meXAAAAAPQCwV3tHRf0xz/+UbfccotMJpMa\nGhrkcDiUlJTkU93HHntMTU1NXutHpqSk6A9/+INne+vWrbrzzjv1i1/8wrPvzjvv1EcffaQbbrih\n5zoCAAAAoNchWexEMJ7c1tbWpnHjxnm2Fy5cqPr6ej355JOeew6XLVumyy+/3KfzjR8//qJlSkpK\ntGjRIq9906ZNU0lJCckiAAAA0M+Z3P1wzuHhw4e9tk+dOqWoqKggRXNhZrNZTqcz2GGE9HuE7omP\nj1djY2Oww0AfxhjrW0Jt6QzGF/yNMQZ/C8YYO3fFhQvhnkUAAAAAgAHJIgAAAADAgGRR4umfPuA9\nAgAAAPoXkkVJYWFhIXFfYKhyOp0KC2OoAAAAAP0JT0OVFBkZqfb2dnV0dMhkMgU7HC8DBw5UR0dH\n0Np3u90KCwtTZGRk0GIAAAAAEHgki5JMJpMGDRoU7DA6xRO4AAAAAAQDcwsBAAAAAAYkiwAAAAAA\nA5JFAAAAAIABySIAAAAAwIBkEQAAAABgwNNQAQAIgjMPzuhS+fDfvOGnSAAA6BxXFgEAAAAABiSL\nAAAAAACDgE1DrampUXFxsVwul7KyspSbm+t1/NChQ9qwYYMOHDigOXPmaMaMf07PWbRokSIjIxUW\nFqbw8HAVFBRIkk6cOKHCwkIdO3ZMCQkJevTRRxUdHR2oLgEAAABAnxWQZNHlcqmoqEjLli2T1WrV\n0qVLZbPZNHToUE+Z6Oho3XfffaqsrOz0HE8++aSGDBnita+kpERjxoxRbm6uSkpKVFJSonnz5vm1\nLwAAAADQHwRkGmpdXZ2SkpKUmJgos9mszMxMQ1IYExOjESNGKDw83OfzVlZWavLkyZKkyZMnnzfR\nBAAAAAB0TUCuLDY3N8tqtXq2rVaramtru3SOFStWKCwsTFOnTpXdbpcktba2Ki4uTpIUGxur1tbW\nngsaAAAAAPqxXrF0xooVK2SxWNTa2qqnnnpKycnJGj16tFcZk8kkk8nUaf3S0lKVlpZKkgoKChQf\nH+/3mHuK2WzuVfGi92GMwd/68hg7+oPMLpVP/F3FP+t2sa1QeQ9DLe6+PL4QGhhj8LdQHmMBSRYt\nFouampo8201NTbJYLF2qL52dqjp+/HjV1dVp9OjRiomJkcPhUFxcnBwOh+Gexm/Z7XbP1UhJamxs\nvMSeBF58fPxF42WtLnSHL2MM6A7G2D91533ore+hv+NmfMHfGGPwt2CMseTkZJ/KBeSexfT0dNXX\n16uhoUFOp1MVFRWy2Ww+1W1vb1dbW5vn9Z49ezRs2DBJks1mU1lZmSSprKxM48eP908HAAAAAKCf\nCciVxfDwcC1YsEArV66Uy+XSlClTlJqaqp07d0qSsrOz1dLSovz8fLW1tclkMumdd97RM888o+PH\nj2vNmjWSpDNnzmjixIm67rrrJEm5ubkqLCzUe++951k6AwAAAADQfQG7ZzEjI0MZGRle+7Kzsz2v\nY2NjtWnTJkO9qKgorV69utNzDh48WMuXL+/ZQAEAAAAAgZmGCgAAAADoXUgWAQAAAAAGJIsAAAAA\nAINesc4iAAAAgL6Dpd96B64sAgAAAAAMSBYBAAAAAAYkiwAAAAAAA5JFAAAAAIABD7gBNxgDAAAA\nMODKIgAAAADAgCuLAAAAF8AMHFxMV8eIxDhB70CyCAAAAC8kyAAkkkV0E79MAAAAgL6JZBEA0CPO\n98ejo+cpzx+PAAAIbSSLIYL/ZAEAAAAIJTwNFQAAAABgQLIIAAAAADBgGioAAICf8CA4AL0ZySIA\ndKI//gePdcIA9AS+S4C+g2QRANCr9cfEHgCAQOCeRQAAAACAAckiAAAAAMAgYNNQa2pqVFxcLJfL\npaysLOXm5nodP3TokDZs2KADBw5ozpw5mjHj7LSixsZGrV+/Xi0tLTKZTLLb7Zo2bZokafv27Xr3\n3Xc1ZMgQSdLcuXOVkZERqC4BAAAAQJ8VkGTR5XKpqKhIy5Ytk9Vq1dKlS2Wz2TR06FBPmejoaN13\n332qrKz0qhseHq577rlHaWlpamtrU35+vq655hpP3ZycHE9iCQDn4l42XAwP4ggsfiYBoHcJyDTU\nuro6JSUlKTExUWazWZmZmYakMCYmRiNGjFB4eLjX/ri4OKWlpUmSBg0apJSUFDU3NwcibAAAAADo\ntwJyZbG5uVlWq9WzbbVaVVtb2+XzNDQ06MCBAxoxYoRn344dO1ReXq60tDTde++9io6ONtQrLS1V\naWmpJKmgoEDx8fGX0Av/OtrF8uf2oTt1g902AufoDzI733+e8om/q/BfMAHC2O6arvZZCo33LJCf\nc3frh8L71V2h1mez2ez39ybU+hwIwfy5CjW+jLG+1udA6K0/G/4QiO+xS9Vrls5ob2/X2rVrNX/+\nfEVFRUmSsrOzNXPmTEnStm3btGXLFuXl5Rnq2u122e12z3ZjY2Nggvaj7vShu/0PZtsInP74WTG2\nu643vme99Tuwt44xf/c5Pj4+5N4bPufA1/cnf42xUO5zKOrL71cwvseSk5N9KheQaagWi0VNTU2e\n7aamJlksFp/rO51OrV27VjfffLMmTJjg2R8bG6uwsDCFhYUpKytL+/fv79G4AQAAAKC/CsiVxfT0\ndNXX16uhoUEWi0UVFRVasmSJT3Xdbrc2bdqklJQUTZ8+3euYw+FQXFycJGnXrl1KTU3t8dgBAADQ\n953vAUznmy7JA5jQHwQkWQwPD9eCBQu0cuVKuVwuTZkyRampqdq5c6eks9NJW1palJ+fr7a2NplM\nJr3zzjt65pln9PXXX6u8vFzDhg3T448/LumfS2Rs3bpVBw8elMlkUkJCghYuXBiI7gAIIJ6eCAAA\nEBwBu2cxIyPDsAZidna253VsbKw2bdpkqDdq1Cht376903MuXry4Z4NEQJEEAAAAAKErIPcsAgAA\nAAB6l17zNFQAAIBLxWwWAOg6n5NFp9Op999/XwcPHlR7e7vXsYcffrjHAwNCFf/hAAAAQH/gc7L4\n3HPP6auvvtK4ceMUExPjz5gAAAAAAEHmc7L4ySef6LnnntNll13mz3gAoN/j6jUAwFf8zoA/+Zws\nxsfH65tvvvFnLAAAAAAQsvpbcu5zsjhp0iStXr1a3//+9xUbG+t17Oqrr+7xwAAY9bcvKAAAAASP\nz8nijh07JEkvv/yy136TyaTnnnuuZ6MCAAAAAASVz8ni+vXr/RkHAAAAACCEdGmdxTNnzujLL79U\nc3OzrFarRo4cqfDwcH/FBgAAAAAIEp+TxUOHDmnVqlU6ffq0rFarmpqaNGDAAP30pz/V0KFD/Rkj\nAAA4B/cvA53jZwPoWT4ni5s3b5bdbtdtt90mk8kkSXrjjTdUVFSkJ5980m8BAgAAAAACL8zXggcP\nHtT06dM9iaIk5eTk6ODBg/6ICwAAAAAQRD4nixaLRXv37vXa9/nnnysuLq7HgwIAAAAABJfP01Dn\nzp2rVatWady4cYqPj1djY6Oqqqq0ePFif8YHAOgHuM8IAIDQ43OyaLPZtGrVKv3pT3+Sw+FQamqq\nZs2apeTkZH/GBwC9TlcTH4nkBwAAhJ4uLZ2RnJysO++801+xAAhRXPUBAADofy6YLP7617/WQw89\nJEl69tlnvR5uc66HH3645yMDAAAAAATNBZPFyy+/3PM6KSnJ78EAALqHq8AAAKCnXDBZ/MEPfuB5\nPXXqVMXGxhrKtLS09HxUwEVwTxgAAADgXz4vnfHII490uv/RRx/tsWAAAAAAAKHB5wfcuN1uw75T\np04pLMy3fLOmpkbFxcVyuVzKyspSbm6u1/FDhw5pw4YNOnDggObMmaMZM2ZctO6JEydUWFioY8eO\nKSEhQY8++qiio6N97RIAAACAfoTZaV1z0WTx3//93yVJp0+f9rz+1okTJ3TTTTddtBGXy6WioiIt\nW7ZMVqtVS5culc1m09ChQz1loqOjdd9996mystLnuiUlJRozZoxyc3NVUlKikpISzZs3z6eOAwAA\nAADO76LJ4uLFi+V2u/Xzn/9cixcv9joWGxvr0zqLdXV1SkpKUmJioiQpMzNTlZWVXsliTEyMYmJi\nVFVV5XPdyspK/exnP5MkTZ48WT/72c9IFgEAAACgB1w0WRw9erQkqaioSAMHDrykRpqbm2W1Wj3b\nVqtVtbW13a7b2tqquLg4SWcT19bW1k7PUVpaqtLSUklSQUGB4uPjL6kf/nS0i+XP7UN36gaz7UDW\n/df63dHd9ztYbQdzjHVHb+xzd8cncYdu3WC23d9/Js/HbDb7VK4/flbd0R9/R/fHPgdTsOIO5vf+\n+fj6PRYMPt+zWFJSct5js2fP7pFgusNkMp13HUi73S673e7ZbmxsDFRYftOdPnS3/8FqO5hx98Z2\nu9t2MD+r7uitfSbu3lE3mG3zM9m5+Ph4v/SvP35W3dEff0f3xz4HE99jPcuX2aFSF5LFpqYmr+2W\nlhbt3btX119//UXrWiwWr/pNTU2yWCw+tXuhujExMXI4HIqLi5PD4dCQIUN8OicQLP1xDTxuJAcA\nAOidfE4W8/LyDPtqamr0wQcfXLRuenq66uvr1dDQIIvFooqKCi1ZssSndi9U12azqaysTLm5uSor\nK9P48eN97Q4AAP1Wf/zDFQCj7v5Bl++Svs/nZLEz11xzjQoLCy9aLjw8XAsWLNDKlSvlcrk0ZcoU\npaamaufOnZKk7OxstbS0KD8/X21tbTKZTHrnnXf0zDPPKCoqqtO6kpSbm6vCwkK99957nqUzAAAA\nAADd53OyePSo9+2cHR0d+uCDD3y+GTMjI0MZGRle+7Kzsz2vY2NjtWnTJp/rStLgwYO1fPlyn9oH\nAAAAAPjO52TxX6eNRkREaPjw4Vq0aFGPBwUAAAAACC6fk8Vt27b5Mw4AAACA++BwUYyRwOnSPYsu\nl0v79u2Tw+GQxWLRlVdeqbCwMH/FBgAAAPR5JD8IVT4ni1999ZVWr16tb775RhaLRc3NzRowYIAe\ne+wxXXHFFX4MEQAAAAAQaD4nixs3btStt96q6dOny2Qyye126+2339bGjRu1atUqf8YIAAAAAAgw\nn5PF+vp65eTkyGQySZJMJpOmTZumV155xW/BAQAASOefpne0071n9fapet1dAw8AusvnGw7Hjh2r\n3bt3e+3bvXu3xo4d2+NBAQAAAACC64JXFp999lnPlUSXy6Vf/vKXSktLk9VqVVNTk/7617/KZrMF\nJFAAAAAAQOBcMFlMSkry2k5NTfW8Hjp0qK699lr/RAUAAAAACKoLJot33XVXoOIAAAAAAISQCyaL\ne/fu1ejRoyVJn3766XnLXX311T0bFQAAAAAgqC6YLBYVFWnt2rWSzi6d0RmTyaTnnnuu5yMDAADA\nJWOhdwDddcFk8dtEUTr7sJuwMJ8fngoAAAAA6MV8yv5cLpfuueceffPNN/6OBwAAAAAQAnxKFsPC\nwpScnKzjx4/7Ox4AAAAAQAi44DTUc02cOFGrVq3S97//fVmtVs/6ixIPuAEAAACAvsbnZHHnzp2S\npFdeecVrPw+4AQAAAIC+x+dkcf369f6MAwAAAAAQQnx+vOkvfvGLTvevWbOmx4IBAAAAAIQGn5PF\nzz77rEv7AQAAAAC910WnoW7btk2S5HQ6Pa+/dfToUSUkJPgnMgAAAABA0Fw0WWxqapJ0dq3Fb19/\nKz4+XrNmzfKpoZqaGhUXF8vlcikrK0u5ublex91ut4qLi1VdXa2BAwcqLy9PaWlpOnz4sAoLCz3l\nGhoaNGvWLOXk5Gj79u169913NWTIEEnS3LlzlZGR4VM8AAAAAIDzu2iymJeXJ0kaOXKk7Hb7JTXi\ncrlUVFSkZcuWyWq1aunSpbLZbBo6dKinTHV1tY4cOaJ169aptrZWmzdv1tNPP63k5GStXr3ac56H\nHnpI119/vadeTk6OZsyYcUlxAQAAAAA65/PTUO12u06dOqXDhw+rvb3d69jF1lmsq6tTUlKSEhMT\nJUmZmZmqrKz0ShZ3796tSZMmyWQyaeTIkTp58qQcDofi4uI8Zf7yl78oKSmJqa8AAAAA4Gc+J4vv\nv/++ioqKFBkZqYiICM9+X9ZZbG5ultVq9WxbrVbV1tYaysTHx3uVaW5u9koWP/zwQ910001e9Xbs\n2KHy8nKlpaXp3nvvVXR0tKH90tJSlZaWSpIKCgq82gkVR7tY/tw+dKduMNsOZN1/rd8dfFb+rRvM\ntulz1/XGuPt7n4PZNp+Vf+sGs+3+GHd/7HMw2+4LfT4fs9kckvmJ1IVk8eWXX9ZPfvITjR071p/x\nnJfT6dTHH3+su+++27MvOztbM2fOlHT2QTxbtmzxTJs9l91u95pC29jY6P+A/aw7fehu/4PVdjDj\nDla7fFb9o24w2+6PcdPn3tN2f4y7P/Y5mG3T597Tdl/uc3x8fMD/n5qcnOxTOZ+XznC5XLr22msv\nKRiLxeL1cJympiZZLBZDmXPfpH8tU11dreHDhys2NtazLzY2VmFhYQoLC1NWVpb2799/SfEBAAAA\nALz5nCzefvvtevXVV+VyubrcSHp6uurr69XQ0CCn06mKigrZbDavMjabTeXl5XK73dq3b5+ioqIu\nOgXV4XB4Xu/atUupqaldjg0AAAAAYOTzNNS3335bLS0teuONNwz3BW7cuPGCdcPDw7VgwQKtXLlS\nLpdLU6ZMUWpqqnbu3Cnp7HTSsWPHqqqqSkuWLFFERITXdNL29nbt2bNHCxcu9Drv1q1bdfDgQZlM\nJiUkJBiOAwAAAAAujc/J4uLFi7vVUEZGhmENxOzsbM9rk8mkBx54oNO6kZGReuGFF3o8JgAAAABA\n53xOFkePHu3POAAAAAAAIcTnZNHpdOq1115TeXm5Z/3DSZMm6Y477pDZ7PNpAAAAAAC9gM9Z3tat\nW7V//349+OCDSkhI0LFjx/Tqq6/q1KlTmj9/vh9DBAAAAAAEms/J4p///GetXr1agwcPlnR2bY7h\nw4fr8ccfJ1lEr3PmwRldKh/+mzf8FAkAAAAQmnxeOsPtdvszDgAAAABACPH5yuKNN96oVatWaebM\nmYqPj1djY6NeffVV3XDDDf6MDwAAAAAQBD4ni/PmzdOrr76qoqIiORwOWSwW3XTTTbrzzjv9GR8A\nAAAAIAgumix+8cUX+vjjj/XDH/5Qs2fP1uzZsz3Htm7dqr/+9a8aOXKkX4MEAAAAAATWRe9Z/N3v\nfqerrrqq02NXX321XnvttR4PCgAAAAAQXBdNFg8ePKjrrruu02NjxozRgQMHejwoAAAAAEBwXTRZ\nbGtrk9Pp7PTYmTNn1NbW1uNBAQAAAACC66LJYkpKij755JNOj33yySdKSUnp8aAAAAAAAMF10WQx\nJydHzz//vD766CO5XC5Jksvl0kcffaTf/OY3ysnJ8XuQAAAAAIDAuujTUCdOnKiWlhatX79e33zz\njYYMGaJ//OMfGjBggGbNmqWJEycGIk4AAAAAQAD5tM7i9OnT9b3vfU/79u3TiRMnFB0drZEjRyoq\nKsrf8QEAAAAAgsCnZFGSoqKizvtUVAAAAABA33LRexYBAAAAAP0PySIAAAAAwIBkEQAAAABgQLII\nAAAAADAgWQQAAAAAGPj8NNTuqqmpUXFxsVwul7KyspSbm+t13O12q7i4WNXV1Ro4cKDy8vKUlpYm\nSVq0aJEiIyMVFham8PBwFRQUSJJOnDihwsJCHTt2TAkJCXr00UcVHR0dqC4BAAAAQJ8VkGTR5XKp\nqKhIy5Ytk9Vq1dKlS2Wz2TR06FBPmerqah05ckTr1q1TbW2tNm/erKefftpz/Mknn9SQIUO8zltS\nUqIxY8YoNzdXJSUlKikp0bx58wLRJQAAAADo0wIyDbWurk5JSUlKTEyU2WxWZmamKisrvcrs3r1b\nkyZNkslk0siRI3Xy5Ek5HI4LnreyslKTJ0+WJE2ePNlwTgAAAADApQnIlcXm5mZZrVbPttVqVW1t\nraFMfHy8V5nm5mbFxcVJklasWKGwsDBNnTpVdrtdktTa2uo5Hhsbq9bWVn93BQAAAAD6hYDds9gd\nK1askMViUWtrq5566iklJydr9OjRXmVMJpNMJlOn9UtLS1VaWipJKigo8EpKQ8XRLpY/tw/dqRvM\ntgNZN5ht81l1XW+Muz/2OZht0+dLrxvMtvms/Fs3mG33x7j7Y5+D2XZf6PP5mM3mkMxPpAAlixaL\nRU1NTZ7tpqYmWSwWQ5nGxsZOy3z7b0xMjMaPH6+6ujqNHj1aMTExcjgciouLk8PhMNzT+C273e65\nGinJq53eqjt96G7/g9V2f4y7P/Y5mG3T597TNn0OXN1gtt0f4+6PfQ5m2/S597Tdl/scHx8f8Pwk\nOTnZp3IBuWcxPT1d9fX1amhokNPpVEVFhWw2m1cZm82m8vJyud1u7du3T1FRUYqLi1N7e7va2tok\nSe3t7dqzZ4+GDRvmqVNWViZJKisr0/jx4wPRHQAAAADo8wJyZTE8PFwLFizQypUr5XK5NGXKFKWm\npmrnzp2SpOzsbI0dO1ZVVVVasmSJIiIilJeXJ+nsfYlr1qyRJJ05c0YTJ07UddddJ0nKzc1VYWGh\n3nvvPc/SGQAAAACA7gvYPYsZGRnKyMjw2pedne15bTKZ9MADDxjqJSYmavXq1Z2ec/DgwVq+fHnP\nBgoAAAAACMw0VAAAAABA70KyCAAAAAAwIFkEAAAAABiQLAIAAAAADEgWAQAAAAAGJIsAAAAAAAOS\nRQAAAACAAckiAAAAAMCAZBEAAAAAYECyCAAAAAAwIFkEAAAAABiQLAIAAAAADEgWAQAAAAAGJIsA\nAAAAAAOSRQAAAACAAckiAAAAAMCAZBEAAAAAYECyCAAAAAAwIFkEAAAAABiQLAIAAAAADEgWAQAA\nAAAG5kA1VFNTo+LiYrlcLmVlZSk3N9fruNvtVnFxsaqrqzVw4EDl5eUpLS1NjY2NWr9+vVpaWmQy\nmWS32zWBGvI5AAAQDUlEQVRt2jRJ0vbt2/Xuu+9qyJAhkqS5c+cqIyMjUF0CAAAAgD4rIMmiy+VS\nUVGRli1bJqvVqqVLl8pms2no0KGeMtXV1Tpy5IjWrVun2tpabd68WU8//bTCw8N1zz33KC0tTW1t\nbcrPz9c111zjqZuTk6MZM2YEohsAAAAA0G8EZBpqXV2dkpKSlJiYKLPZrMzMTFVWVnqV2b17tyZN\nmiSTyaSRI0fq5MmTcjgciouLU1pamiRp0KBBSklJUXNzcyDCBgAAAIB+KyDJYnNzs6xWq2fbarUa\nEr7m5mbFx8dfsExDQ4MOHDigESNGePbt2LFDjz32mDZs2KATJ074qQcAAAAA0L8E7J7F7mpvb9fa\ntWs1f/58RUVFSZKys7M1c+ZMSdK2bdu0ZcsW5eXlGeqWlpaqtLRUklRQUOCVlIaKo10sf24fulM3\nmG0Hsm4w2+az6rreGHd/7HMw26bPl143mG3zWfm3bjDb7o9x98c+B7PtvtDn8zGbzSGZn0gBShYt\nFouampo8201NTbJYLIYyjY2NnZZxOp1au3atbr75Zk2YMMFTJjY21vM6KytLq1at6rR9u90uu93u\n2T63nd6qO33obv+D1XZ/jLs/9jmYbdPn3tM2fQ5c3WC23R/j7o99Dmbb9Ln3tN2X+xwfHx/w/CQ5\nOdmncgGZhpqenq76+no1NDTI6XSqoqJCNpvNq4zNZlN5ebncbrf27dunqKgoxcXFye12a9OmTUpJ\nSdH06dO96jgcDs/rXbt2KTU1NRDdAQAAAIA+LyBXFsPDw7VgwQKtXLlSLpdLU6ZMUWpqqnbu3Cnp\n7HTSsWPHqqqqSkuWLFFERIRnOumXX36p8vJyDRs2TI8//rikfy6RsXXrVh08eFAmk0kJCQlauHBh\nILoDAAAAAH1ewO5ZzMjIMKyBmJ2d7XltMpn0wAMPGOqNGjVK27dv7/Scixcv7tkgAQAAAACSAjQN\nFQAAAADQu5AsAgAAAAAMSBYBAAAAAAYkiwAAAAAAA5JFAAAAAIABySIAAAAAwIBkEQAAAABgQLII\nAAAAADAgWQQAAAAAGJAsAgAAAAAMSBYBAAAAAAYkiwAAAAAAA5JFAAAAAIABySIAAAAAwIBkEQAA\nAABgQLIIAAAAADAgWQQAAAAAGJAsAgAAAAAMSBYBAAAAAAYkiwAAAAAAA5JFAAAAAICBOVAN1dTU\nqLi4WC6XS1lZWcrNzfU67na7VVxcrOrqag0cOFB5eXlKS0u7YN0TJ06osLBQx44dU0JCgh599FFF\nR0cHqksAAAAA0GcF5Mqiy+VSUVGR/vM//1OFhYX68MMP9fe//92rTHV1tY4cOaJ169Zp4cKF2rx5\n80XrlpSUaMyYMVq3bp3GjBmjkpKSQHQHAAAAAPq8gCSLdXV1SkpKUmJiosxmszIzM1VZWelVZvfu\n3Zo0aZJMJpNGjhypkydPyuFwXLBuZWWlJk+eLEmaPHmy4ZwAAAAAgEsTkGSxublZVqvVs221WtXc\n3GwoEx8fbyhzobqtra2Ki4uTJMXGxqq1tdWf3QAAAACAfsPkdrvd/m7kz3/+s2pqavTjH/9YklRe\nXq7a2lrdf//9njIFBQXKzc3VqFGjJEn/9V//pR/+8Ic6duzYeevOnz9fL774oucc9913n4qLiw3t\nl5aWqrS01NMOAAAAAODCAnJl0WKxqKmpybPd1NQki8ViKNPY2Ggoc6G6MTExcjgckiSHw6EhQ4Z0\n2r7dbldBQUGvTBTz8/ODHQL6OMYY/I0xBn9ifMHfGGPwt1AeYwFJFtPT01VfX6+GhgY5nU5VVFTI\nZrN5lbHZbCovL5fb7da+ffsUFRWluLi4C9a12WwqKyuTJJWVlWn8+PGB6A4AAAAA9HkBWTojPDxc\nCxYs0MqVK+VyuTRlyhSlpqZq586dkqTs7GyNHTtWVVVVWrJkiSIiIpSXl3fBupKUm5urwsJCvffe\ne56lMwAAAAAA3ReQexZx6UpLS2W324MdBvowxhj8jTEGf2J8wd8YY/C3UB5jJIsAAAAAAIOA3LMI\nAAAAAOhdAnLPIi5NTU2NiouL5XK5lJWVpdzc3GCHhF5uw4YNqqqqUkxMjNauXStJOnHihAoLC3Xs\n2DHPvb/R0dFBjhS9UWNjo9avX6+WlhaZTCbZ7XZNmzaNMYYec/r0aT355JNyOp06c+aMbrjhBs2a\nNYsxhh7lcrmUn58vi8Wi/Px8xhd63KJFixQZGamwsDCFh4eroKAgZMcZ01BDlMvl0iOPPKJly5bJ\narVq6dKleuSRRzR06NBgh4ZebO/evYqMjNT69es9yeLWrVsVHR2t3NxclZSU6MSJE5o3b16QI0Vv\n5HA45HA4lJaWpra2NuXn5+vxxx/X+++/zxhDj3C73ero6FBkZKScTqeWL1+u+fPna9euXYwx9Ji3\n3npL+/fv93yP8XsSPW3RokX6+c9/7rXsX6iOM6ahhqi6ujolJSUpMTFRZrNZmZmZqqysDHZY6OVG\njx5t+CtVZWWlJk+eLEmaPHky4wyXLC4uTmlpaZKkQYMGKSUlRc3NzYwx9BiTyaTIyEhJ0pkzZ3Tm\nzBmZTCbGGHpMU1OTqqqqlJWV5dnH+EIghOo4YxpqiGpubpbVavVsW61W1dbWBjEi9FWtra2Ki4uT\nJMXGxqq1tTXIEaEvaGho0IEDBzRixAjGGHqUy+XST3/6Ux05ckS33nqrrrzySsYYesyLL76oefPm\nqa2tzbOP8QV/WLFihcLCwjR16lTZ7faQHWckiwA8TCaTTCZTsMNAL9fe3q61a9dq/vz5ioqK8jrG\nGEN3hYWFafXq1Tp58qTWrFmjr7/+2us4YwyX6uOPP1ZMTIzS0tL02WefdVqG8YWesGLFClksFrW2\ntuqpp55ScnKy1/FQGmckiyHKYrGoqanJs93U1CSLxRLEiNBXxcTEyOFwKC4uTg6Hw2v+PNBVTqdT\na9eu1c0336wJEyZIYozBPy677DJ997vfVU1NDWMMPeLLL7/U7t27VV1drdOnT6utrU3r1q1jfKHH\nfft/+piYGI0fP151dXUhO864ZzFEpaenq76+Xg0NDXI6naqoqJDNZgt2WOiDbDabysrKJEllZWUa\nP358kCNCb+V2u7Vp0yalpKRo+vTpnv2MMfSUf/zjHzp58qSks09G3bNnj1JSUhhj6BF33323Nm3a\npPXr1+s//uM/dPXVV2vJkiWML/So9vZ2zzTn9vZ27dmzR8OGDQvZccbTUENYVVWVXnrpJblcLk2Z\nMkV33HFHsENCL/fLX/5Se/fu1fHjxxUTE6NZs2Zp/PjxKiwsVGNjY0g9qhm9zxdffKHly5dr2LBh\nnukzc+fO1ZVXXskYQ4/46quvtH79erlcLrndbt14442aOXOmjh8/zhhDj/rss8/05ptvKj8/n/GF\nHnX06FGtWbNG0tkHdU2cOFF33HFHyI4zkkUAAAAAgAHTUAEAAAAABiSLAAAAAAADkkUAAAAAgAHJ\nIgAAAADAgGQRAAAAAGBAsggAQBA1NjbqnnvukcvlCnYoAAB4YekMAECf9cEHH+itt97SoUOHNGjQ\nIF1xxRW64447NGrUqGCHFhDr16+X1WrVnDlz1NDQoIcfflgDBw6UJEVGRio9PV3Tpk3TNddcE+RI\nAQChyBzsAAAA8Ie33npLJSUlevDBB3XttdfKbDbrk08+0e7du/tNstiZF198UeHh4WppaVFFRYVW\nr16t+++/X7fcckuwQwMAhBiSRQBAn3Pq1Clt27ZNeXl5mjBhgmf/uHHjNG7cOElSXV2diouLdejQ\nIUVERGjChAn60Y9+JLP57K/GWbNm6f7779fbb7+tlpYWTZs2Tbfccouee+45/e1vf9O1116rJUuW\nyGw267PPPtOzzz6r7Oxsvf3224qMjNScOXN08803S5Kqqqr0v//7vzp69KiioqI0ZcoUzZo1S5I8\nV/xefvllhYeHq6GhQevXr9eBAwd05ZVX6t/+7d906tQpLVmyxFM2Ly9P27Zt0+nTp5WTk6M77rij\ny+9RbGyspk2bJqfTqf/+7//WpEmTFBbG3SkAgH/itwIAoM/Zt2+fvvnmG11//fXnLRMWFqYf/ehH\nKioq0lNPPaVPP/1Uv//9773KfPLJJyooKNDKlSv1xhtv6Pnnn9fixYu1ceNG/e1vf9MHH3zgKdvS\n0qLjx49r06ZNWrRokZ5//nkdPnxYkjRw4EA9/PDDKi4uVn5+vv7whz9o165dncb1q1/9Sunp6Xrh\nhRd011136f/+7/8MZb744gv96le/0hNPPKHf/va3+vvf/34pb5MkacKECWptbfXECgDAt0gWAQB9\nzvHjxzV48GCFh4eft0xaWppGjhyp8PBwXX755bLb7dq7d69XmRkzZigqKkqpqalKTU3VNddco8TE\nREVFRWns2LE6ePCgV/nZs2drwIABGj16tMaOHauKigpJ0ne/+10NGzZMYWFh+s53vqObbrrJ0JZ0\n9mE3+/fv1+zZs2U2mzVq1CjPldBz3XXXXYqIiNAVV1yh73znO/rqq68u4V06Ky4uTpJ04sSJSz4H\nAKBvYhoqAKDPGTx4sI4fP64zZ86cN2E8fPiwtmzZov379+v06dM6c+aM0tLSvMrExsZ6XkdERBi2\nW1paPNuXXXaZIiMjPdsJCQlyOBySpNraWv3P//yPvv76azmdTjmdTt1www2GmJqbmxUdHe15CI0k\nxcfHq7Gx8bxxDRw4UO3t7Rd8Py6kublZkhQdHX3J5wAA9E1cWQQA9DkjR47UgAEDVFlZed4ymzdv\nVnJystatW6eXXnpJc+fOVXceEH7y5EmvpK2xsdFz1W7dunUaN26cNm7cqJdeeklTp07ttK24uDid\nOHFCHR0dXufxp127dikmJkbJycl+bQcA0PuQLAIA+pyoqCjNmjVLRUVF2rVrlzo6OuR0OlVdXa2t\nW7dKktra2hQVFaXIyEgdOnRIO3fu7Ha727dvl9Pp1Oeff66qqirdeOONnraio6MVERGhuro6r3sd\nz5WQkKD09HS98sorcjqd2rdvnz7++ONux9WZlpYW7dixQ7/97W81d+5cHm4DADBgGioAoE+67bbb\nFBsbq9dee03PPvusIiMjlZaW5nly6D333KPnn39er7/+uoYPH67MzEx9+umnl9xebGysoqOj9dBD\nDykiIkIPPvigUlJSJEkPPPCAtmzZohdeeEGjR4/WjTfeqJMnT3Z6nsWLF2vDhg1asGCBRowYoczM\nTLlcrkuO61/Nnz9f0tnpq+np6frJT36i6667rsfODwDoO0zu7sy5AQAAnqUzNm3a1OPnLiwsVEpK\nimepDQAAAoU5JwAAhJC6ujodOXJELpdLNTU12r17t8aPHx/ssAAA/RDTUAEACCEtLS1au3atjh8/\nLqvVqgceeEDDhw8PdlgAgH6IaagAAAAAAAOmoQIAAAAADEgWAQAAAAAGJIsAAAAAAAOSRQAAAACA\nAckiAAAAAMCAZBEAAAAAYPD/UhgGf/38QhcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20dc63400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lstm_a = (attributions/campaign_freq)\n",
    "\n",
    "fig = plt.figure(figsize=(15, 4))\n",
    "ax = fig.add_subplot(111)\n",
    "plt.bar( range(len(lstm_a[campaign_idx])), lstm_a[campaign_idx], label='LSTM-A' )\n",
    "plt.xlabel('Campaign ID')\n",
    "plt.ylabel('Contribution')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA34AAAENCAYAAABQALdSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9//H3TIYhhCAwEyQNRIEAlyKKkLAYEEFiuAWX\nFAVEtFAQtKyltypY0PtTsCByUSCC1DTgA4tcRNwovZBaBE2VsIQWkZKwFVnNxpqQTGZ+f1CmDglh\nSGbJnLyef835zvfM+czkO5N5zznne0wul8slAAAAAIBhmYNdAAAAAADAvwh+AAAAAGBwBD8AAAAA\nMDiCHwAAAAAYHMEPAAAAAAyO4AcAAAAABkfwAwAAAACDI/gBAAAAgMER/AAAAADA4Ah+AAAAAGBw\nlmAXUFPHjx8Pdgk3LCoqSnl5ecEuAwbDuII/MK7gD4wr+APjCv4QCuMqJibGq37s8QMAAAAAgyP4\nAQAAAIDBEfwAAAAAwOBC/hy/q7lcLpWUlMjpdMpkMgW7nEqdOnVKly5dCnYZQeVyuWQ2mxUeHl5r\n/04AAACAURgu+JWUlKhevXqyWGrvU7NYLAoLCwt2GUHncDhUUlKiBg0aBLsUAAAAwNAMd6in0+ms\n1aEP/2axWOR0OoNdBgAAAGB4hgt+HDYYWvh7AQAAAP5nuOAHAAAAAPBk+GMiyxe97NPHC5s087p9\n2rVrp5ycHPfyggUL9Mc//lGStG/fPv34xz+Wy+XSiBEjNGrUKEnSvffeq9tuu02LFi3yab0AAAAA\nYPjgVxtMnTpVU6dOlcPh0O23367PPvtMDofDff+3336rsLAwZWZmqri4mMlOAAAAAC/N2ny00vYZ\nfWMDXEntxqGetcCHH36oRx55RImJidq0aVOwywEAAABgMAS/WuDTTz/Vgw8+qJSUFH300UfBLgcA\nAACAwRD8gmzHjh2Kjo7Wj370I/Xp00fZ2dk6e/ZssMsCAAAAYCAEvyD76KOPtG/fPvXo0UO9e/fW\nuXPntGHDhmCXBQAAAMBAmNwliJxOp9avX6/NmzerWbNmkqQtW7ZoyZIlGjZsWJCrAwAAAGAUhg9+\n3lx+wdeKi4sVHx/vXh43bpyeeuqpCv2+/PJLxcbGukOfJCUmJmrSpEn6/vvvPdoBAAAAoLoMH/yC\n4bvvvqu03WKx6Ntvv3Uv33333br77rsr9Nm9e7df6wMAAABQt3COHwAAAAAYHMEPAAAAAAyO4AcA\nAAAABkfwAwAAAACDI/gBAAAAgMER/AAAAADA4Ax/OYdZm4/69PFm9I29bp927dopJydHkvTnP/9Z\nL774ot577z21bNnSp7VU5pFHHtHMmTPVuXNnnz7mqVOnVL9+fVmtVr366qvq1KmTzx4fAAAAgH+x\nx8+Ptm7dqhdeeEErV670OvQ5HA4/V1U9ixcvVkZGhkaOHKlZs2YFuxwAAAAAN4Dg5ydfffWVnn32\nWa1YsUKtWrWSJOXn52vs2LEaMGCABg4cqKysLEnS/PnzNWnSJD300EOaPHmyjh49qp/+9KcaMGCA\nBgwY4O536tQpDR48WPfdd5/uvfdeff31117VUlJSoqlTp6p///5KTk7Wl19+KUkqLi7WU089pb59\n+2rMmDG6//77r3vx+Pj4eJ08edK9/Pnnn+uBBx7QgAEDNG7cOF24cEHS5T2dffr00X/+539q5syZ\n+tnPfnZDrx8AAAAA3zH8oZ7BUFpaqjFjxmjNmjVq27atu/2FF17Q2LFjlZiYqCNHjuixxx7T559/\nLknKycnRunXr1KBBAxUXF2vVqlUKDw/XwYMHNWHCBG3YsEHr1q3TPffcoylTpqi8vFzFxcVe1bN8\n+XKZTCb9+c9/Vm5uroYPH66tW7dqxYoVaty4sTZv3qx9+/YpOTn5uo/1l7/8RQMGDJAkFRQU6I03\n3tDq1asVERGh1NRULVu2TL/4xS/03HPP6YMPPtAtt9yi8ePHV+NVBAAAAOArBD8/sFgsio+P13vv\nvaeXXnrJ3b5161bt379fJpNJLpdL58+fd+8hS05OVoMGDSRJZWVl+s1vfqO9e/fKbDbr4MGDkqQ7\n77xT//Vf/yWHw6EBAwZ4fZ5dVlaWfv7zn0uS2rZtq5YtW+rgwYPatm2bxowZI0nq0KGDfvzjH1/z\nMSZOnKiysjJduHBBGzdulCTt2LFD+/fv10MPPeSuOz4+Xrm5ubr11lt1yy23SJJSUlK0cuVKr18/\nAAAAAL7FoZ5+YDab9dZbb2nXrl1auHChu93pdOqTTz7RZ599pk2bNmnHjh1q2LChJCkiIsLd73e/\n+52aNWumTZs2acOGDSorK5Mk9ezZU2vXrlV0dLSmTp2qNWvWBOw5LV68WH/96181ZMgQzZw5U5Lk\ncrnUp08fbdq0SZs2bdLmzZs1f/78gNUEAAAAwDsEPz9p0KCB3nnnHa1bt06rVq2SJN1zzz1KT093\n99mzZ0+l6549e1Y333yzzGaz1q5dq/LycknSd999p2bNmmnEiBF67LHH9Pe//92rWrp3765169ZJ\nkg4cOKBjx44pLi5O3bp10yeffCJJ2r9/v/bt21fl45hMJj3zzDPauXOncnNzFR8fr6ysLB06dEiS\ndPHiRR04cEBxcXE6cuSIjh69PKPqxx9/7FWdAAAAAPzD8Id6enP5BX9p2rSpVq5cqYcfflh2u10v\nv/yynn/+efXt21cOh0M9evTQ3LlzK6w3cuRIjRs3Tu+//7769evn3huYmZmppUuXymKxqGHDhnrj\njTcq3e7PfvYzWSyX/7Tx8fFauHChpk+frv79+yssLEwLFixQ/fr1NXLkSE2ZMkV9+/ZV27Zt1b59\nezVq1KjK59SgQQONGzdOS5Ys0fz587VgwQJNmDBBpaWlkqRnn31WcXFxeuWVVzRixAhFRET49NIS\nAAAAAG6cyeVyuQKxoezsbKWnp8vpdKp///5KSUmptF9ubq5mzJihX/7yl+rZs+d1H/f48eMeyxcv\nXvQ4bLI2slgsteKyDeXl5SorK1N4eLgOHz6sRx99VFu2bJHVaq3xY1+4cEENGzaUy+XS888/r9at\nW2vcuHEV+oXC3ytUREVFKS8vL9hlwGAYV/AHxhX8gXFVd13rut2+2AEUCuMqJibGq34B2ePndDqV\nlpamGTNmyG63a/r06UpISKhwbTun06l3332XPUQBUlxcrCFDhrjPIXzllVd8Evok6d1339WaNWtU\nVlamTp066YknnvDJ4wIAAAC4cQEJfrm5uYqOjlbz5s0lSYmJicrKyqoQ/DZs2KAePXrowIEDgSir\nzouMjNSGDRv88tjjxo2rdA8fAAAAgMALyOQuBQUFstvt7mW73a6CgoIKfbZt2+bVteSqEqAjV+Ej\n/L0AAAAA/6s1k7ssX75cI0aMkNlcdRbNyMhQRkaGJGnOnDmKioryuN9kMsnpdKpevXp+q9UXrky+\nUpeVlZUpMjLS40cBVJ/FYqnwfgBqinEFf2BcwR8YV3WX1Xqq0nZfjAcjjauApA+bzab8/Hz3cn5+\nvmw2m0efAwcOuGepPHv2rHbt2iWz2azu3bt79EtKSlJSUpJ7+eqTLV0ul0pKSnTx4kWZTCZfPxWf\nqF+/vi5duhTsMoLK5XLJbDYrPDy81p8wGypC4eRjhB7GFfyBcQV/YFzVXVdml7+aL8ZDKIyrWjW5\nS1xcnE6cOKHTp0/LZrMpMzNTkydP9uiTmprqcTs+Pr5C6POGyWRSgwYNalyzP4XCAAIAAABgHAEJ\nfmFhYRo9erRmz54tp9Opfv36KTY2Vhs3bpSkGp/XBwAAAAC4toCdaNa1a1d17drVo+1agW/ChAmB\nKAkAAAAA6oSAzOoJAAAAAAgegh8AAAAAGBzBDwAAAAAMjuAHAAAAAAZH8AMAAAAAgyP4AQAAAIDB\nBexyDgAAAFUpX/Rype1hk2YGuBIAMB72+AEAAACAwRH8AAAAAMDgCH4AAAAAYHAEPwAAAAAwOCZ3\nAQAAAGAorkM5Kv/78grtdXmyKPb4AQAAAIDBEfwAAAAAwOA41NPguCYSAAAAAIIfgDqJH0UAAEBd\nwqGeAAAAAGBwBD8AAAAAMDiCHwAAAAAYHMEPAAAAAAyO4AcAAAAABkfwAwAAAACDI/gBAAAAgMER\n/AAAAADA4Ah+AAAAAGBwBD8AAAAAMDhLsAsAAAAAgKuVL3q50vawSTMDXIkxsMcPAAAAAAyO4AcA\nAAAABkfwAwAAAACDI/gBAAAAgMExuQtqJU7mBfyD9xYAAHUTe/wAAAAAwOC83uPncDi0efNmHT58\nWCUlJR73TZw40eeFAQAAAAB8w+vgt3jxYh05ckTx8fFq3LixP2sCAAAAAPiQ18Fv9+7dWrx4sRo2\nbOjPegAAAAAAPuZ18IuKilJZWVm1N5Sdna309HQ5nU71799fKSkpHvdnZWVp9erVMplMCgsL06hR\no9ShQ4dqbw8AAAAAcJnXwa9Pnz6aN2+efvKTn6hJkyYe93Xq1KnKdZ1Op9LS0jRjxgzZ7XZNnz5d\nCQkJatmypbvP7bffroSEBJlMJh05ckQLFizQ66+/foNPBwAAAABwNa+D35/+9CdJ0qpVqzzaTSaT\nFi9eXOW6ubm5io6OVvPmzSVJiYmJysrK8gh+4eHh7tuXLl2SyWTytjQAAAAAQBW8Dn6pqanV3khB\nQYHsdrt72W63Kycnp0K/bdu26Q9/+IPOnDmj6dOnV3t7AAAAAIB/u6ELuJeXl+sf//iHO8i1b99e\nYWFhPiume/fu6t69u/bu3avVq1dr5syKFxTOyMhQRkaGJGnOnDmKiory2fYDxWKxBKzuQqu10vam\ntfx1C9W6gymQ48oI6uoYu9HnzbiCP1xrXNXV9yV8g88r4/H2M8FqPVWhT6nZLGsl69/o54mRxpXX\nwe/YsWOaO3euSktLZbfblZ+fr3r16um5557zOGSzMjabTfn5+e7l/Px82Wy2a/bv2LGj3nzzTZ09\ne1Y33XSTx31JSUlKSkpyL+fl5Xn7FGqNqKiogNVdXlpaaXttf91Cte5gCuS4MoK6OsZu9HkzruAP\n1xpXdfV9Cd/g88p4vP1MKK2kn8vprLT9RsdIKIyrmJgYr/p5HfzefvttJSUl6YEHHnCff/fxxx8r\nLS1NL774YpXrxsXF6cSJEzp9+rRsNpsyMzM1efJkjz4nT55U8+bNZTKZdPDgQZWVlalRo0belgcA\nAAAAuAavg9/hw4c1c+ZMj0lXBg0apHXr1l133bCwMI0ePVqzZ8+W0+lUv379FBsbq40bN0qSkpOT\n9dVXX2nLli0KCwuT1WrV1KlTmeAFAAAAAHzA6+Bns9m0d+9ej0s3fPvtt2ratKlX63ft2lVdu3b1\naEtOTnbfTklJqXBtPwAAAABAzXkd/IYPH665c+cqPj7efazrzp07NWnSJH/Whxs0a/NRj2VXZC9J\n0vPnvwxGOQAAAABqAa+DX0JCgubOnau//vWvKiwsVGxsrIYOHer1yYQAAAAAgOC4ocs5xMTE6OGH\nH/ZXLQAAAAAAP6gy+L311lt66qmnJEmLFi265mQrEydO9H1lAAAg5JQvernS9rBJFa/Ni9DF3/nG\n8Zoh2KoMfjfffLP7dnR0tN+LAQAAAAD4XpXB76c//an79n333acmTZpU6FNUVOT7qgAAAAAAPmP2\ntuOUKVMqbZ86darPigEAAAAA+J7Xwc/lclVou3jxosxmrx8CAAAAABAE153V8xe/+IUkqbS01H37\nivPnz6tXr17+qQwAAAAA4BPXDX6TJk2Sy+XSb3/72woXa2/SpAnX8QMAADC4WZuPeiy7Ii//8P/8\n+S+DUQ6Aarhu8OvYsaMkKS0tTfXr1/d7QQAAAAAA3/L6Au4ffvjhNe8bNmyYT4oBAAAINVyfDUAo\n8Dr45efneywXFRVp79696t69u8+LAgAAAAD4jtfBb/z48RXasrOz9cUXX/i0IAAAAACAb3kd/Cpz\nxx13aMGCBb6qBQDqhFA9LCxU6wYAADcQ/E6dOuWxfOnSJX3xxReKioryeVEAAABAXXX1LKpXzOgb\nG+BKYCReB7/Jkyd7LFutVrVu3VoTJkzweVEAAAAAAN/xOvitXr3an3UAAAAAAPzkhs7xczqd2r9/\nvwoLC2Wz2dSuXTuZzWZ/1QYAAAAA8AGvg9+RI0c0b948lZWVyWazqaCgQPXq1dOvf/1rtWrVyo8l\nAgAAAABqwuvgt2TJEg0YMED333+/TCaTXC6X1q9fryVLlmju3Ln+rBEAAAAAUANeB78TJ05o0KBB\nMplMkiSTyaSBAwdqzZo1fisOoY2p3wEAAIDawevg16VLF23fvl3du3d3t23fvl1dunTxS2EAAKB2\nq2zKeVdkLz1//ssgVAMAvvHDnReFVqvKS0slhf7OiyqD36JFi9x7+JxOp15//XW1adNGdrtd+fn5\nOnjwoBISEgJSKIzt6i8PrsheksSXB9QKXE8JAACEuiqDX3R0tMdybOy/v+S0bNlSnTt39k9VAAAA\ndQCnRQAIlCqD35AhQwJVBwAAAADAT6oMfnv37lXHjh0lSXv27Llmv06dOvm2KiCI+PUVgK/weQIA\nqC2qDH5paWmaP3++pMuXc6iMyWTS4sWLfV8ZgDqBL8YAAAD+V2XwuxL6pMsTvZjNZr8XBAD+wARC\nAFD3MDkX8G9eJTmn06knnnhCZWVl/q4HAAAAAOBjXl3Hz2w2KyYmRufOnZPNZvN3TQBgKOxtDBx+\n3QcAoHJeX8C9d+/emjt3rn7yk5/Ibre7r+8nMbkLah/OGwNwhetQjsr/vrxCO58HAIC6xOvgt3Hj\nRknSmjVrPNqZ3AVAXUSYAGqOveEArqjsiA1XZC8+D3zI6+CXmprqzzoAAICPcNQDAOBqXge/V199\nVc8++2yF9tdee02//vWvfVoUgKpV9quY1XpKzyY2D8q2Jc6hAmqC9xWAqrgO5UhShSNN+DEHN8Lr\n6zN88803N9QOAAAAAKgdrrvHb/Xq1ZIkh8Phvn3FqVOn1KxZM/9UBgAA4CUObwWAql03+OXn50u6\nfC2/K7eviIqK0tChQ73aUHZ2ttLT0+V0OtW/f3+lpKR43L9161Z99NFHcrlcatCggZ588km1atXK\ny6cBAABQ+zGhDYBguW7wGz9+vCSpffv2SkpKqtZGnE6n0tLSNGPGDNntdk2fPl0JCQlq2bKlu8/N\nN9+s//7v/1ZkZKR27dqlZcuW6ZVXXqnW9uA7/IIKAMHBzLEAAF/yenKXpKQkXbx4UcePH1dJSYnH\nfde7jl9ubq6io6PVvPnliScSExOVlZXlEfz+4z/+w327Xbt2FfYu1iWhepI/v2ICtQtTYwMAgCu8\nDn6bN29WWlqawsPDZbVa3e3eXMevoKBAdrvdvWy325WTk3PN/p999pm6dOlS6X0ZGRnKyMiQJM2Z\nM0dRUVHePoVaw2KxVFm31Xqq0nZvnuvV65aazf9qt3q0N/XydSu8ar2q1q/JtoNZty/XDZTKxojJ\nZA7I+6Em47MygXq9fTk+r6x/9brerl/T8e2tmtYteX5e1eh9NfuZytf9zbwKbc9+XPmkYa8+eNt1\nt+OL5+xL3r5mvq47WO8rybu6r4wrX743gvnZHaj/lcEUCnVX9v3K1/+zaqIujJOaqMnnoD8/Q80m\nk/txQv319jr4rVq1Sr/61a+uGch8Zc+ePfrLX/6il156qdL7k5KSPA45zcvL82s9/hAVFVVl3aWl\npR7LV6bwPfXiMo/2yg73qbCu01lpu7evW/lV61W1fk22Hcy6fbluoFz9ukiX/xkEosbKti1V//UJ\n1Ovty/F5Zf3K2gMxvr1V07olz8+rQL2vKq37UE6Fz0Cp4uegL56zL3n7vH1dd7DeV5J3dV8ZV758\nbwTzsztQ/yuDKRTqruz7la//Z9VEoMZJqB45VpPPQX9+hlqtVvfj1Lb35RUxMTFe9fM6+DmdTnXu\n3LlaxdhsNo9DN/Pz82Wz2Sr0O3LkiN566y1Nnz5djRo1qta2AACA8XF6Qd11rfkH9P/eCGwhdQTz\nPRiH19fxe+ihh7R27Vo5//WLw42Ii4vTiRMndPr0aTkcDmVmZiohIcGjT15enl577TVNnDjR69QK\nAAAAALg+r/f4rV+/XkVFRfr4448VGRnpcd+SJUuqXDcsLEyjR4/W7Nmz5XQ61a9fP8XGxmrjxo2S\npOTkZL3//vs6f/683n77bfc6c+bMudHnAwAAgFqOvUhA4Hkd/CZNmlSjDXXt2lVdu3b1aEtOTnbf\nfvrpp/X000/XaBsAAACoXZhhGKgdvA5+HTt29GcdAAAAAAA/8Tr4ORwOffDBB9qyZYsKCwvVtGlT\n9enTR4MHD5bF4vXDAEC1hepMZQCMxSgTy3C4JVC3eJ3YVq5cqQMHDmjs2LFq1qyZvv/+e61du1YX\nL17UqFGj/FgiAAAAAKAmvA5+X331lebNm+e+zEJMTIxat26tZ555huAHAAAAALWY18HP5XL5sw4A\nQC0za/NRWa2n3BeuDdXD2YCrcdg4gLrI6+B31113ae7cuXrkkUcUFRWlvLw8rV27Vj179vRnfUBI\n4XwJAAhNrkM5Kv/78grtfH4DMAqvg9/jjz+utWvXKi0tTYWFhbLZbOrVq5cefvhhf9YHwA/4tRvw\nDj/mAACM4rrBb9++fdqxY4dGjBihYcOGadiwYe77Vq5cqYMHD6p9+/Z+LRLA9ZXmfKvyHcsqtPMF\nFcFglFkPAQAwiusGv3Xr1mnAgAGV3tepUyd98MEHmjZtms8LAwKBi8oC8BU+TwCEOj7HjM18vQ6H\nDx/WnXfeWel9t99+uw4dOuTzogAAAAAAvnPdPX7FxcVyOByyWq0V7isvL1dxcbFfCkPg8SsPAAAA\nYEzXDX4tWrTQ7t271a1btwr37d69Wy1atPBLYQAAAEBtwoRPCGXXPdRz0KBBWrZsmb7++ms5nU5J\nktPp1Ndff63f/e53GjRokN+LBAAAAABU33X3+PXu3VtFRUVKTU1VWVmZbrrpJp09e1b16tXT0KFD\n1bt370DUCQAAAACoJq+u43f//ffr3nvv1f79+3X+/HlFRkaqffv2ioiI8Hd9AAAAAIAa8voC7hER\nEdec3RMAEBicXwIAAKrD6+AHAAAABNrVs467IntJErOOAzeI4AfUIsHam+M6lHN5+39fHtDtAgAA\nIDAIfgAA/At7FgAARkXwAwAACFGc9wvAWwQ/AACAOoA92oFz9WstXX69ea0RTNe9gDsAAAAAILQR\n/AAAAADA4DjUEwAAALgKh2vCaAh+AAAAAHyOyYdqF4IfAAAA6gSuW4u6jHP8AAAAAMDg2OMH1ABT\nY1/mOpRT4ddTiV9QAQAAagv2+AEAAACAwRH8AAAAAMDgONTTDyqb/leSZvSNDXAlAAAAAEDwAxDC\nmJ0NAADAOwQ/hDwusAoAAIDqqiuT9XGOHwAAAAAYHMEPAAAAAAyOQz0D5IfXOSu0WlVeWiqJc5EA\nAAAA+F/Agl92drbS09PldDrVv39/paSkeNx/7Ngxvfnmmzp06JAeffRRPfjgg4EqDQAA/ED5opcr\nbefHSgAIXQEJfk6nU2lpaZoxY4bsdrumT5+uhIQEtWzZ0t0nMjJSP//5z5WVlRWIkoCgY1IaeKOu\nnHAOAAD8KyDn+OXm5io6OlrNmzeXxWJRYmJihYDXuHFjtW3bVmFhYYEoCQAAAADqjIDs8SsoKJDd\nbncv2+125eTkBGLTAAAAQJ32w7kmfojDt+uWkJvcJSMjQxkZGZKkOXPmKCoqKsgVVWS1nqrQVmo2\ny2q1SpLMJpP7dtNK6r96/VKz+V/tVo92f697dd2B3Hag6q5Nz7mmdZdVst3aVndlCitZz9tth8IY\nC/W6TaZ/9w+FumvT3yqQdV/tRt5XNVm/unVbLBZFRUWF9HsjGHVXhs/Qf7syrmpD3bVpjAWy7qvV\nZHzeyLb9+Xpf73t7KAlI8LPZbMrPz3cv5+fny2azVeuxkpKSlJSU5F7Oy8urcX2+VvqvGTt/yOV0\nututVqv7dmX1X72+y+mstN3f615ddyC3Hai6a9Nzrmnd12qvTXVXpvwaz8UoYyzU6/7h51Uo1F2b\n/laBrLvCuaDWbpIqngvq7f9Mb9+X1a07KipKeXl5If3eCEbdlZ4bbu1W6Tm/tanuQL3eDofjumM0\nVN7ToVr31WryP/5Gtu3P1/t639trg5iYGK/6BeQcv7i4OJ04cUKnT5+Ww+FQZmamEhISArFpAAAA\nAKjzArLHLywsTKNHj9bs2bPldDrVr18/xcbGauPGjZKk5ORkFRUVadq0aSouLpbJZNIf//hH/c//\n/I8iIiICUSIAAAAAGFbAzvHr2rWrunbt6tGWnJzsvt2kSRMtXbo0UOUAAAAAQJ0RkEM9AQAAAADB\nQ/ADAAAAAIMj+AEAAACAwRH8AAAAAMDgCH4AAAAAYHABm9UTAAAAgHHN2nzUY9kV2UuS9Pz5L4NR\nDq7CHj8AAAAAMDj2+AEAUIdd/Qu9dPlXen6hBwBjIfgBCBi+YAIAAAQHh3oCAAAAgMER/AAAAADA\n4Ah+AAAAAGBwBD8AAAAAMDiCHwAAAAAYHMEPAAAAAAyO4AcAAAAABkfwAwAAAACDI/gBAAAAgMER\n/AAAAADA4Ah+AAAAAGBwBD8AAAAAMDiCHwAAAAAYHMEPAAAAAAyO4AcAAAAABkfwAwAAAACDI/gB\nAAAAgMER/AAAAADA4Ah+AAAAAGBwBD8AAAAAMDiCHwAAAAAYHMEPAAAAAAyO4AcAAAAABkfwAwAA\nAACDI/gBAAAAgMER/AAAAADA4Ah+AAAAAGBwlkBtKDs7W+np6XI6nerfv79SUlI87ne5XEpPT9eu\nXbtUv359jR8/Xm3atAlUeQAAAABgWAHZ4+d0OpWWlqbnn39eCxYs0JdffqnvvvvOo8+uXbt08uRJ\nLVy4UOPGjdPbb78diNIAAAAAwPACEvxyc3MVHR2t5s2by2KxKDExUVlZWR59tm/frj59+shkMql9\n+/a6cOGCCgsLA1EeAAAAABhaQIJfQUGB7Ha7e9lut6ugoKBCn6ioqCr7AAAAAABunMnlcrn8vZGv\nvvpK2dnXCT4PAAALRUlEQVTZevrppyVJW7ZsUU5OjsaMGePuM2fOHKWkpKhDhw6SpJdeekkjRoxQ\nXFycx2NlZGQoIyPDvQ4AAAAAoGoB2eNns9mUn5/vXs7Pz5fNZqvQJy8vr8o+kpSUlKQ5c+aEdOib\nNm1asEuAATGu4A+MK/gD4wr+wLiCPxhpXAUk+MXFxenEiRM6ffq0HA6HMjMzlZCQ4NEnISFBW7Zs\nkcvl0v79+xUREaGmTZsGojwAAAAAMLSAXM4hLCxMo0eP1uzZs+V0OtWvXz/FxsZq48aNkqTk5GR1\n6dJFO3fu1OTJk2W1WjV+/PhAlAYAAAAAhhew6/h17dpVXbt29WhLTk523zaZTHryyScDVU5QJSUl\nBbsEGBDjCv7AuII/MK7gD4wr+IORxlVAJncBAAAAAARPQM7xAwAAAAAET8AO9YSUnZ2t9PR0OZ1O\n9e/fXykpKcEuCSHqzTff1M6dO9W4cWPNnz9fknT+/HktWLBA33//vZo1a6apU6cqMjIyyJUiVOTl\n5Sk1NVVFRUUymUxKSkrSwIEDGVeokdLSUr344otyOBwqLy9Xz549NXToUMYVfMLpdGratGmy2Wya\nNm0a4wo+MWHCBIWHh8tsNissLExz5swxzNjiUM8AcTqdmjJlimbMmCG73a7p06drypQpatmyZbBL\nQwjau3evwsPDlZqa6g5+K1euVGRkpFJSUvThhx/q/Pnzevzxx4NcKUJFYWGhCgsL1aZNGxUXF2va\ntGl65plntHnzZsYVqs3lcunSpUsKDw+Xw+HQCy+8oFGjRmnbtm2MK9TYp59+qgMHDrg/s/g/CF+Y\nMGGCfvvb3+qmm25ytxllbHGoZ4Dk5uYqOjpazZs3l8ViUWJiorKysoJdFkJUx44dK/zSlJWVpXvu\nuUeSdM899zC+cEOaNm2qNm3aSJIaNGigFi1aqKCggHGFGjGZTAoPD5cklZeXq7y8XCaTiXGFGsvP\nz9fOnTvVv39/dxvjCv5ilLHFoZ4BUlBQILvd7l622+3KyckJYkUwmjNnzrivfdmkSROdOXMmyBUh\nVJ0+fVqHDh1S27ZtGVeoMafTqeeee04nT57UgAED1K5dO8YVamz58uV6/PHHVVxc7G5jXMFXXn75\nZZnNZt13331KSkoyzNgi+AEGZDKZZDKZgl0GQlBJSYnmz5+vUaNGKSIiwuM+xhWqw2w2a968ebpw\n4YJee+01/fOf//S4n3GFG7Vjxw41btxYbdq00TfffFNpH8YVquvll1+WzWbTmTNnNGvWLMXExHjc\nH8pji+AXIDabTfn5+e7l/Px82Wy2IFYEo2ncuLEKCwvVtGlTFRYWehybDnjD4XBo/vz5uvvuu9Wj\nRw9JjCv4TsOGDXXbbbcpOzubcYUa+cc//qHt27dr165dKi0tVXFxsRYuXMi4gk9c+X7euHFjdevW\nTbm5uYYZW5zjFyBxcXE6ceKETp8+LYfDoczMTCUkJAS7LBhIQkKCPv/8c0nS559/rm7dugW5IoQS\nl8ulpUuXqkWLFrr//vvd7Ywr1MTZs2d14cIFSZdn+Pzb3/6mFi1aMK5QI4899piWLl2q1NRU/fKX\nv1SnTp00efJkxhVqrKSkxH34cElJif72t7/plltuMczYYlbPANq5c6dWrFghp9Opfv36afDgwcEu\nCSHq9ddf1969e3Xu3Dk1btxYQ4cOVbdu3bRgwQLl5eWF9FTDCI59+/bphRde0C233OI+hGX48OFq\n164d4wrVduTIEaWmpsrpdMrlcumuu+7SI488onPnzjGu4BPffPONPvnkE02bNo1xhRo7deqUXnvt\nNUmXJ6Tq3bu3Bg8ebJixRfADAAAAAIPjUE8AAAAAMDiCHwAAAAAYHMEPAAAAAAyO4AcAAAAABkfw\nAwAAAACDI/gBAOBDeXl5euKJJ+R0OoNdCgAAblzOAQAQMr744gt9+umnOnbsmBo0aKBWrVpp8ODB\n6tChQ7BLC4jU1FTZ7XY9+uijOn36tCZOnKj69etLksLDwxUXF6eBAwfqjjvuCHKlAIDaxhLsAgAA\n8Mann36qDz/8UGPHjlXnzp1lsVi0e/dubd++vc4Ev8osX75cYWFhKioqUmZmpubNm6cxY8aob9++\nwS4NAFCLEPwAALXexYsXtXr1ao0fP149evRwt8fHxys+Pl6SlJubq/T0dB07dkxWq1U9evTQyJEj\nZbFc/lc3dOhQjRkzRuvXr1dRUZEGDhyovn37avHixTp69Kg6d+6syZMny2Kx6JtvvtGiRYuUnJys\n9evXKzw8XI8++qjuvvtuSdLOnTv13nvv6dSpU4qIiFC/fv00dOhQSXLviVu1apXCwsJ0+vRppaam\n6tChQ2rXrp1+9KMf6eLFi5o8ebK77/jx47V69WqVlpZq0KBBGjx48A2/Rk2aNNHAgQPlcDj07rvv\nqk+fPjKbOaMDAHAZ/xEAALXe/v37VVZWpu7du1+zj9ls1siRI5WWlqZZs2Zpz549+r//+z+PPrt3\n79acOXM0e/Zsffzxx1q2bJkmTZqkJUuW6OjRo/riiy/cfYuKinTu3DktXbpUEyZM0LJly3T8+HFJ\nUv369TVx4kSlp6dr2rRp2rRpk7Zt21ZpXW+88Ybi4uL0+9//XkOGDNHWrVsr9Nm3b5/eeOMNzZw5\nU++//76+++676rxMkqQePXrozJkz7loBAJAIfgCAEHDu3Dk1atRIYWFh1+zTpk0btW/fXmFhYbr5\n5puVlJSkvXv3evR58MEHFRERodjYWMXGxuqOO+5Q8+bNFRERoS5duujw4cMe/YcNG6Z69eqpY8eO\n6tKlizIzMyVJt912m2655RaZzWbdeuut6tWrV4VtSZcnejlw4ICGDRsmi8WiDh06uPdQ/tCQIUNk\ntVrVqlUr3XrrrTpy5Eg1XqXLmjZtKkk6f/58tR8DAGA8HOoJAKj1GjVqpHPnzqm8vPya4e/48eN6\n5513dODAAZWWlqq8vFxt2rTx6NOkSRP3bavVWmG5qKjIvdywYUOFh4e7l5s1a6bCwkJJUk5Ojv7w\nhz/on//8pxwOhxwOh3r27FmhpoKCAkVGRronYJGkqKgo5eXlXbOu+vXrq6SkpMrXoyoFBQWSpMjI\nyGo/BgDAeNjjBwCo9dq3b6969eopKyvrmn3efvttxcTEaOHChVqxYoWGDx+umkxcfeHCBY8AlpeX\n596btnDhQsXHx2vJkiVasWKF7rvvvkq31bRpU50/f16XLl3yeBx/2rZtmxo3bqyYmBi/bgcAEFoI\nfgCAWi8iIkJDhw5VWlqatm3bpkuXLsnhcGjXrl1auXKlJKm4uFgREREKDw/XsWPHtHHjxhpv93//\n93/lcDj07bffaufOnbrrrrvc24qMjJTValVubq7HuYE/1KxZM8XFxWnNmjVyOBzav3+/duzYUeO6\nKlNUVKQ//elPev/99zV8+HAmdgEAeOBQTwBASHjggQfUpEkTffDBB1q0aJHCw8PVpk0b9wyYTzzx\nhJYtW6aPPvpIrVu3VmJiovbs2VPt7TVp0kSRkZF66qmnZLVaNXbsWLVo0UKS9OSTT+qdd97R73//\ne3Xs2FF33XWXLly4UOnjTJo0SW+++aZGjx6ttm3bKjEx0acXdx81apSky4eIxsXF6Ve/+pXuvPNO\nnz0+AMAYuIA7AABXuXI5h6VLl/r8sRcsWKAWLVq4L/8AAEAgcBwIAAB+lJubq5MnT8rpdCo7O1vb\nt29Xt27dgl0WAKCO4VBPAAD8qKioSPPnz9e5c+dkt9v15JNPqnXr1sEuCwBQx3CoJwAAAAAYHId6\nAgAAAIDBEfwAAAAAwOAIfgAAAABgcAQ/AAAAADA4gh8AAAAAGBzBDwAAAAAM7v8DNDhJo/f1T/MA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1680cac50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(15, 4))\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "ratio = max(lta[idx]) / max(keras_logreg[idx])\n",
    "plt.bar(np.linspace(0, len(campaign_idx), len(campaign_idx)), lta[campaign_idx], width=0.4, alpha=0.7, label='LTA' )\n",
    "plt.bar(np.linspace(0, len(campaign_idx), len(campaign_idx)) - 0.3, keras_logreg[campaign_idx], width=0.4, alpha=0.7, label='Keras Log Reg' )\n",
    "plt.xlabel('Campaign ID')\n",
    "plt.ylabel('Contribution')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4UAAAENCAYAAAC4rMA8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8FHWe//F3HzQhBkK6o4mBMEoEEQ8kBNCIRyTEx4Cy\nWQ+8dxjEY7hc5uHsgAsyPwEnDDioiKhrJurD0WU9FhyvxTxGRckqQYiuIkOiwCgiMZ2Dwxx0qn9/\naHpo0kk6Sd/9ev6Trurvp+tTlW9X96frW1Umt9vtFgAAAAAgLpnDnQAAAAAAIHwoCgEAAAAgjlEU\nAgAAAEAcoygEAAAAgDhGUQgAAAAAcYyiEAAAAADiGEUhAAAAAMSxkBWFFRUVuvvuuzV37lxt2LCh\nw3ZVVVW64YYb9OGHH3Y7FgAAAADQPSEpCg3DUHFxse69916tXr1aW7Zs0TfffOOz3Z///GeNGjWq\n27EAAAAAgO4LSVFYVVWl9PR0paWlyWq1Kjc3V+Xl5e3avfnmmxo/frwGDBjQ7VgAAAAAQPdZQ7GQ\n2tpaORwOz7TD4VBlZWW7Nlu3btWSJUu0bt26bsW2KS0tVWlpqSSpqKhILS0tgVyNoLNarXK5XOFO\nAzGGfoVgoF8hGOhXCAb6FYIlGvqWzWbzq11IikJ/PP3007r55ptlNvf84GV+fr7y8/M90zU1NYFI\nLWRSU1OjLmdEPvoVgoF+hWCgXyEY6FcIlmjoWxkZGX61C0lRaLfb5XQ6PdNOp1N2u92rzZdffqmH\nH35YknTo0CHt2LFDZrPZr1gAAAAAQM+EpCjMysrSgQMHVF1dLbvdrrKyMs2bN8+rzdq1a70ejxkz\nRuPGjVNra2uXsQAAAACAnglJUWixWDRjxgwtX75chmEoLy9PmZmZ2rRpkySpoKCg27E94Xa71dTU\nJMMwZDKZevQawXTw4EE1NzeHfLlut1tms1kJCQkRuV0AAAAABI/J7Xa7w51EsHz77bde042NjerT\np4+s1og5ldJLOE9WdblcOnbsmPr16xeW5SN4omG8O6IP/QrBQL9CMNCvECzR0Lf8PacwZDevjwSG\nYURsQRhuVqtVhmGEOw0AAAAAIRZXRSFDIzvH9gEAAADiT1wVhQAAAAAAb3E9lrJ1zdKAvp5l7uIu\n2wwbNkyVlZWe6dWrV+uNN96QJO3atUsjRoyQJN18882aPn26JOnyyy/X2WefrTVr1gQ0XwAAAACI\n66IwEsyfP1/z58+Xy+XSueeeq7ffftvr+S+++EIWi0VlZWVqbGzkQjAAIk5HP7D580MZAAAIP4aP\nRrgNGzbo2muvVW5ubruCEQAAAAB6i6Iwwr322muaOnWqCgsLtXHjxnCnAwAAACDGUBRGsI8//ljp\n6ek69dRTdckll6iiokKHDh0Kd1oAAAAAYghFYQTbuHGjdu3apfHjx2vChAk6fPiw3nzzzXCnBQAA\nACCGcKGZCGUYhl5//XW9++67OvnkkyVJmzdv1rp163T99deHOTsAAAAAsSKui8JwXBmvsbFRY8aM\n8UzfcccduvPOO9u127JlizIzMz0FoSTl5uZq7ty5+v77773mAwAAAEBPxXVRGA7ffPONz/lWq1WV\nlZVyuVySpIsvvlgXX3xxuzaffPJJ0HMEAAAAED84pxAAAAAA4hhFIQAAAADEMYpCAAAAAIhjFIUA\nAAAAEMcoCgEAAAAgjlEUAgAAAEAcC9ktKSoqKlRSUiLDMDRx4kQVFhZ6PV9eXq7169fLZDLJYrFo\n+vTpGjFihCRp9uzZSkhIkNlslsViUVFRUUByWvbu1wF5nTaLLsvsss2wYcNUWVnpNa+qqkoLFizQ\noUOH1NzcrPHjx2vy5Mlavny5JGnv3r1KT09XQkKCzjrrLN1www267rrrtHLlSt10002SpM8++0xX\nXHGFFi9erLvuusvnst966y3ddttteu+993TGGWf0cm0BAAAAxIKQFIWGYai4uFiLFi2Sw+HQwoUL\nlZOTo8GDB3vanHvuucrJyZHJZNK+ffu0evVqPfTQQ57nlyxZogEDBoQi3ZC77777dPvtt2vKlCly\nuVz64osvdNZZZ+myyy6TJF177bVavHixRo0aJUkqKyvTiBEj9Je//MVTFG7cuFEjR47sdDkbNmzQ\nuHHjtGHDBt1zzz1BXScAAAAA0SEkw0erqqqUnp6utLQ0Wa1W5ebmqry83KtNQkKCTCaTJKm5udnz\nOB5UV1fr1FNP9UyfddZZXcYMGjRIzc3N+v777+V2u/XOO+/o8ssv77D90aNHVV5erlWrVmnjxo0B\nyRsAAABA9AvJkcLa2lo5HA7PtMPhaDeEUpK2bt2q559/Xg0NDVq4cKHXc0uXLpXZbNakSZOUn5/v\nczmlpaUqLS2VJBUVFSk1NdXr+YMHD8pq/ccqm0yBrYmPf+3utLvzzjs1bdo0jR07VpdeeqluvPFG\nJScnH5fnj0Nq2+IsFotMJpOuuuoqvfHGGzr33HM1atQozxBbX3m8/fbbysvL05lnnim73a7PP//c\nc+SxTd++fdttM0Q/q9XK/xUBd3y/qrPZfLZJod+hm9hfIRjoVwiWWOpbITun0B/jxo3TuHHjtHPn\nTq1fv16LFy+W9GNBaLfb1dDQoGXLlikjI8PnUMn8/HyvgrGmpsbr+ebmZlksFs+0220ENH+Xy9Wj\ndtddd50uvvhibd68WW+++aaeffZZvf322+rbt+9PebrV2trqiWttbZXb7daUKVP0q1/9Srt379bU\nqVO1bds2GYbhM49XXnlFM2fOlMvl0tSpU/Xyyy/r7LPP9mrT3Nzcbpsh+qWmpvJ/RcAd369aW1p8\ntqHfobvYXyEY6FcIlmjoWxkZGX61C0lRaLfb5XQ6PdNOp1N2u73D9iNHjtRjjz2mQ4cOacCAAZ62\nycnJGjt2rKqqqro8fy7apKen66abbtK0adN0+eWX629/+5vOO++8TmNOOeUUWa1Wbd68Wffff7+2\nbdsmSdq/f7+mT58uSbr11lt11VVXacuWLdq1a5dMJpNaW1tlMpm0ePHiuBqmCwDwX0cXY/PnomoA\ngOgSknMKs7KydODAAVVXV8vlcqmsrEw5OTlebb777ju53W5J0ldffaVjx46pf//+ampqUmNjoySp\nqalJn376qYYMGRKKtEPmnXfe0bFjxyT9eH5hXV2d0tPT/Yq955579O///u9eR0AHDRqkt99+W2+/\n/bb+5V/+Ra+//rquueYabd26VR999JG2bdumIUOG6KOPPgrK+gAAAACIHiE5UmixWDRjxgwtX75c\nhmEoLy9PmZmZ2rRpkySpoKBAH374oTZv3iyLxSKbzab58+fLZDKpoaFBq1atkvTjsMkJEybo/PPP\nD0he4fi1s7GxUWPGjPFM33HHHTpw4IDuu+8+JSQkyO12a9GiRTrllFP8er2xY8d22WbDhg2aPXu2\n17zJkydrw4YNuuCCC7q3AgAAAABiisnddnguBn377bde0z/88IMSExPDlE3XrFar3+clBkOkbx/0\nTDSMd0f08TqncM1Sn20scxeHMiUEWDiGj7K/QjDQrxAs0dC3/D2nMCTDRwEAAAAAkYmiEAAAAADi\nWFwVhTE8UjYg2D4AAABA/ImrotBsNof1nL1I5nK5ZDbHVXcAAAAAoAi7eX2wJSQkqKmpSc3NzRF5\nf76+ffuqubk55Mt1u90ym81KSEgI+bIBAAAAhFdcFYUmk0n9+vULdxodioYrGAEAAACILYwXBAAA\nAIA4RlEIAAAAAHEsroaPAgAQKK1rlvqcb5m7OMSZAADQOxwpBAAAAIA4RlEIAAAAAHGM4aMAACDo\nGG4LAJGLohAAAKADFLMA4gHDRwEAAAAgjnGkMI7x6ycABN+yd7/2OX/RZZkhzgQAAN84UggAAAAA\ncYyiEAAAAADiGEUhAAAAAMSxkJ1TWFFRoZKSEhmGoYkTJ6qwsNDr+fLycq1fv14mk0kWi0XTp0/X\niBEj/IoFAAAAAPRMSIpCwzBUXFysRYsWyeFwaOHChcrJydHgwYM9bc4991zl5OTIZDJp3759Wr16\ntR566CG/YgEAAAAAPROS4aNVVVVKT09XWlqarFarcnNzVV5e7tUmISFBJpNJktTc3Ox57E8sAAAA\nAKBnQnKksLa2Vg6HwzPtcDhUWVnZrt3WrVv1/PPPq6GhQQsXLuxWrCSVlpaqtLRUklRUVKTU1NRA\nrkbQWa3WkOZcZ7P5nJ8SZdsNnQt1v0J8OL5fxeu+xN/1ttkO+mwX6e/LQOftz/aKxP1VvPbvWBKJ\n/QqxIZb6VkTdp3DcuHEaN26cdu7cqfXr12vx4u7dLy8/P1/5+fme6ZqamkCnGFSpqamd5hzoe121\ntrT4nB9t2w2d66pfAT1xfL+Kh32Jr/2v2zZW9x7Z0m7+ievdEqXbJ9B5+9NPInF/FQ/9O9ZFYr9C\nbIiGvpWRkeFXu5AMH7Xb7XI6nZ5pp9Mpu93eYfuRI0fq4MGDOnToULdjAQAAAAD+C0lRmJWVpQMH\nDqi6uloul0tlZWXKycnxavPdd9/J7XZLkr766isdO3ZM/fv39ysWAAAAANAzIRk+arFYNGPGDC1f\nvlyGYSgvL0+ZmZnatGmTJKmgoEAffvihNm/eLIvFIpvNpvnz53tuT+ErFgCAaOXeU6nW/3u63XzL\n3O6dNgEAQCCE7JzC7OxsZWdne80rKCjwPC4sLOzw/oO+YgEAAAAAvReS4aMAAAAAgMhEUQgAAAAA\ncYyiEAAAAADiWETdpxAAAADh1bpmqc/5XAgJPRXoe20j8DhSCAAAAABxjKIQAAAAAOIYw0cBPzGc\nBgAAALGIojBO+BrL7U66SPce2RKGbAAAAABECoaPAgAAAEAcoygEAAAAgDjG8FEAAICfnHi6hTvp\nIknidAt44RYLiDUUhfALOz8AAGIThTAAikIAQLcse/dr2WwH1dLSIokvkAAARDuKwhDjiBsAAEDg\ndXTrKP2/h0ObCBCF/C4KXS6X3n33Xe3du1dNTU1ez82ZMyfgiQEAAAAAgs/vovDRRx/Vvn37NGbM\nGCUnJwczJwAAAABAiPhdFH7yySd69NFHddJJJwUzHwAAAABACPldFKampurYsWPBzAWIaR2d62CZ\nuzjEmeB48Xqeb7yuNxBM8fi+isd1BmKR30XhJZdcopUrV+rnP/+5Bg4c6PXcOeec02V8RUWFSkpK\nZBiGJk6cqMLCQq/n33//fW3cuFFut1v9+vXTzJkzddppp0mSZs+erYSEBJnNZlksFhUVFfmbdkxz\n76mUJLX+39Ne8ykyAMQqvoACABB4fheFb731liTphRde8JpvMpn06KOPdhprGIaKi4u1aNEiORwO\nLVy4UDk5ORo8eLCnzSmnnKLf/e53SkpK0o4dO/Tkk0/qgQce8Dy/ZMkSDRgwwN90AQAAAAB+8Lso\nXLt2bY8XUlVVpfT0dKWlpUmScnNzVV5e7lUUnnnmmZ7Hw4YNk9Pp7PHyACBcGCYMAACiTbfuU9ja\n2qq//e1vqq2tlcPh0PDhw2WxWLqMa2vfxuFwqLKyssP2f/3rXzV69GiveUuXLpXZbNakSZOUn5/f\nnbQBAAAAAB3wuyjcv3+/VqxYoZaWFjkcDjmdTvXp00e//e1vvY749dZnn32md955R/fff79n3tKl\nS2W329XQ0KBly5YpIyNDI0eObBdbWlqq0tJSSVJRUZFSU1MDlleg2GwHfc5PTU2V1WrtNOcTY1vM\n5p/m27zmp/h4DV/LbTGb28V2J74t73hR52NbSb63VzDie6qrfhUL/u3Vz33O/8PUs7uMDXTfDtf/\nubt6s94220GZTP/Yf3RnX9Rbvck70P3E331ob/e/4dLb98aJ27tlwMWSpCVNH3nNP36dI2F/FejP\nWinyPytjZZ072v8Go19F0npHg1jdXpGwzwoUv4vCp556Svn5+brqqqtkMpkkSa+++qqKi4u1ZMmS\nTmPtdrvXcFCn0ym73d6u3b59+/TEE09o4cKF6t+/v1e8JCUnJ2vs2LGqqqryWRTm5+d7HUWsqanx\nd/VCpqWlxef8mpoapaamdprzibFuw/A539dr+Fqu2zB8zvc3vqO2saq1l9ugt/E91VW/igW96Z+B\n7tvh+j93V2+3mc1m87xGd/ZFvRWu/3Vv9qG93f+GS2/fGz35zIqE/VWgP2s7ahtJYmWdO9r/ulyu\ngOcTSesdDWJ1e0XCPqsrGRkZfrUz+/uCe/fu1ZVXXukpCCVpypQp2rt3b5exWVlZOnDggKqrq+Vy\nuVRWVqacnByvNjU1NVq1apXmzJnjlXxTU5MaGxs9jz/99FMNGTLE37QBAAAAAJ3w+0ih3W7Xzp07\nvW4/8cUXXyglJaXLWIvFohkzZmj58uUyDEN5eXnKzMzUpk2bJEkFBQV66aWXdOTIET311FOemKKi\nIjU0NGjVqlWSfjynccKECTr//PO7tZJAPDnxkv0220G1tLRwyX4AAAD45HdReOONN2rFihUaM2aM\n51Dp9u3bNXfuXL/is7OzlZ2d7TWvoKDA8/iuu+7SXXfd1S4uLS1NK1eu9DdNAAAQJO49le3ujStx\ndV14o58A0cfvojAnJ0crVqzQ//7v/6qurk6ZmZmaNm2a3+NUAQDwhdt4AAi1E0fVtGFUTejw40Fk\n6dYtKTIyMnTNNdcEKxcAcYwPaAAAgPDotCh84okndOedd0qS1qxZ43WRmePNmTMn8JkhZvBlHwAA\nAIhcnRaFp5xyiudxenp60JMBAAAAAIRWp0XhP//zP3seT5o0SQMHDmzXpr6+PvBZARHgxCOc7qSL\nJEn3HtkSjnQAAH6IpNEp7j2VktTuvCnOmQIQafw+p/Duu+/WM888027+/PnzVVJSEtCkAAChxZdX\nAOg59qGIdn7fvN7tdreb98MPP8hs9vslAAAAAAARpssjhb/61a8kSS0tLZ7HbY4cOaKLLrooOJkB\nQIyLpGFu8Yhf9gFEGz43ECxdFoVz586V2+3W73//+3Y3qh84cCD3KYxTfJkCgPDgvo4AEBrxVIR3\nWRSOHDlSklRcXKy+ffsGPSEg3sXTDggAEBz8eACgO/y+0MyGDRs6fO76668PSDIAAAAAgNDyuyh0\nOp1e0/X19dq5c6fGjRsX8KQAAACAUHLvqWx3WozE0VXEB7+LwlmzZrWbV1FRoQ8++CCgCQEAAABd\n4X7CQOD4XRT6ct5552n16tWBygUAAJyAL74AgGDzuyg8ePCg13Rzc7M++OADpaamBjwpAACAQGFY\nIAB0zu+icN68eV7TNptNp59+umbPnh3wpAAAAAAAoeF3Ubh+/fpg5gEAiAO+brniTrqIoZAAAIRR\nt84pNAxDu3fvVl1dnex2u4YNGyaz2Rys3AAg4nG+FwAAiHZ+F4X79u3TypUrdezYMdntdtXW1qpP\nnz665557dNppp3UZX1FRoZKSEhmGoYkTJ6qwsNDr+ffff18bN26U2+1Wv379NHPmTM/rdhULxDrO\nhwEAAOi91jVLfc6P9+9UfheF69at0xVXXKErr7xSJpNJbrdbr7/+utatW6cVK1Z0GmsYhoqLi7Vo\n0SI5HA4tXLhQOTk5Gjx4sKfNKaecot/97ndKSkrSjh079OSTT+qBBx7wKxZAYPga2idJiy7LDHEm\ngH/ceyolqd2PJvH+4Q4AQHf4XRQeOHBAU6ZMkclkkiSZTCZNnjxZL774YpexVVVVSk9PV1pamiQp\nNzdX5eXlXoXdmWee6Xk8bNgwOZ1Ov2MB4ESxfnSVAh4AAASK30Xh6NGjtW3bNo0bN84zb9u2bRo9\nenSXsbW1tXI4HJ5ph8OhysrKDtv/9a9/9bxud2MRW/jiCwAA0LFY/xEUodFpUbhmzRrPkUHDMPTQ\nQw9p6NChcjgccjqd+uqrr5STkxPQhD777DO98847uv/++7sdW1paqtLSUklSUVFRRN5D0WY76HN+\namqqrFZrpzmfGNvy00V+bDab1/wUH6/ha7ktZnO7WH/je7tsSX79f3oT21u9WWdf6nxsa1/xgf5f\nmUw/xkbb9u7OsnuzzXzGVn4hyxPPt4/995VdLjta3hu9zbutX/Uk9kTh2heFKu9wrnNvhCPvrj4H\n/RGtfSzQ/P3MkcL7/aI3/M3bV7+K1/dluERS3r15b7Rp60+B2GdFik6LwvT0dK/pzMx/HJ0ZPHiw\nRo0a5ddC7Ha7ZzioJDmdTtnt9nbt9u3bpyeeeEILFy5U//79uxUrSfn5+crPz/dM19TU+JVfKLW0\ntPicX1NTo9TU1E5zPjHWbRg+5/t6DV/LdRuGz/n+xPd22R21DWRsb/VmnX1p9XNdAv2/stlsamlp\nibrt3Z1l92abxet7o7d5t/WrnsSeKFzbO1R5h3OdeyMceXf1OeiPaO1jgebvZ44U3u8XveFv3i6X\ni/dlmIUz73ZXCbeNldT+KuE9+ZwOxD4r2DIyMvxq12lReN111wUkmaysLB04cEDV1dWy2+0qKyvT\nvHnzvNrU1NRo1apVmjNnjlfy/sQCABhCBAAAeqbTonDnzp0aOXKkpB+HdXbknHPO6XQhFotFM2bM\n0PLly2UYhvLy8pSZmalNmzZJkgoKCvTSSy/pyJEjeuqppzwxRUVFHcYCAAAAAHqv06KwuLhYDz74\noKQfb0nhi8lk0qOPPtrlgrKzs5Wdne01r6CgwPP4rrvu0l133eV3LAAAAID4w70GA6/TorCtIJR+\nvOiM+aeTaAEAAAAAscGvW1IYhqFbb71VTz/9tPr06RPsnBDj4uFm075upeFOuqjdSc2xhNuHAAAA\nRCe/ikKz2ayMjAwdPny4wyt/AgCAyMKPNQDa9HZ/wP4ktvl98/oJEyZoxYoV+vnPfy6Hw+G5f6HU\n9YVmAAAAACDaHX+l7zqbzXP7l2gf8eZ3Udh2pdAXX3zRa76/F5oBEB4tlV/IbRgxPVwXAAAAPed3\nUbh27dpg5gEAAIA4w5BEdCUer9MQDn4XhX/4wx/0b//2b+3mr1q1Svfcc09AkwI6ww26AQBAtIqV\nQjgeLhwYT/wuCj///PNuzQcAAJGHH9YAACfqsihcv369JMnlcnketzl48KBOPvnk4GQGAAAAAAi6\nLotCp9Mp6cd7FbY9bpOamqpp06YFJzMAABCXYmV4XXfE4zoj/Bg5gDZdFoWzZs2SJA0fPlz5+flB\nTwgAAAAAEDp+n1OYn5+vH374Qd9++62ampq8nuM+hQAAAAAQnfwuCt99910VFxcrISFBNpvNM5/7\nFALoCMNSuocruQEAgHDwuyh84YUX9Otf/1qjR48OZj4AAAAAgBAy+9vQMAyNGjUqmLkAAAAAAELM\n7yOF//RP/6SXX35Z11xzjcxmv2tJAAAABNmJVy91J10kSbr3yJZwpAMgyvhdFL7++uuqr6/Xq6++\nqqSkJK/n1q1bF/DEAAAAAADB53dROHfu3GDmAQBeuOgKAABAaPhdFI4cObJXC6qoqFBJSYkMw9DE\niRNVWFjo9fz+/fv12GOPac+ePbrhhhs0depUz3OzZ89WQkKCzGazLBaLioqKepULAAAAIg8/CALh\n4XdR6HK59Morr2jz5s2qq6tTSkqKLrnkEl199dWyWjt/GcMwVFxcrEWLFsnhcGjhwoXKycnR4MGD\nPW2SkpL0y1/+UuXl5T5fY8mSJRowYIC/6QIAAAAA/OB3Ufjcc8/pyy+/1O23366TTz5Z33//vV5+\n+WX98MMPmj59eqexVVVVSk9PV1pamiQpNzdX5eXlXkVhcnKykpOTtX379p6tCQAAAACg2/wuCj/8\n8EOtXLlS/fv3lyRlZGTo9NNP129+85sui8La2lo5HA7PtMPhUGVlZbcSXbp0qcxmsyZNmqT8/Pxu\nxQIAAAAAfPO7KHS73cHMo1NLly6V3W5XQ0ODli1bpoyMDJ/nOJaWlqq0tFSSVFRUpNTU1FCn2iWb\n7aDP+ampqbJarZ3mfGJsy0+3BrHZbF7zU3y8hq/ltpjN7WL9je/NskOZd2+EK+9A/6+OSZKPeH/z\nluTXeymS+lh3lh1JeUfTe9pk+seyoinvQMV2J+9IWudIz7vtczBW9kW9/V9J/q1zoJcda+9pX9+v\nIinvNsHu35G2zrGQt9lk8jwO9PfQUPO7KLzwwgu1YsUKXXvttUpNTVVNTY1efvllXXDBBV3G2u12\nOZ1Oz7TT6ZTdbvc7yba2ycnJGjt2rKqqqnwWhfn5+V5HEWtqavxeRqi0tLT4nF9TU+PZrv7Gug3D\n53xfr+FruW7D8Dnfn/jeLDuUefdGuPIO9P+qo3h/Yztq29Pl+rvsaN3e0fLe6O2ybTabp2005R2o\n2O7kHUnrHOl5t30Oxsq+qLf/q47a+hJJeUfa9na5XBHRvztbdkdt/YmN5Pd0rOd9/GdhJNYd0o+j\nO/3hd1F4yy236OWXX1ZxcbHq6upkt9t10UUX6ZprrukyNisrSwcOHFB1dbXsdrvKyso0b948v5bb\n1NQkt9utfv36qampSZ9++qmuvfZaf9MGAAAAAHSiy6Jw165d+vjjj3XzzTfr+uuv1/XXX+957rnn\nntNXX32l4cOHd/oaFotFM2bM0PLly2UYhvLy8pSZmalNmzZJkgoKClRfX68FCxaosbFRJpNJb7zx\nhv74xz/q8OHDWrVqlSSptbVVEyZM0Pnnn9+bdQYAAAAA/KTLovC///u/dcUVV/h87pxzztErr7yi\nBQsWdLmg7OxsZWdne80rKCjwPB44cKAef/zxdnGJiYlauXJll68PAAAAAOi+LovCvXv3dnhk7txz\nz9W6desCnhQAAMDx3Hsq293QXOKm5gAQCF0WhY2NjXK5XD6v8NPa2qrGxsagJAYAAADEOveeH2/T\nduKPHvzggVAyd9Vg0KBB+uSTT3w+98knn2jQoEEBTwoAAAAAEBpdFoVTpkzRk08+qY8++kjGT5dw\nNQxDH330kf7jP/5DU6ZMCXqSAAAAAIDg6HL46IQJE1RfX6+1a9fq2LFjGjBggA4dOqQ+ffpo2rRp\nmjBhQijyBAAAAAAEgV/3Kbzyyit1+eWXa/fu3Tpy5IiSkpI0fPhwJSYmBjs/AAAAAEAQ+X3z+sTE\nRO4PCADuJZYJAAATJUlEQVQAAAAxpstzCgEAAAAAscvvI4VANFr27tc+5y+6LDPEmQAAAACRiSOF\nAAAAABDHOFIYZsffsLTOZlNrS4skblgKAAAAIDQoCgF0yr2nUq3/93S7+fxwAQAAEBsYPgoAAAAA\ncYyiEAAAAADiGEUhAAAAAMQxikIAAAAAiGMUhQAAAAAQx7j6KAAAQATi6s8AQiVkRWFFRYVKSkpk\nGIYmTpyowsJCr+f379+vxx57THv27NENN9ygqVOn+h0LAAAAAOiZkAwfNQxDxcXFuvfee7V69Wpt\n2bJF33zzjVebpKQk/fKXv9RVV13V7VgAAAAAQM+EpCisqqpSenq60tLSZLValZubq/Lycq82ycnJ\nOuOMM2SxWLodCwAAAADomZAUhbW1tXI4HJ5ph8Oh2traoMcCAAAAADoXUxeaKS0tVWlpqSSpqKhI\nqampYc6oPZvtoNd0i9n803ybzCaTbDabJCnFR+6dxR7Pn9i2+BNjQ7HscObdxp++Ea68A73OxyTJ\nR3yk5x2t2zta3xvdXbbJ9I9lRVPegYrtTt6RtM6RnrfValVqamrU5R2MWPIOXN5t/Sra8o6F93Ss\n593Vd/doEpKi0G63y+l0eqadTqfsdnvAY/Pz85Wfn++Zrqmp6WHGwdPS0uI17TYMz3ybzeZ53lfu\nncUez5/Ytnhf84O97HDm3VnbrmJDlXcw1tlXfKTnHa3bO1rfG91d9vH7q2jKO1Cx3ck7ktY50vNO\nTU1VTU1N1OUdjFjyDlzeLpeLftLLWPL2veyuvrtHgoyMDL/ahWT4aFZWlg4cOKDq6mq5XC6VlZUp\nJycn6LEAAAAAgM6F5EihxWLRjBkztHz5chmGoby8PGVmZmrTpk2SpIKCAtXX12vBggVqbGyUyWTS\nG2+8oT/+8Y9KTEz0GQsAAAAA6L2QnVOYnZ2t7Oxsr3kFBQWexwMHDtTjjz/udywAAAAAoPdCMnwU\nAAAAABCZKAoBAAAAII5RFAIAAABAHKMoBAAAAIA4RlEIAAAAAHGMohAAAAAA4hhFIQAAAADEMYpC\nAAAAAIhjFIUAAAAAEMes4U4ACCX3nkpJUuv/Pe013zJ3cRiyAQAAAMKPI4UAAAAAEMcoCgEAAAAg\njlEUAgAAAEAcoygEAAAAgDhGUQgAAAAAcYyiEAAAAADiGEUhAAAAAMQxikIAAAAAiGMhu3l9RUWF\nSkpKZBiGJk6cqMLCQq/n3W63SkpKtGPHDvXt21ezZs3S0KFDJUmzZ89WQkKCzGazLBaLioqKQpU2\nAAAAAMS0kBSFhmGouLhYixYtksPh0MKFC5WTk6PBgwd72uzYsUPfffedHnnkEVVWVuqpp57SAw88\n4Hl+yZIlGjBgQCjSBQAAAIC4EZLho1VVVUpPT1daWpqsVqtyc3NVXl7u1Wbbtm265JJLZDKZNHz4\ncB09elR1dXWhSA8AAAAA4lZIjhTW1tbK4XB4ph0OhyorK9u1SU1N9WpTW1urlJQUSdLSpUtlNps1\nadIk5efnhyJtAAAAAIh5ITunsDeWLl0qu92uhoYGLVu2TBkZGRo5cmS7dqWlpSotLZUkFRUVeRWZ\nkcJmO+g13WI2/zTfJrPJJJvNJklK8ZF7Z7HH8ye2Lf7E2FAsm7w7zzvQ63xMknzER3re0bq9o6GP\nBWLZJtM/lhVNeQcqtjt5R9I6R3reVqtVqampUZd3MGLJO3B5t/WraMs7Ft7TsZ53V9/do0lIikK7\n3S6n0+mZdjqdstvt7drU1NT4bNP2Nzk5WWPHjlVVVZXPojA/P9/rKOLxrxcpWlpavKbdhuGZb7PZ\nPM/7yr2z2OP5E9sW72t+sJdN3p3nHeh17ig+0vOO1u0dDX0sEMs+fn8VTXkHKrY7eUfSOkd63qmp\nqaqpqYm6vIMRS96By9vlctFPehlL3r6X3dV390iQkZHhV7uQnFOYlZWlAwcOqLq6Wi6XS2VlZcrJ\nyfFqk5OTo82bN8vtdmv37t1KTExUSkqKmpqa1NjYKElqamrSp59+qiFDhoQibQAAAACIeSE5Umix\nWDRjxgwtX75chmEoLy9PmZmZ2rRpkySpoKBAo0eP1vbt2zVv3jzZbDbNmjVLktTQ0KBVq1ZJklpb\nWzVhwgSdf/75oUgbAAAAAGJeyM4pzM7OVnZ2tte8goICz2OTyaSZM2e2i0tLS9PKlSuDnh8AAAAA\nxKOQDB8FAAAAAEQmikIAAAAAiGMUhQAAAAAQxygKAQAAACCOURQCAAAAQByjKAQAAACAOEZRCAAA\nAABxjKIQAAAAAOIYRSEAAAAAxDGKQgAAAACIYxSFAAAAABDHKAoBAAAAII5RFAIAAABAHKMoBAAA\nAIA4RlEIAAAAAHGMohAAAAAA4hhFIQAAAADEMYpCAAAAAIhj1lAtqKKiQiUlJTIMQxMnTlRhYaHX\n8263WyUlJdqxY4f69u2rWbNmaejQoX7FAgAAAAB6JiRHCg3DUHFxse69916tXr1aW7Zs0TfffOPV\nZseOHfruu+/0yCOP6I477tBTTz3ldywAAAAAoGdCUhRWVVUpPT1daWlpslqtys3NVXl5uVebbdu2\n6ZJLLpHJZNLw4cN19OhR1dXV+RULAAAAAOiZkBSFtbW1cjgcnmmHw6Ha2tp2bVJTU9u18ScWAAAA\nANAzJrfb7Q72Qj788ENVVFTorrvukiRt3rxZlZWVuu222zxtioqKVFhYqBEjRkiS7r//ft188836\n/vvvu4xtU1paqtLSUs/rAQAAAAA6F5IjhXa7XU6n0zPtdDplt9vbtampqWnXxp/YNvn5+SoqKora\ngnDBggXhTgExiH6FYKBfIRjoVwgG+hWCJZb6VkiKwqysLB04cEDV1dVyuVwqKytTTk6OV5ucnBxt\n3rxZbrdbu3fvVmJiolJSUvyKBQAAAAD0TEhuSWGxWDRjxgwtX75chmEoLy9PmZmZ2rRpkySpoKBA\no0eP1vbt2zVv3jzZbDbNmjWr01gAAAAAQO+F7D6F2dnZys7O9ppXUFDgeWwymTRz5ky/Y2NRfn5+\nuFNADKJfIRjoVwgG+hWCgX6FYImlvhWSC80AAAAAACJTSM4pBAAAAABEppANH0XnKioqVFJSIsMw\nNHHiRBUWFoY7JUShxx57TNu3b1dycrIefPBBSdKRI0e0evVqff/99zr55JM1f/58JSUlhTlTRJOa\nmhqtXbtW9fX1MplMys/P1+TJk+lb6JWWlhYtWbJELpdLra2tuuCCCzRt2jT6FQLCMAwtWLBAdrtd\nCxYsoF+h12bPnq2EhASZzWZZLBYVFRXFVL9i+GgEMAxDd999txYtWiSHw6GFCxfq7rvv1uDBg8Od\nGqLMzp07lZCQoLVr13qKwueee05JSUkqLCzUhg0bdOTIEd1yyy1hzhTRpK6uTnV1dRo6dKgaGxu1\nYMEC/eY3v9G7775L30KPud1uNTc3KyEhQS6XS/fdd5+mT5+urVu30q/Qa6+99pq+/PJLzz6Lz0L0\n1uzZs/X73/9eAwYM8MyLpX7F8NEIUFVVpfT0dKWlpclqtSo3N1fl5eXhTgtRaOTIke1+oSovL9el\nl14qSbr00kvpW+i2lJQUDR06VJLUr18/DRo0SLW1tfQt9IrJZFJCQoIkqbW1Va2trTKZTPQr9JrT\n6dT27ds1ceJEzzz6FYIhlvoVw0cjQG1trRwOh2fa4XCosrIyjBkhljQ0NCglJUWSNHDgQDU0NIQ5\nI0Sz6upq7dmzR2eccQZ9C71mGIZ++9vf6rvvvtMVV1yhYcOG0a/Qa08//bRuueUWNTY2eubRrxAI\nS5culdls1qRJk5Sfnx9T/YqiEIgjJpNJJpMp3GkgSjU1NenBBx/U9OnTlZiY6PUcfQs9YTabtXLl\nSh09elSrVq3S3//+d6/n6Vforo8//ljJyckaOnSoPv/8c59t6FfoiaVLl8put6uhoUHLli1TRkaG\n1/PR3q8oCiOA3W6X0+n0TDudTtnt9jBmhFiSnJysuro6paSkqK6uzmssPOAvl8ulBx98UBdffLHG\njx8vib6FwDnppJN09tlnq6Kign6FXvnb3/6mbdu2aceOHWppaVFjY6MeeeQR+hV6re27eXJyssaO\nHauqqqqY6lecUxgBsrKydODAAVVXV8vlcqmsrEw5OTnhTgsxIicnR++9954k6b333tPYsWPDnBGi\njdvt1uOPP65Bgwbpyiuv9Mynb6E3Dh06pKNHj0r68Uqkn376qQYNGkS/Qq/cdNNNevzxx7V27Vr9\n67/+q8455xzNmzePfoVeaWpq8gxHbmpq0qeffqohQ4bEVL/i6qMRYvv27XrmmWdkGIby8vJ09dVX\nhzslRKGHHnpIO3fu1OHDh5WcnKxp06Zp7NixWr16tWpqaqL+cskIj127dum+++7TkCFDPENjbrzx\nRg0bNoy+hR7bt2+f1q5dK8Mw5Ha7deGFF+raa6/V4cOH6VcIiM8//1x/+ctftGDBAvoVeuXgwYNa\ntWqVpB8vjDVhwgRdffXVMdWvKAoBAAAAII4xfBQAAAAA4hhFIQAAAADEMYpCAAAAAIhjFIUAAAAA\nEMcoCgEAAAAgjlEUAgAQAjU1Nbr11ltlGEa4UwEAwAu3pAAARL0PPvhAr732mvbv369+/frptNNO\n09VXX60RI0aEO7WQWLt2rRwOh2644QZVV1drzpw56tu3ryQpISFBWVlZmjx5ss4777wwZwoAiETW\ncCcAAEBvvPbaa9qwYYNuv/12jRo1SlarVZ988om2bdsWN0WhL08//bQsFovq6+tVVlamlStX6rbb\nbtNll10W7tQAABGGohAAELV++OEHrV+/XrNmzdL48eM988eMGaMxY8ZIkqqqqlRSUqL9+/fLZrNp\n/Pjx+sUvfiGr9cePwGnTpum2227T66+/rvr6ek2ePFmXXXaZHn30UX399dcaNWqU5s2bJ6vVqs8/\n/1xr1qxRQUGBXn/9dSUkJOiGG27QxRdfLEnavn27/vM//1MHDx5UYmKi8vLyNG3aNEnyHMF74YUX\nZLFYVF1drbVr12rPnj0aNmyYTj31VP3www+aN2+ep+2sWbO0fv16tbS0aMqUKbr66qu7vY0GDhyo\nyZMny+Vy6c9//rMuueQSmc2cPQIA+Ac+FQAAUWv37t06duyYxo0b12Ebs9msX/ziFyouLtayZcv0\n2Wef6X/+53+82nzyyScqKirS8uXL9eqrr+rJJ5/U3LlztW7dOn399df64IMPPG3r6+t1+PBhPf74\n45o9e7aefPJJffvtt5Kkvn37as6cOSopKdGCBQv09ttva+vWrT7zevjhh5WVlaU//elPuu666/T+\n+++3a7Nr1y49/PDDWrx4sV566SV98803PdlMkqTx48eroaHBkysAAG0oCgEAUevw4cPq37+/LBZL\nh22GDh2q4cOHy2Kx6JRTTlF+fr527tzp1Wbq1KlKTExUZmamMjMzdd555yktLU2JiYkaPXq09u7d\n69X++uuvV58+fTRy5EiNHj1aZWVlkqSzzz5bQ4YMkdls1s9+9jNddNFF7ZYl/XjRmS+//FLXX3+9\nrFarRowY4TmyebzrrrtONptNp512mn72s59p3759PdhKP0pJSZEkHTlypMevAQCITQwfBQBErf79\n++vw4cNqbW3tsDD89ttv9eyzz+rLL79US0uLWltbNXToUK82AwcO9Dy22Wztpuvr6z3TJ510khIS\nEjzTJ598surq6iRJlZWVev755/X3v/9dLpdLLpdLF1xwQbucamtrlZSU5LkYjCSlpqaqpqamw7z6\n9u2rpqamTrdHZ2prayVJSUlJPX4NAEBs4kghACBqDR8+XH369FF5eXmHbZ566illZGTokUce0TPP\nPKMbb7xRvbnw9tGjR72Ks5qaGs9RuEceeURjxozRunXr9Mwzz2jSpEk+l5WSkqIjR46oubnZ63WC\naevWrUpOTlZGRkZQlwMAiD4UhQCAqJWYmKhp06apuLhYW7duVXNzs1wul3bs2KHnnntOktTY2KjE\nxEQlJCRo//792rRpU6+X+1//9V9yuVz64osvtH37dl144YWeZSUlJclms6mqqsrrXMTjnXzyycrK\nytKLL74ol8ul3bt36+OPP+51Xr7U19frrbfe0ksvvaQbb7yRi8wAANph+CgAIKpdddVVGjhwoF55\n5RWtWbNGCQkJGjp0qOdKnbfeequefPJJbdy4Uaeffrpyc3P12Wef9Xh5AwcOVFJSku68807ZbDbd\nfvvtGjRokCRp5syZevbZZ/WnP/1JI0eO1IUXXqijR4/6fJ25c+fqscce04wZM3TGGWcoNzc3oDe2\nnz59uqQfh51mZWXp17/+tc4///yAvT4AIHZw83oAAPzUdkuKxx9/POCvvXr1ag0aNMhzCwsAAEKF\nMSQAAIRBVVWVvvvuOxmGoYqKCm3btk1jx44Nd1oAgDjE8FEAAMKgvr5eDz74oA4fPiyHw6GZM2fq\n9NNPD3daAIA4xPBRAAAAAIhjDB8FAAAAgDhGUQgAAAAAcYyiEAAAAADiGEUhAAAAAMQxikIAAAAA\niGMUhQAAAAAQx/4/6cs6PxMFHncAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x205b27a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(15, 4))\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "ratio = max(lta[campaign_idx]) / max(lstm_a[campaign_idx])\n",
    "plt.bar(np.linspace(0, len(campaign_idx), len(campaign_idx)), lta[campaign_idx], width=0.4, alpha=0.7, label='LTA' )\n",
    "plt.bar(np.linspace(0, len(campaign_idx), len(campaign_idx)) - 0.3, lstm_a[campaign_idx], width=0.4, alpha=0.7, label='LSTM-A'  )\n",
    "plt.xlabel('Campaign ID')\n",
    "plt.ylabel('Contribution')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simulation\n",
    "\n",
    "In this section, we simulate what would happen if we allocate the campaign budgets proportionally to the attribution weights. This is done using a simulator that perform a counterfactual evaluation based on historical events. This procedure enables us to evaluate and compare attribution models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Key assumption: If one of the campaigns in a journey runs out of budget, \n",
    "# then the conversion reward is fully lost for the entire journey\n",
    "# including both past and future campaigns\n",
    "\n",
    "def simulate_budget_roi(df, budget_total, attribution, verbose=False):\n",
    "    budgets = np.ceil(attribution * (budget_total / np.sum(attribution)))\n",
    "    \n",
    "    if(verbose):\n",
    "        print(budgets)\n",
    "    \n",
    "    blacklist = set()\n",
    "    conversions = set()\n",
    "    for i in range(df.shape[0]):\n",
    "        campaign_id = get_campaign_id(df.loc[i]['campaigns']) \n",
    "        jid = df.loc[i]['jid']\n",
    "        if jid not in blacklist:\n",
    "            if budgets[campaign_id] >= 1:\n",
    "                budgets[campaign_id] = budgets[campaign_id] - 1\n",
    "                if(df.loc[i]['conversion'] == 1):\n",
    "                    conversions.add(jid)\n",
    "            else:\n",
    "                blacklist.add(jid)\n",
    "        \n",
    "        if(verbose):\n",
    "            if(i % 10000 == 0):\n",
    "                print('{:.2%} : {:.2%} budget spent'.format(i/df.shape[0], 1.0 - np.sum(budgets)/budget_total ))\n",
    "        \n",
    "        if(np.sum(budgets) < budget_total * 0.02):\n",
    "            break\n",
    "            \n",
    "    return len(conversions.difference(blacklist))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pitches = [0.1, 0.25, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0]\n",
    "attributions = [lta, keras_logreg, lstm_a]\n",
    "\n",
    "for i, pitch in enumerate(pitches):\n",
    "    for j, attribution in enumerate(attributions):\n",
    "        reward = simulate_budget_roi(df6, 10000, attribution**pitch)\n",
    "        print('{} {} : {}'.format(p, j, reward))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}