notebooks/00_quick_start/rbm_movielens.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "inputHidden": false,
    "outputHidden": false
   },
   "source": [
    "# Movie recommender with multinomial RBM (Tensorflow, GPU)\n",
    "\n",
    "A Restricted Boltzmann Machine (RBM) is a generative neural network model typically used to perform unsupervised learning. The main task of an RBM is to learn the joint probability distribution $P(v,h)$, where $v$ are the visible units and $h$ the hidden ones. The hidden units represent latent variables while the visible units are clamped on the input data. Once the joint distribution is learnt, new examples are generated by sampling from it.  \n",
    "\n",
    "In this notebook, we provide an example of how to utilize the RBM to perform user/item recommendations. In particular, we use as a case study the [movielens dataset](https://movielens.org), comprising user's ranking of movies on a scale of 1 to 5. \n",
    "\n",
    "This notebook provides a quick start, showing the basic steps needed to use and evaluate the algorithm. A detailed discussion of the RBM model together with a deeper analysis of the recommendation task is provided in the [RBM Deep Dive section](../02_model/rbm_deep_dive.ipynb). The RBM implementation presented here is based on the article by Ruslan Salakhutdinov, Andriy Mnih and Geoffrey Hinton [Restricted Boltzmann Machines for Collaborative Filtering](https://www.cs.toronto.edu/~rsalakhu/papers/rbmcf.pdf) with the exception that here we use multinomial units instead of the one-hot encoded used in the paper.  \n",
    "\n",
    "### Advantages of RBM: \n",
    "\n",
    "The model generates ratings for a user/movie pair using a collaborative filtering based approach. While matrix factorization methods learn how to reproduce an instance of the user/item affinity matrix, the RBM learns the underlying probability distribution. This has several advantages: \n",
    "\n",
    "- Generalizability : the model generalize well to new examples.\n",
    "- Stability in time: if the recommendation task is time-stationary, the model does not need to be trained often to accomodate new ratings/users. \n",
    "- The tensorflow implementation presented here allows fast training on GPU "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0 Global Settings and Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44) \n",
      "[GCC 7.3.0]\n",
      "Pandas version: 0.23.4\n"
     ]
    }
   ],
   "source": [
    "#load libraries\n",
    "\n",
    "from __future__ import print_function\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "\n",
    "# set the environment path to find Recommenders\n",
    "import sys\n",
    "sys.path.append(\"../../\")\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import papermill as pm\n",
    "\n",
    "from reco_utils.recommender.rbm.rbm import RBM\n",
    "from reco_utils.dataset.python_splitters import numpy_stratified_split\n",
    "from reco_utils.dataset.sparse import AffinityMatrix\n",
    "\n",
    "\n",
    "from reco_utils.dataset import movielens\n",
    "from reco_utils.evaluation.python_evaluation import map_at_k, ndcg_at_k, precision_at_k, recall_at_k\n",
    "\n",
    "#For interactive mode only\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Pandas version: {}\".format(pd.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Load Data \n",
    "\n",
    "Here we select the size of the movielens dataset. In this example we consider the 100k ratings datasets, provided by  943 users on 1682 movies. The data are imported in a pandas dataframe including the user ID, the item ID, the ratings and a timestamp denoting when a particular user rated a particular item.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Select MovieLens data size: 100k, 1m, 10m, or 20m\n",
    "MOVIELENS_DATA_SIZE = '100k'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>movieID</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "      <td>881250949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>891717742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>878887116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>880606923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>886397596</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID  movieID  rating  timestamp\n",
       "0     196      242       3  881250949\n",
       "1     186      302       3  891717742\n",
       "2      22      377       1  878887116\n",
       "3     244       51       2  880606923\n",
       "4     166      346       1  886397596"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = movielens.load_pandas_df(\n",
    "    size=MOVIELENS_DATA_SIZE,\n",
    "    header=['userID','movieID','rating','timestamp']\n",
    ")\n",
    "\n",
    "# Convert to 32-bit in order to reduce memory consumption \n",
    "data.loc[:, 'rating'] = data['rating'].astype(np.int32) \n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Split the data using the stratified splitter  \n",
    "\n",
    "As a second step we generate the user/item affiity matrix and then split the data into train and test set. If you are familiar with training supervised learning model, here you will notice the first difference. In the former case, we cut off a certain proportion of training examples from dataset (e.g. images), here corresponding to users (or items), ending up with two matrices (train and test) having different row dimensions. Here we need to mantain the same matrix size for the train and test set, but the two will contain different amounts of ratings, see the [deep dive notebook](../02_model/rbm_deep_dive.ipynb) for more details. The affinity matrix reads     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false,
    "tags": [
     "sparse_matrix"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating the user/item affinity matrix...\n",
      "Matrix generated, sparseness percentage: 93\n"
     ]
    }
   ],
   "source": [
    "#to use standard names across the analysis \n",
    "header = {\n",
    "        \"col_user\": \"userID\",\n",
    "        \"col_item\": \"movieID\",\n",
    "        \"col_rating\": \"rating\",\n",
    "    }\n",
    "\n",
    "#instantiate the sparse matrix generation  \n",
    "am = AffinityMatrix(DF = data, **header)\n",
    "\n",
    "#obtain the sparse matrix \n",
    "X = am.gen_affinity_matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method also returns informations on the sparsness of the dataset and the size of the user/affinity matrix. The former is given by the ratio between the unrated elements and the total number of matrix elements. This is what makes a recommendation task hard: we try to predict 93% of the missing data with only 7% of information!\n",
    "\n",
    "We split the matrix using the default ration of 0.75, i.e. 75% of the ratings will constitute the train set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": [
     "split"
    ]
   },
   "outputs": [],
   "source": [
    "Xtr, Xtst = numpy_stratified_split(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The splitter returns:\n",
    "\n",
    "- Xtr: a matrix containing the train set ratings \n",
    "- Xtst: a matrix containing the test elements \n",
    "\n",
    "Note that the train/test matrices have exactly the same dimension, but different entries as it can be explicitly verified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train matrix size (943, 1682)\n",
      "test matrix size (943, 1682)\n"
     ]
    }
   ],
   "source": [
    "print('train matrix size', Xtr.shape)\n",
    "print('test matrix size', Xtst.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "model",
     "train"
    ]
   },
   "source": [
    "## 2 Train the RBM model\n",
    "\n",
    "The model has been implemented as a Tensorflow (TF) class. TF does not support probabilistic models natively, so the implementation of the algorithm has a different structure than the one you may be used to see in popular supervised models. The class has been implemented in such a way that the TF session is hidden inside the `fit()` method and no explicit call is needed. The algorithm operates in three different steps: \n",
    "\n",
    "- Model initialization: This is where we tell TF how to build the computational graph. The main parameters to specify are the number of hidden units, the number of training epochs and the minibatch size. Other parameters can be optionally tweaked for experimentation and to achieve better performance, as explained in the [RBM Deep Dive section](../02_model/rbm_deep_dive.ipynb).\n",
    "\n",
    "- Model fit: This is where we train the model on the data. The method takes two arguments: the training and test set matrices. Note that the model is trained **only** on the training set, the test set is used to display the generalization accuracy of the trained model, useful to have an idea of how to fix the hyper parameters. \n",
    "\n",
    "- Model prediction: This is where we generate ratings for the unseen items. Once the model has been trained and we are satisfied with its overall accuracy, we sample new ratings from the learned distribution. In particular, we extract the top_k (e.g. 10) most relevant recommendations according to some predefined score. The prediction is then returned in a dataframe format ready to be analysed and deployed.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false,
    "tags": [
     "initialization"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 1.12.0\n"
     ]
    }
   ],
   "source": [
    "#First we initialize the model class\n",
    "model = RBM(hidden_units= 600, training_epoch = 30, minibatch_size= 60, keep_prob=0.9,with_metrics =True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the first time the fit method is called it may take longer to return the result. This is due to the fact that TF needs to initialized the GPU session. You will notice that this is not the case when training the algorithm the second or more times.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "inputHidden": false,
    "outputHidden": false,
    "tags": [
     "training"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Creating the computational graph\n",
      "Initialize Gibbs protocol\n",
      "training epoch 0 rmse 0.943123\n",
      "training epoch 10 rmse 0.807337\n",
      "training epoch 20 rmse 0.786743\n",
      "training epoch 30 rmse 0.779240\n",
      "done training, Training time 2.5419012\n",
      "Train set accuracy 0.3806472\n",
      "Test set accuracy 0.3675472\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEOCAYAAACqzTG4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8nXXZx/HPld00SdM2aZsm6QBaumdaZhkitZSNiK2AggriI4iKIG4ffFQcj4NHEAERRaFWEAFlyh4Fmk5autKZ7nTP7Ov549wthzSkOe1ZSb7v1+u8zn3uda4fB/LlXr+fuTsiIiLRkpLoAkREpH1RsIiISFQpWEREJKoULCIiElUKFhERiSoFi4iIRJWCRUREoiquwWJmk8xsiZlVmNmtzSzva2YvmNl8M3vZzErCljWY2dzg9UTY/P5m9nawz7+ZWUa82iMiIoeyeD0gaWapwFLgbGAtMBOY6u7vha3zd+Bf7v4nM/sIcLW7Xxks2+PuOc3sdzrwD3efZmZ3A/Pc/XdxaJKIiDQjnsFyEvADd/9Y8PmbAO7+k7B1FgKT3L3SzAzY6e55wbJDgiVYpwro5e71Tb/jwxQUFHi/fv2i2DoRkfZv1qxZW9y98HDrpcWjmEAxUBn2eS1wQpN15gGXAL8BLgZyzay7u28FssysHKgHbnf3fwLdgR3uXh+2z+LDFdKvXz/Ky8uPqjEiIh2Nma1uzXrJdvH+68DpZjYHOB1YBzQEy/q6exnwKeDXZnZsJDs2s2vNrNzMyquqqqJatIiIvC+ewbIOKA37XBLMO8jd17v7Je4+Gvh2MG9H8L4ueF8BvAyMBrYC+WaW9mH7DNv3Pe5e5u5lhYWHPZITEZEjFM9gmQkMCO7iygCmAE+Er2BmBWZ2oKZvAvcH87uaWeaBdYBTgPc8dIHoJeDSYJvPAI/HvCUiIvKh4naNJbi4fj3wLJAK3O/uC83sNqDc3Z8AzgB+YmYOvAp8Kdh8MPB7M2skFIa3h91N9g1gmpn9DzAH+EO82iQiHUddXR1r166luro60aXEXFZWFiUlJaSnpx/R9nG7KyyZlJWVuS7ei0gkVq5cSW5uLt27dyd0Q2r75O5s3bqV3bt3079//w8sM7NZwbXuFiXbxXsRkaRUXV3d7kMFwMzo3r37UR2ZKVhERFqpvYfKAUfbTgVLBJ6ct56/vt2q27hFRDosBUsEnlmwkV89v5T6hsZElyIiHcyOHTu46667It5u8uTJ7NixIwYVfTgFSwTOG1HElj21vL1yW6JLEZEO5sOCpb6+vpm13/fUU0+Rn58fq7KapWCJwJmDetA5I5Un561PdCki0sHceuutLF++nFGjRjFu3DgmTJjABRdcwJAhQwC46KKLGDt2LEOHDuWee+45uF2/fv3YsmULq1atYvDgwVxzzTUMHTqUiRMnsn///pjUGs++wtq8rPRUJg7txdMLNnLbhcPISFMui3RE//3kQt5bvyuq+xzSO4/vnz/0Q5fffvvtLFiwgLlz5/Lyyy9z7rnnsmDBgoO3BN9///1069aN/fv3M27cOD7+8Y/TvXv3D+xj2bJlPPzww9x7771cdtllPProo1xxxRVRbQfoiCVi548sYuf+Ot6o2JLoUkSkAxs/fvwHnjO54447GDlyJCeeeCKVlZUsW7bskG369+/PqFGjABg7diyrVq2KSW06YonQqccV0qVTOk/OW8+Zg3okuhwRSYCWjizipXPnzgenX375Zf7zn/8wY8YMsrOzOeOMM5p9DiUzM/PgdGpqasxOhemIJUIZaSlMGtqL597bRHVdw+E3EBGJgtzcXHbv3t3ssp07d9K1a1eys7NZvHgxb731Vpyr+yAFyxE4f2Rv9tTU8/KSzYkuRUQ6iO7du3PKKacwbNgwbr755g8smzRpEvX19QwePJhbb72VE088MUFVhqivsCNQ39DICT9+gROP7c6dnxoTxcpEJFktWrSIwYMHJ7qMuGmuveorLIbSUlOYPLyIFxZtYm9Ny/eQi4h0NAqWI3T+yN5U1zXyn0WbEl2KiEhSUbAcobK+XemVl8W/5m9IdCkiEicd5dLB0bZTwXKEUlKMc0cU8cqSKnbur0t0OSISY1lZWWzdurXdh8uB8ViysrKOeB96juUonD+yN394fSXPLdzIJ8pKE12OiMRQSUkJa9eupaqqKtGlxNyBESSPlILlKIws6UJpt048OX+DgkWknUtPTz9kREVpnk6FHQUz47wRvXmjYgvb9tYmuhwRkaSgYDlK54/oTUOj8/QCXcQXEQEFy1EbXJTLsYWd1ZW+iEggrsFiZpPMbImZVZjZrc0s72tmL5jZfDN72cxKgvmjzGyGmS0Mln0ybJsHzGylmc0NXqPi3CbOG9Gbt1duY/OuQzt9ExHpaOIWLGaWCtwJnAMMAaaa2ZAmq/0C+LO7jwBuA34SzN8HfNrdhwKTgF+bWfiQaDe7+6jgNTemDWnG+SOLcId/v6vTYSIi8TxiGQ9UuPsKd68FpgEXNllnCPBiMP3SgeXuvtTdlwXT64HNQGFcqm6F43rkMrgoT6fDRESIb7AUA5Vhn9cG88LNAy4Jpi8Gcs3sA0Ogmdl4IANYHjb7R8Epsl+ZWSYJcP7IImav2cHa7fsS8fUiIkkj2S7efx043czmAKcD64CDg56YWRHwIHC1uzcGs78JDALGAd2AbzS3YzO71szKzaw8Fg84nTe8NwD/VhcvItLBxTNY1gHhTxGWBPMOcvf17n6Ju48Gvh3M2wFgZnnAv4Fvu/tbYdts8JAa4I+ETrkdwt3vcfcydy8rLIz+WbQ+3bMZWZrPk/N1OkxEOrZ4BstMYICZ9TezDGAK8ET4CmZWYGYHavomcH8wPwN4jNCF/UeabFMUvBtwEbAgpq1owfkjiliwbhcrqvYkqgQRkYSLW7C4ez1wPfAssAiY7u4Lzew2M7sgWO0MYImZLQV6Aj8K5l8GnAZc1cxtxX81s3eBd4EC4H/i06JDnTuiCEA9HotIh6YRJKPssrtnsGN/Lc999fSY7F9EJFE0gmSCnD+yiKWb9rBk4+5ElyIikhAKlig7Z3gRKYaeaRGRDkvBEmUFOZmcfGwBT85f3+4HBBIRaY6CJQYuGNWb1Vv3MXvNjkSXIiISdwqWGDhnWC+y0lN4bM7aRJciIhJ3CpYYyM1K52NDe/HkvA3U1DccfgMRkXZEwRIjl4wpYef+Ol5avDnRpYiIxJWCJUZOObY7PXIzeXT2usOvLCLSjihYYiQtNYWLRhfz0uLNbNtbm+hyRETiRsESQxePLqa+0fmXOqYUkQ5EwRJDg4vyGFyUp9NhItKhKFhi7ONjiplXuYOKzerxWEQ6BgVLjF0wqjcphp5pEZEOQ8ESYz1yszhtYCH/nLOexkZ18SIi7Z+CJQ4uHl3Muh37eXvltkSXIiIScwqWOJg4pBc5mWn8Y7ZOh4lI+6dgiYNOGalMHt6Lp97dwP5adfEiIu2bgiVOLhlTwt7aBp57b2OiSxERiSkFS5yM79eN4vxOeqZFRNo9BUucpKQYl4wp5vVlVWzeVZ3ockREYkbBEkcXjy6m0eHxueriRUTaLwVLHB1TmMOo0nwe1d1hItKOxTVYzGySmS0xswozu7WZ5X3N7AUzm29mL5tZSdiyz5jZsuD1mbD5Y83s3WCfd5iZxas9R+LjY4pZvHE3763flehSRERiIm7BYmapwJ3AOcAQYKqZDWmy2i+AP7v7COA24CfBtt2A7wMnAOOB75tZ12Cb3wHXAAOC16QYN+WonDeiN+mppmdaRKTdiucRy3igwt1XuHstMA24sMk6Q4AXg+mXwpZ/DHje3be5+3bgeWCSmRUBee7+lrs78Gfgolg35Gh07ZzBRwb14J9z11Pf0JjockREoi6ewVIMVIZ9XhvMCzcPuCSYvhjINbPuLWxbHEy3tE8AzOxaMys3s/KqqqojbkQ0XDy6hC17ani9YktC6xARiYVku3j/deB0M5sDnA6sA6LyqLq73+PuZe5eVlhYGI1dHrEzBxWSn53OP/RMi4i0Q/EMlnVAadjnkmDeQe6+3t0vcffRwLeDeTta2HZdMP2h+0xGmWmpnD+iN88u3Mju6rpElyMiElXxDJaZwAAz629mGcAU4InwFcyswMwO1PRN4P5g+llgopl1DS7aTwSedfcNwC4zOzG4G+zTwOPxaMzRumRMMTX1jTz9rrp4EZH2JW7B4u71wPWEQmIRMN3dF5rZbWZ2QbDaGcASM1sK9AR+FGy7DfghoXCaCdwWzAP4L+A+oAJYDjwdnxYdnVGl+RxT0FnPtIhIu2Ohm6k6lrKyMi8vL090Gfz2xWX84rmlPPuV0zi+V26iyxERaZGZzXL3ssOtl2wX7zuUK07sS25WGr94bkmiSxERiRoFSwLlZ2fwhdOO4fn3NjF7zfZElyMiEhUKlgS7+pT+FORk8LNnFtMRT0uKSPujYEmwzplpXH/mcby1YhuvLdMDkyLS9ilYksDUE/pQnN+Jnz+7REctItLmKViSQGZaKl87eyDvrtvJ0wv0XIuItG0KliRx0ehiBvbM4RfPLVHnlCLSpilYkkRqinHTxONZUbVXD02KSJumYEkiE4f0ZFRpPr/+zzKq66LS96aISNwpWJKImXHLpOPZsLOav7y1OtHliIgcEQVLkjn52AImDCjgzpcq1POxiLRJCpYkdPPHjmf7vjrue21loksREYmYgiUJjSjJ55xhvbjvtRVs3VOT6HJERCKiYElSN008nv11Ddz18vJElyIiEhEFS5I6rkcOl44t4cEZq1m3Y3+iyxERaTUFSxK78aMDAfjNf5YmuBIRkdZTsCSx4vxOXHFiXx6ZtZaKzXsSXY6ISKsoWJLcl848lk7pqfzyeQ0GJiJtg4IlyXXPyeTzE47hqXc3Mn/tjkSXIyJyWAqWNuDzE/rTrXMGP39WRy0ikvwULG1AblY6XzrzOF5btoU3KjQYmIgkt7gGi5lNMrMlZlZhZrc2s7yPmb1kZnPMbL6ZTQ7mX25mc8NejWY2Klj2crDPA8t6xLNN8XJ5MBiYhjAWkWQXt2Axs1TgTuAcYAgw1cyGNFntO8B0dx8NTAHuAnD3v7r7KHcfBVwJrHT3uWHbXX5gubtvjnljEiArPZWvfHQA89bu5BkNBiYiSSyeRyzjgQp3X+HutcA04MIm6ziQF0x3AdY3s5+pwbYdziVjShjQI4efazAwEUli8QyWYqAy7PPaYF64HwBXmNla4Cnghmb280ng4Sbz/hicBvuumVlzX25m15pZuZmVV1VVHVEDEi01xfj6xzQYmIgkt2S7eD8VeMDdS4DJwINmdrBGMzsB2OfuC8K2udzdhwMTgteVze3Y3e9x9zJ3LyssLIxdC2JMg4GJSLKLZ7CsA0rDPpcE88J9DpgO4O4zgCygIGz5FJocrbj7uuB9N/AQoVNu7ZaZ8Y1Jg9iws5oHZ2gwMBFJPhEFi5llmNkFZnaTmXUJ5vUzs/xWbD4TGGBm/c0sg1BIPNFknTXAWcF+BxMKlqrgcwpwGWHXV8wszcwKgul04DxgAe3cScd257SBhdz5cgW7NBiYiCSZVgeLmfUB5hM6Yvgp0D1Y9BXg9sNt7+71wPXAs8AiQnd/LTSz28zsgmC1m4BrzGxe8D1X+fv31p4GVLr7irDdZgLPmtl8YC6hI6B7W9umtuyWjx3Pjn113PvqisOvLCISR9baZyLM7FGgHvg0sAUY6e4rzOxM4PfuPjB2ZUZXWVmZl5eXJ7qMo3b9Q7N5YdFmXr3lTApzMxNdjoi0c2Y2y93LDrdeJKfCJgC3uXvTIQ1XcujdXRIHN008ntqGRn774rJElyIiclAkwdIJqG1mfiFQHZ1yJBL9CzrzyXGlPPTOGtZs3ZfockREgMiC5U1CtwMfcOAc2o3Aq1GrSCJy41kDSDHjVxoMTESSRCTB8i3gZjN7CEgDvmlmbwHnE+qKRRKgZ14WV5/Sn3/OXceiDbsSXY6ISOuDxd1nAScANcBy4FRgKXCCuy+MTXnSGl88/VhyM9P4hbrVF5EkkBbJyu7+HnB1jGqRI9QlO53rzjiWnz2zhJmrtjGuX7dElyQiHVgkz7H0NrPeYZ9Hm9nPzeyqmFQmEbn65P70yM3kp0+rW30RSaxIrrE8BEwCMLPuwIvAucAdZnZzDGqTCHTKSOXLZw2gfPV2XlnaNjvZFJH2IZJgGQ68HUxfAqxw9yHAZ4DPR7swidwnx5VSkJPJX95ak+hSRKQDiyRYsoEDtx19BHgymC7ng51LSoKkp6Zw6dgSXlqymU279GiRiCRGJMGyEjjNzHKAs4H/BPMLgd3RLkyOzCfHldLQ6DwyS+O1iEhiRBIsvwQeIDRY1xrgjWD+aXSAHoXbiv4FnTnxmG5Mm7mGxkZdxBeR+IvkOZb7CD3HcjUwIazX4RWERn6UJDF1fB8qt+1nxoqtiS5FRDqgiMZjcffZ7v5Pd98bNu8Jd38t+qXJkfrY0F506ZTOw+/oIr6IxF9ED0ia2SBCF+570CSU3P17UaxLjkJWeioXjy7mobfXsG1vLd06ZyS6JBHpQCJ5QPJLwELgfwiN/viJsNelMalOjtiU8aXUNjTyj9m6iC8i8RXJqbBvAt8Gurv7IHcfHPYaEqP65AgN6pXHqNJ8ps2s1JP4IhJXkT7HMt31V6rNmDq+lIrNe5i9ZnuiSxGRDiSSYPkLcMFh15Kkcd6I3nTOSOXhdyoTXYqIdCCRXLz/OvBYMMb9fKAufKG73xbNwuTodc5M44JRvfnnnPV87/wh5GWlJ7okEekAIgmWzwPnEHrKfijvjyBJMK1gSUJTxvXh4XcqeWLueq44sW+iyxGRDiCSU2HfJzRSZL67H+fuA8JeA1uzAzObZGZLzKzCzG5tZnkfM3vJzOaY2XwzmxzM72dm+81sbvC6O2ybsWb2brDPO8zMImhTuzeipAuDeuUybaaeaRGR+IgkWDKAaUd68d7MUoE7CR31DAGmmlnTu8m+Q+gGgdGEbmm+K2zZcncfFbyuC5v/O+AaYEDwmnQk9bVXZsbU8X1YsG4XC9btTHQ5ItIBRBIsD3N0F+/HAxXuvsLda4FpwIVN1nEgL5juAqxvaYdmVgTkuftbQeD9GbjoKGpsly4aVUxmWoqOWkQkLiK5xrIF+J6ZnQbM49CL9z8+zPbFhDqwPGAtob7Hwv0AeM7MbgA6Ax8NW9bfzOYQ6rr/O0E3MsXBfsL3Wdyq1nQgXbLTmTy8iMfnrOdbkweTnRFRhwsiIhGJ5IjlSmAnMBq4itDppwOvaA30NRV4wN1LgMnAg2aWAmwA+gSnyL4GPGRmeS3s5xBmdq2ZlZtZeVVVxxthccq4UnbX1PPUuxsTXYqItHOtChYzSweygPPcvX8zr2NasZt1fHBAsJJgXrjPAdMB3H1G8J0F7l7j7luD+bOA5cDAYPuSw+yTYLt73L3M3csKCwtbUW77Mr5/N44p6Mw0dUwpIjHWqmBx9zpC1z8ajuK7ZgIDzKy/mWUQujj/RJN11gBnAZjZYELBUmVmhcHFf8zsGEIX6Ve4+wZgl5mdGNwN9mng8aOosd0yMz45rpTy1dup2Kxx2UQkdiI5FXYf8OUj/SJ3rweuB54FFhG6+2uhmd1mZgduCrgJuMbM5hG6WeCq4KL8acB8M5sLPAJc5+7bgm3+K6itgtCRzNNHWmN79/GxJaSlGNP0JL6IxJC19u5hM7uPUE/GG4BZwN7w5e5+bdSri5GysjIvLy9PdBkJ8cW/zOKtFVt561tnkZmWmuhyRKQNMbNZ7l52uPUiOWI5FphNKFh68/5zIwOA446kSIm/KeP7sH1fHc+/tynRpYhIO9Xq+07d/cxYFiLxMeG4AorzOzHtnUrOG9E70eWISDsU0dDE0valpBiXlZXyesUWKrftS3Q5ItIOKVg6oMvGlZBicPMj89i8uzrR5YhIO6Ng6YCKunTiZ5eOZG7lDib/5nVeX7Yl0SWJSDuiYOmgLh1bwuNfOpWu2elcef/b/O9zS6hvaEx0WSLSDihYOrDje+Xy+PWn8ImxJfzfixV86t632bBzf6LLEpE2TsHSwWVnpPGzS0fyq0+OZMH6nUz+zWu8uFi3IovIkVOwCAAXjy7hXzecSq8unfjsA+X8+KlF1Nbr1JiIRE7BIgcdU5jDY/91Mlee2Jd7Xl3BZb+foVuSRSRiChb5gKz0VH540TDuunwMyzfvYfIdr/Hq0o43zICIHDkFizRr8vAinrpxAr3ysrjlkflU1x1Nx9Yi0pEoWORDlXbL5r8vGMrGXdUax0VEWk3BIi066djunNC/G3e+vFxHLSLSKgoWaZGZ8bWzB1K1u4a/vLU60eWISBugYJHDOuGY7px6XAF3v7KcfbX1iS5HRJKcgkVa5atnD2DLnloenKGjFhFpmYJFWmVs326cPrCQu19Zzp4aHbWIyIdTsEirffXsgWzfV8ef3lyV6FJEJIkpWKTVRpXmc9agHtzz6gp2VdcluhwRSVIKFonIV88eyM79dfzx9VWJLkVEklRcg8XMJpnZEjOrMLNbm1nex8xeMrM5ZjbfzCYH8882s1lm9m7w/pGwbV4O9jk3ePWIZ5s6mmHFXZg4pCf3vb6Cnft01CIih4pbsJhZKnAncA4wBJhqZkOarPYdYLq7jwamAHcF87cA57v7cOAzwINNtrvc3UcFr80xa4QAoaOW3dX1/OH1FYkuRUSSUDyPWMYDFe6+wt1rgWnAhU3WcSAvmO4CrAdw9znuvj6YvxDoZGaZcahZmjG4KI9zhxdx/xur2L63NtHliEiSiWewFAOVYZ/XBvPC/QC4wszWAk8BNzSzn48Ds929JmzeH4PTYN81M4tizfIhbvzoAPbW1nPvazpqEZEPSraL91OBB9y9BJgMPGhmB2s0s6HAT4EvhG1zeXCKbELwurK5HZvZtWZWbmblVVXqBv5oDeyZy/kjevPAm6vYuqfm8BuISIcRz2BZB5SGfS4J5oX7HDAdwN1nAFlAAYCZlQCPAZ929+UHNnD3dcH7buAhQqfcDuHu97h7mbuXFRYWRqVBHd2XzxpAdV0D97yqoxYReV88g2UmMMDM+ptZBqGL8080WWcNcBaAmQ0mFCxVZpYP/Bu41d3fOLCymaWZ2YHgSQfOAxbEvCUCwHE9crhoVDF/mrGKqt06ahGRkLgFi7vXA9cDzwKLCN39tdDMbjOzC4LVbgKuMbN5wMPAVe7uwXbHAd9rcltxJvCsmc0H5hI6Aro3Xm0SuOGsAdQ1OHe/svzwK4tIh2Chv9sdS1lZmZeXlye6jHbj5r/P44l563n1ljPpmZeV6HJEJEbMbJa7lx1uvWS7eC9t0A0fGUBDo/PL55bSEf9HRUQ+SMEiR61P92yuOrkffyuv5FuPLaC+oTHRJYlIAqUlugBpH741eTCZ6Snc+dJy1u/Yz52XjyEnU/96iXREOmKRqEhJMW7+2CB+cslwXq/YwmV3z2DTrupElyUiCaBgkaiaOr4Pf/hMGau37uWiO99g8cZdiS5JROJMwSJRd8bxPZh+3Uk0uvOJ383g9WVbEl2SiMSRgkViYmjvLjz2X6dQ3LUTV/3xHaaXVx5+IxFpFxQsEjO98zvx9+tO4qRju3PLI/P55XNLdDuySAegYJGYys1K5/6rxnFZWQl3vFjBTdPnUVuv25FF2jPdDyoxl56awk8/PoLSrtn87/NLWbtjP7+ZMoqiLp0SXZqIxICOWCQuzIwbzhrAb6aM4t21O5n069f41/z1h99QRNocBYvE1YWjinnqxgn0L+jM9Q/N4at/m8uu6rpElyUiUaRgkbjrX9CZR647ia98dABPzFvPOb9+jbdXbE10WSISJQoWSYi01BS+8tGBPHLdSaSnGlPufYufPL2ImvqGRJcmIkdJwSIJNbpPV/795QlMGVfK719ZwcV3vsnSTbsTXZaIHAWNxyJJ4/n3NnHro/PZXVPPrZMGcdXJ/UhJMQDqGxrZsqeWjbuq2RS8Nu6sZtOuGmrqG/ja2QM5pjAnwS0Qad9aOx6LgkWSStXuGr7x6HxeXLyZIUV5pKUaG3dWs2VPDY1N/lVNSzF65Gayu7qezPRU/vr5Ezi+V25iChfpABQsLVCwJDd356F31vC3mZXkZ2fQMzeTXl2y6JmXRa+80HvPLpkUdM4kJcWo2LyHy+97i9r6Rh783AkMK+6S6CaItEsKlhYoWNqf1Vv38ql732ZXdR0PXD2esX27JrokkXZHQxNLh9K3e2emX3cS3TtncOUf3mbGct2+LJIoChZpN4rzOzH9CydRnB/qUfmVpVWJLkmkQ4prsJjZJDNbYmYVZnZrM8v7mNlLZjbHzOab2eSwZd8MtltiZh9r7T6lY+mRl8W0a0/k2MIcrvlTOc8t3JjokkQ6nLgFi5mlAncC5wBDgKlmNqTJat8Bprv7aGAKcFew7ZDg81BgEnCXmaW2cp/SwXTPyeTha05kSO88vvjX2Tw5T32SicRTPI9YxgMV7r7C3WuBacCFTdZxIC+Y7gIc+ItwITDN3WvcfSVQEeyvNfuUDqhLdjp/+fwJjO3blRunzeHvGmhMJG7iGSzFQPh/3WuDeeF+AFxhZmuBp4AbDrNta/YpHVROZhp/uno8pxxXwM2PzOfBGasSXZJIh5Bs47FMBR5w9/81s5OAB81sWDR2bGbXAtcC9OnTJxq7lDagU0Yq9366jOsfms13H1/ID/+1CDNIMSPFICXF3p82w8xISzEuHNWbmyYeT0aa7m8RiVQ8g2UdUBr2uSSYF+5zhK6h4O4zzCwLKDjMtofbJ8H+7gHugdBzLEfWBGmLstJT+d0VY/nzjNWhJ/gbnUZ3Gh0a3fHg/cC8Lbtr+P2rK3hr5TZ+O3U0pd2yE90EkTYlnsEyExhgZv0J/fGfAnyqyTprgLOAB8xsMJAFVAFPAA+Z2S+B3sAA4B3AWrFPEdJTU/jcqf1bvf7T727glkfnM/mO1/j5pSOYNKwohtWJtC9xO85393rgeuBZYBGhu78WmtltZnZBsNpNwDVmNg94GLjKQxYC04H3gGeAL7l7w4ftM15tkvbrnOFF/PuGCRxT0Jnr/jKb7z++QF36i7SSunSFcTIcAAAPiUlEQVQRaUFtfSO3P72Y+99YybDiPH47dQz9CjonuiyRhFCXLiJRkJGWwvfOH8K9ny6jctt+zvu/1/nX/Miei2lodBqbds0s0o4l211hIknp7CE9+feXT+XLD8/h+ofm8ObyrXzvvCFkpaceXKex0Vm3Yz9LN+1m6aY9wftuKjbvISczjctP7MsVJ/ahR25WAlsiEns6FSYSgbqGRn7x3BJ+/8oKBvXK5eLRxVRsDoXIss172Ff7/nWYXnlZDOyVy8AeOazaupcXFm8mPSWF80f25rOn9mNob3XvL22Lus1vgYJFjtZLizfztelz2b6vjoKcTAb2zGFgz1wG9szl+F45HNcjly6d0j+wzcote/nTm6uYXl7JvtoGTujfjc+e2p+PDu5JajBSZjJpbHT21tazr7aBPTX17KtpYH9dA4OLcsnNSj/8DqTdUbC0QMEi0bC/NvSHtlvnjIi227m/jukzK3ngzVWs27Gf0m6duOrk/lxWVvKBP9h7a+qp3L6Pym37qdy2L5gOfd5VXceIki6M69eNE/p3Z3BRLmmpkV0y3ba3ltmrtzNrzXbmr93Btr117KutZ29NA3tr6tlf1/xdcIW5mXz//CGcO7wIs+QLRIkdBUsLFCySDOobGnn+vU384fWVlK/eTk5mGicd253Nu2tYu20fW/fWfmD97IxUSrtmU9otm+yMVOZUbqdy234g1H3NmL5dOaF/N8b168aIki4fuP7T0Ogs3bSb2Wu2M2v1duas2cHKLXuB0BDPg4vy6JmXSefMNLIz0sjJTCU7I43Omal0zkyjc0YanTPTaHTn/15cxoJ1uzjz+EJ+eNEwSrrqAdKOQsHSAgWLJJt5lTv44xsrmb92J73zO1HarROl3bIPBklp105065xxyBHChp37eWflNmau2sY7K7exdNMeIHQ326iSfIb0zqNi8x7mVu5gT009AAU5GYzp05Uxfbsypk/XQ0LocOobGnngzVX88vmluMPXzh7I1af0i/iISdoeBUsLFCzSXm3fW0v56u28s3Ir76zazuINuzi2MIexfbsypm8+Y/t0o7Rbp6icwlq7fR/ff3whLyzezNDeefzkkuGMKMmPQiskWSlYWqBgEYkOd+fpBRv5wRML2bKnhs+c3I+bJh5PTmbzTzLs3FfH/HU7mFe5g3lrdzKvcgcOTBzSk3OGFXHCMd1I15FP0lKwtEDBIhJdu6rr+PkzS/jL26vplZfFbRcOY8KAAhau38ncyp3MXxsKk1Vb9x3c5pjCzowqyaemvpEXF29mf10D+dnpnD24J5OHF3Hycd3JTGv9KTqJPQVLCxQsIrExa/V2vv3YuyzeuJvUFKMh6HGgV14WI0u7MKIkn1Gl+Qwr7vKB27H31zbwytIqnlmwgRcWbWZ3TT25mWmcNbgHk4YVccbxhQevA9U1NLJxZzWV2/axdvt+KrcH78HnPTX1nNC/G2cO6sGZg3pQnN8pIf8s2iMFSwsULCKxU9fQyENvr2Hz7mpGluQzsjSfnnmt722gpr6BNyu28tS7G3h+0SZ27KujU3oqg4ty2bSrho27qg8GFkCKQVGXTpR07URJ12wy01N4fdkW1mwLHR0N6pXLGcf34CODejCmT75uMjgKCpYWKFhE2oa6hkbeXrGNpxdsYNmmPfTOz6K0WzYlXTtR2jWbkq7ZFOVnHXJdxt1ZXrWXlxZv5sXFm5m5ahv1jU6XTumcNrCQjwwqZMKAQrIzUqlrcBoanfqGRuoanYYGp66xkfoGp76xEXc4rkdORHfOtVcKlhYoWEQ6ll3Vdby+bAsvLt7My0uq2LKnJqLtczLT+OjgHpw7ojcTBhR02JBRsLRAwSLScTU2OgvW7+SdldtoaHRSU4z01JTg3UhLSSEteD9wnejVpVU8s3AjO/fXkZuZxkeH9OTc4UVMGFjQoW4wULC0QMEiIpGqa2jkzeVb+ff89Ty7cNPBkDl7aChkTh3wwZCpa2hkb009u6vr2VMTvKrr2V1TT01dAw4Q/Pl1QkNkOwTvoc/52emM7tOV3l2ykqL7HAVLCxQsInI0ausbeXP5Fv49fwPPvReETFYaBTmZQZDUUV3XGLXv65mXyejS0EOuY/p0ZVhxZL0lRIuCpQUKFhGJltr6Rt5YvoXnFm5kT00DOZlp5GalkZMZvLLSyMtKIycznZysUD9s4Uc2ZmBm2IFpLHiHTbtqmFO5ndmrtzN7zY6Dd7qlpxpDendhTJ98RvfpysiSLhTnd4r5HW8KlhYoWESkLaraXcOcNaGQmbNmO/PX7jzYC3VqilHUJYs+B/uYC/qbCz4X5Bza11ykWhssGkFSRKSNKMzNZOLQXkwc2gsIXcdZsnE3763fReX2fazZFhpa4cUlm6na/cE73zqlp1LStRO/v3IsxxTmxLROBYuISBuVnprCsOIuDCs+dDTS/bUNrN0eGsdnzdZ9VAa9E+RnRzZ+0JGIa7CY2STgN0AqcJ+7395k+a+AM4OP2UAPd883szOBX4WtOgiY4u7/NLMHgNOBncGyq9x9bgybISKS9DplpDKgZy4DeubG/bvjFixmlgrcCZwNrAVmmtkT7v7egXXc/ath698AjA7mvwSMCuZ3AyqA58J2f7O7PxLzRoiIyGHFs9Oc8UCFu69w91pgGnBhC+tPBR5uZv6lwNPuvq+ZZSIikmDxDJZioDLs89pg3iHMrC/QH3ixmcVTODRwfmRm883sV2aWGY1iRUTkyCRrN59TgEfcvSF8ppkVAcOBZ8Nmf5PQNZdxQDfgG83t0MyuNbNyMyuvqqqKTdUiIhLXYFkHlIZ9LgnmNae5oxKAy4DH3L3uwAx33+AhNcAfCZ1yO4S73+PuZe5eVlhYeEQNEBGRw4tnsMwEBphZfzPLIBQeTzRdycwGAV2BGc3s45DrLsFRDBZ68uciYEGU6xYRkQjE7a4wd683s+sJncZKBe5394VmdhtQ7u4HQmYKMM2bdAlgZv0IHfG80mTXfzWzQkI9IMwFrotdK0RE5HDUpYuIiLSK+gprgZlVAauPcPMCYEsUy0mk9tKW9tIOUFuSVXtpy9G2o6+7H/YidYcMlqNhZuWtSey2oL20pb20A9SWZNVe2hKvdiTr7cYiItJGKVhERCSqFCyRuyfRBURRe2lLe2kHqC3Jqr20JS7t0DUWERGJKh2xiIhIVClYImBmk8xsiZlVmNmtia7nSJnZKjN718zmmlmbeqDHzO43s81mtiBsXjcze97MlgXvXRNZY2t9SFt+YGbrgt9mrplNTmSNrWFmpWb2kpm9Z2YLzezGYH6b+11aaEtb/F2yzOwdM5sXtOW/g/n9zezt4O/Y34KeUKL73ToV1jrBeDJLCRtPBpgaPp5MW2Fmq4Ayd29z9+Wb2WnAHuDP7j4smPczYJu73x4Efld3b7Yz0mTyIW35AbDH3X+RyNoiEXSrVOTus80sF5hFqHulq2hjv0sLbbmMtve7GNDZ3feYWTrwOnAj8DXgH+4+zczuBua5+++i+d06Ymm9SMeTkRhw91eBbU1mXwj8KZj+E6E/BEnvQ9rS5gQdwc4OpncDiwgNidHmfpcW2tLmBJ3z7gk+pgcvBz4CHBgYMSa/i4Kl9Vo9nkwb4MBzZjbLzK5NdDFR0NPdNwTTG4GeiSwmCq4Pxhe6vy2cPgoX9Ok3GnibNv67NGkLtMHfxcxSzWwusBl4HlgO7HD3+mCVmPwdU7B0TKe6+xjgHOBLwSmZdiHovLQtn9/9HXAsoaG4NwD/m9hyWs/McoBHga+4+67wZW3td2mmLW3yd3H3BncfRWiYkvGExq6KOQVL60UynkxSc/d1wftm4DE+ZAybNmRT2PAJRYT+76xNcvdNwR+DRuBe2shvE5zDfxT4q7v/I5jdJn+X5trSVn+XA9x9B/AScBKQb2YHeraPyd8xBUvrtWo8mWRnZp2Di5KYWWdgIm1/DJsngM8E058BHk9gLUflwB/iwMW0gd8muEj8B2CRu/8ybFGb+10+rC1t9HcpNLP8YLoToRuPFhEKmEuD1WLyu+iusAgEtxj+mvfHk/lRgkuKmJkdQ+goBULj8TzUltphZg8DZxDqpXUT8H3gn8B0oA+hXqsvc/ekvyj+IW05g9DpFgdWAV8Iu06RlMzsVOA14F2gMZj9LULXJtrU79JCW6bS9n6XEYQuzqcSOoiY7u63BX8DphEayn0OcEUwAm/0vlvBIiIi0aRTYSIiElUKFhERiSoFi4iIRJWCRUREokrBIiIiUaVgEWljzOwMM3MzK0l0LSLNUbCIiEhUKVhERCSqFCwiETKzG8xssZlVB4NYfftA30vBIGo/MrP7zGyXmW0xsx+bWUrY9rlm9nszqzKzGjMrN7OJTb6jh5n90cw2Bd+zxMw+26SUwWb2qpntCwamOqfJPr5lZiuC76gys2eDrj1EYirt8KuIyAHBQFxXA18B5gKDgbuBLOC7wWo3EOr6ZxyhzgrvJtRly2+C5fcHy64A1gDXAf8ysxHuvjj44/8KsB+4HFgBHEeoC45wvwC+Qagr9G8BfzOzvu6+3cwuAW4Ntp8XbHtGtP45iLREXbqItJKZZQNbgEvc/Zmw+Z8G7nD3/GB0zkp3nxC2/MfAle5eambHAcuAc939qbB1ZgNz3f2zZvY54E7gOHdf20wdZxDqSPDjB3rfNbOehMY8meTuz5rZV4EvAkPdvS66/yREWqZTYSKtNxToBDxqZnsOvIDfA13MrDBYb0aT7d4ASswsDxgSzHu1yTqvBvsHGAu811yoNDH3wIS7bwIaeH8wremERgxcbWYPmNmVB3q1Fok1nQoTab0D/yP2CWBpM8vj3XNvbTPzUiA05o6ZDQLOJDQU7XeBn5rZCe5e2cx2IlGjIxaR1lsIVAPHuHtFM6+GYL0Tm2x3MrAuGIlwYTCv6aidp/H+GB+zgCFH+5yKu9e4+zPufgswHMimDYw7L22fjlhEWsnd9wTXS35sZg78h9B/Q8OB0e7+jWDVUcFF/oeAMuBGggv77r7czP4O3GVmXyA0TskXgWHAp4LtHwZuAZ4ws1sIXZw/Bihw97+1ptbgOk0K8A6wAzgLyAXeO/J/AiKto2ARiYC7/9DMNgDXExr3fD+h02IPhK32f0BfoByoA37L+3eEAXwe+DnwFyCP0KBS57n74uA79pnZ6cDPCA3IlENocKnbIyh1O/D1YB+ZhO4su9bdX4hgHyJHRHeFiURRcFfYfe7+P4muRSRRdI1FRESiSsEiIiJRpVNhIiISVTpiERGRqFKwiIhIVClYREQkqhQsIiISVQoWERGJKgWLiIhE1f8DWJtknTJ6aFUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Model Fit\n",
    "train_time= model.fit(Xtr, Xtst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During training, we can optionlly evauate the root mean squared error to have an idea of how the learning is proceeding. We would generally like to see this quantity decreasing as a function of the learning epochs. To visualise this choose `with_metrics = True` in the `RBM()` model function. \n",
    "\n",
    "Once the model has been trained, we can predict new ratings on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": [
     "top_k"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Extracting top 10 elements\n",
      "Done recommending items, time 0.1904082\n"
     ]
    }
   ],
   "source": [
    "#number of top score elements to be recommended  \n",
    "K = 10\n",
    "\n",
    "#Model prediction on the test set Xtst. \n",
    "top_k, test_time =  model.recommend_k_items(Xtst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`top_k` returns the first K elements having the highest recommendation score. Here the recommendation score is evaluated by multiplying the predicted rating by its probability, i.e. the confidence the algorithm has about its output. So if we have two items both with predicted ratings 5, but one with probability 0.5 and the other 0.9, the latter will be considered more relevant. In order to inspect the prediction and use the evaluation metrics in this repository, we convert both top_k and Xtst to pandas dataframe format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_k_df = am.map_back_sparse(top_k, kind = 'prediction')\n",
    "test_df = am.map_back_sparse(Xtst, kind = 'ratings')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>movieID</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4.989026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>242</td>\n",
       "      <td>4.990839</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>161</td>\n",
       "      <td>4.987231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>170</td>\n",
       "      <td>4.992252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>223</td>\n",
       "      <td>4.997586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>239</td>\n",
       "      <td>4.987444</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>4.989766</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>269</td>\n",
       "      <td>4.990007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>357</td>\n",
       "      <td>4.995893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>647</td>\n",
       "      <td>4.988303</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID  movieID  prediction\n",
       "0       1        4    4.989026\n",
       "1       1      242    4.990839\n",
       "2       1      161    4.987231\n",
       "3       1      170    4.992252\n",
       "4       1      223    4.997586\n",
       "5       1      239    4.987444\n",
       "6       1       31    4.989766\n",
       "7       1      269    4.990007\n",
       "8       1      357    4.995893\n",
       "9       1      647    4.988303"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_k_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 Evaluation metrics \n",
    "\n",
    "Here we evaluate the performance of the algorithm using the metrics provided in the `PythonRankingEvaluation` class. Note that the following metrics take into account only the first K elements, therefore their value may be different from the one displayed from the `model.fit()` method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": [
     "ranking"
    ]
   },
   "outputs": [],
   "source": [
    "def ranking_metrics(\n",
    "    data_size,\n",
    "    data_true,\n",
    "    data_pred,\n",
    "    time_train,\n",
    "    time_test,\n",
    "    K\n",
    "):\n",
    "\n",
    "    eval_map = map_at_k(data_true, data_pred, col_user=\"userID\", col_item=\"movieID\", \n",
    "                    col_rating=\"rating\", col_prediction=\"prediction\", \n",
    "                    relevancy_method=\"top_k\", k= K)\n",
    "\n",
    "    eval_ndcg = ndcg_at_k(data_true, data_pred, col_user=\"userID\", col_item=\"movieID\", \n",
    "                      col_rating=\"rating\", col_prediction=\"prediction\", \n",
    "                      relevancy_method=\"top_k\", k= K)\n",
    "\n",
    "    eval_precision = precision_at_k(data_true, data_pred, col_user=\"userID\", col_item=\"movieID\", \n",
    "                               col_rating=\"rating\", col_prediction=\"prediction\", \n",
    "                               relevancy_method=\"top_k\", k= K)\n",
    "\n",
    "    eval_recall = recall_at_k(data_true, data_pred, col_user=\"userID\", col_item=\"movieID\", \n",
    "                          col_rating=\"rating\", col_prediction=\"prediction\", \n",
    "                          relevancy_method=\"top_k\", k= K)\n",
    "\n",
    "    \n",
    "    df_result = pd.DataFrame(\n",
    "        {   \"Dataset\": data_size,\n",
    "            \"K\": K,\n",
    "            \"MAP\": eval_map,\n",
    "            \"nDCG@k\": eval_ndcg,\n",
    "            \"Precision@k\": eval_precision,\n",
    "            \"Recall@k\": eval_recall,\n",
    "            \"Train time (s)\": time_train,\n",
    "            \"Test time (s)\": time_test\n",
    "        }, \n",
    "        index=[0]\n",
    "    )\n",
    "    \n",
    "    return df_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Dataset</th>\n",
       "      <th>K</th>\n",
       "      <th>MAP</th>\n",
       "      <th>nDCG@k</th>\n",
       "      <th>Precision@k</th>\n",
       "      <th>Recall@k</th>\n",
       "      <th>Train time (s)</th>\n",
       "      <th>Test time (s)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mv 100k</td>\n",
       "      <td>10</td>\n",
       "      <td>0.40663</td>\n",
       "      <td>0.818305</td>\n",
       "      <td>0.698621</td>\n",
       "      <td>0.432109</td>\n",
       "      <td>2.541901</td>\n",
       "      <td>0.190408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Dataset   K      MAP    nDCG@k  Precision@k  Recall@k  Train time (s)  \\\n",
       "0  mv 100k  10  0.40663  0.818305     0.698621  0.432109        2.541901   \n",
       "\n",
       "   Test time (s)  \n",
       "0       0.190408  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_100k= ranking_metrics(\n",
    "    data_size = \"mv 100k\",\n",
    "    data_true =test_df,\n",
    "    data_pred =top_k_df,\n",
    "    time_train=train_time,\n",
    "    time_test =test_time,\n",
    "    K =10)\n",
    "\n",
    "eval_100k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/papermill.record+json": {
       "map": 0.40663037661752643
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/papermill.record+json": {
       "ndcg": 0.8183045030938465
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/papermill.record+json": {
       "precision": 0.6986214209968188
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/papermill.record+json": {
       "recall": 0.43210940348748195
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/papermill.record+json": {
       "train_time": 2.541900634765625
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/papermill.record+json": {
       "test_time": 0.19040775299072266
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Record results with papermill for tests\n",
    "pm.record(\"map\", eval_100k['MAP'][0])\n",
    "pm.record(\"ndcg\", eval_100k['nDCG@k'][0])\n",
    "pm.record(\"precision\", eval_100k['Precision@k'][0])\n",
    "pm.record(\"recall\", eval_100k['Recall@k'][0])\n",
    "pm.record(\"train_time\", train_time)\n",
    "pm.record(\"test_time\", test_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python (reco_full)",
   "language": "python",
   "name": "reco_full"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "nteract": {
   "version": "0.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}