Part 8_Hybrid Recommender Evaluation_in test set for userid 2043.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 8: Hybrid Recommender Evaluation_in test set for userid 2043\n",
    "---\n",
    "\n",
    "- Importing the relevant libraries first...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import confusion_matrix, classification_report, multilabel_confusion_matrix, roc_auc_score\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import joblib\n",
    "\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing X_legit and y which contain the shops that userid 2043 rated and ratings respectively\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(980, 19307)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_legit = pd.read_csv('yelp_data/xlegit.csv')\n",
    "X_legit.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    4.0\n",
       "1    4.0\n",
       "2    5.0\n",
       "3    5.0\n",
       "4    4.0\n",
       "Name: user_rating, dtype: float64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = pd.read_csv('yelp_data/y.csv', squeeze=True)\n",
    "y.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#split the dataset into train and test sets first\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_legit, y, test_size=0.2, random_state=42, stratify=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.0    0.479592\n",
       "5.0    0.383929\n",
       "3.0    0.118622\n",
       "2.0    0.012755\n",
       "1.0    0.005102\n",
       "Name: user_rating, dtype: float64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- The baseline accuracy will be 0.48 since that is the highest proportion among the training dataset's target classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiate scaler since not all of the features are of the same scale, eg. review_count and avg_store_rating\n",
    "ss = StandardScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#fitting the train and transforming both the train and test sets\n",
    "X_train_sc = ss.fit_transform(X_train)\n",
    "X_test_sc = ss.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reading in reconstructed_X_test for userid 2043 from content-based filtering for comparison later on...\n",
    "userid2043_cb_pred_actual_X_test = pd.read_csv('yelp_data/userid2043_cb_pred_actual.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(196, 3)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the dimensions of the read-in content-based filtering dataset....\n",
    "userid2043_cb_pred_actual_X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>predicted_ratings</th>\n",
       "      <th>actual_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>shops_got-luck-cafe-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>shops_symmetry-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>shops_the-bread-project-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               shops  predicted_ratings  actual_ratings\n",
       "0      shops_got-luck-cafe-singapore                4.0             4.0\n",
       "1           shops_symmetry-singapore                4.0             4.0\n",
       "2  shops_the-bread-project-singapore                4.0             4.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the first few rows of the content-based filtering dataset....\n",
    "userid2043_cb_pred_actual_X_test.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cleaning the shops column to remove the \"shops_\" prefix for easier merging later on...\n",
    "userid2043_cb_pred_actual_X_test['shops'] = userid2043_cb_pred_actual_X_test['shops'].apply(lambda x: x[6:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>predicted_ratings</th>\n",
       "      <th>actual_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>got-luck-cafe-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>symmetry-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>the-bread-project-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         shops  predicted_ratings  actual_ratings\n",
       "0      got-luck-cafe-singapore                4.0             4.0\n",
       "1           symmetry-singapore                4.0             4.0\n",
       "2  the-bread-project-singapore                4.0             4.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#confirming that the change has been made\n",
    "userid2043_cb_pred_actual_X_test.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reading in the collaborative filtering dataset for userid2043...\n",
    "userid2043_mbcf_pred_actual_test = pd.read_csv('yelp_data/userid2043_mbcf_pred_actual.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(185, 4)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the dimensions of the model-based collaborative filtering dataset...\n",
    "userid2043_mbcf_pred_actual_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>ratings</th>\n",
       "      <th>prediction_rounded</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>the-bao-makers-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.933102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>little-farms-cafe-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.895577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>lam-yeo-coffee-powder-fty-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.869991</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  ratings  prediction_rounded  \\\n",
       "0             the-bao-makers-singapore      4.0                 4.0   \n",
       "1          little-farms-cafe-singapore      4.0                 4.0   \n",
       "2  lam-yeo-coffee-powder-fty-singapore      5.0                 5.0   \n",
       "\n",
       "   prediction  \n",
       "0    3.933102  \n",
       "1    3.895577  \n",
       "2    4.869991  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the first few rows of the model-based collaborative filtering dataset...\n",
    "userid2043_mbcf_pred_actual_test.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#let's merge both content-based filtering and model-based collaborative filtering predictions for userid 2043 together!\n",
    "con_collab_2043_tst = pd.merge(userid2043_cb_pred_actual_X_test,userid2043_mbcf_pred_actual_test,how=\"left\",on='shops')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(196, 6)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the dimensions of the merged dataset...\n",
    "con_collab_2043_tst.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#keeping only common shops present in both content-based and collaborative filtering...\n",
    "con_collab_2043_tst.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(36, 6)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#looks like we got 36 outlets in common between content-based and model-based collaborative filtering to work with for userid 2043...\n",
    "con_collab_2043_tst.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>predicted_ratings</th>\n",
       "      <th>actual_ratings</th>\n",
       "      <th>ratings</th>\n",
       "      <th>prediction_rounded</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>symmetry-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.922104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dean-and-deluca-singapore-4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.934483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>krispy-kreme-singapore-2</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.933910</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          shops  predicted_ratings  actual_ratings  ratings  \\\n",
       "1            symmetry-singapore                4.0             4.0      4.0   \n",
       "3   dean-and-deluca-singapore-4                4.0             4.0      4.0   \n",
       "15     krispy-kreme-singapore-2                4.0             3.0      3.0   \n",
       "\n",
       "    prediction_rounded  prediction  \n",
       "1                  4.0    3.922104  \n",
       "3                  4.0    3.934483  \n",
       "15                 3.0    2.933910  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the first few rows of the merged dataset that has been trimmed of NaNs...\n",
    "con_collab_2043_tst.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loading decisiontreeclassifier model\n",
    "loaded_model = joblib.load('yelp_data/dtc_gs_model.sav')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8469387755102041"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#decisiontreeclassifier for content-based filtering had a test accuracy score of 0.85\n",
    "loaded_model.best_estimator_.score(X_test_sc,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- Suggest baseline score for hybrid recommender is the average of the content-based and collaborative filtering baseline accuracies, i.e. \n",
    "    \n",
    "    $Hybrid\\ recommender\\ baseline\\ accuracy = \\frac{0.48 + 0.47}{2} = 0.48$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"yelp_data/micro-avg_for_dtc.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- The above shows equal output for micro-averaged precision and recall on the test set using the DecisionTreeClassifier (user-centered content-based filtering) and the corresponding micro-averaged F1 score, or harmonic mean of micro-averaged precision and recall is:\n",
    "    \n",
    "   DecisionTreeClassifier's micro-averaged $F_1 score = 2 \\times \\frac{micro-averaged\\ precision\\ \\times\\ micro-averaged\\ recall}{micro-averaged\\ precision\\ +\\ micro-averaged\\ recall}$ = $2 \\times \\frac{0.85 \\times 0.85}{0.85 + 0.85}$ = $0.85$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"yelp_data/ALS_F1_score.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- The above image shows the $F_1$ score of the model-based collaborative filtering (ALS) on the test set: $0.98$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#since we are not tuning the models further, let's use the respective models' F1 scores to weight each model's rating predictions!\n",
    "con_wt = 0.85 / (0.85 + 0.98)\n",
    "collab_wt = 0.98 / (0.85 + 0.98)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating a new column containing the weighted sum of rating predictions from content-based and collaborative filtering\n",
    "con_collab_2043_tst['final_rating_predictions'] = (con_collab_2043_tst['predicted_ratings']*con_wt) + (con_collab_2043_tst['prediction']*collab_wt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>predicted_ratings</th>\n",
       "      <th>actual_ratings</th>\n",
       "      <th>ratings</th>\n",
       "      <th>prediction_rounded</th>\n",
       "      <th>prediction</th>\n",
       "      <th>final_rating_predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>symmetry-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.922104</td>\n",
       "      <td>3.958285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dean-and-deluca-singapore-4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.934483</td>\n",
       "      <td>3.964914</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>krispy-kreme-singapore-2</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.933910</td>\n",
       "      <td>3.429088</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          shops  predicted_ratings  actual_ratings  ratings  \\\n",
       "1            symmetry-singapore                4.0             4.0      4.0   \n",
       "3   dean-and-deluca-singapore-4                4.0             4.0      4.0   \n",
       "15     krispy-kreme-singapore-2                4.0             3.0      3.0   \n",
       "\n",
       "    prediction_rounded  prediction  final_rating_predictions  \n",
       "1                  4.0    3.922104                  3.958285  \n",
       "3                  4.0    3.934483                  3.964914  \n",
       "15                 3.0    2.933910                  3.429088  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the new df with added column...\n",
    "con_collab_2043_tst.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#rounding the computed final rating predictions to 0 decimal place so that it can be compared to the actual ratings (which are also discrete whole numbers) via the f1 score...\n",
    "con_collab_2043_tst['final_rating_predictions_rd'] = round(con_collab_2043_tst['final_rating_predictions'],0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>predicted_ratings</th>\n",
       "      <th>actual_ratings</th>\n",
       "      <th>ratings</th>\n",
       "      <th>prediction_rounded</th>\n",
       "      <th>prediction</th>\n",
       "      <th>final_rating_predictions</th>\n",
       "      <th>final_rating_predictions_rd</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>symmetry-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.922104</td>\n",
       "      <td>3.958285</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dean-and-deluca-singapore-4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.934483</td>\n",
       "      <td>3.964914</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>krispy-kreme-singapore-2</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.933910</td>\n",
       "      <td>3.429088</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          shops  predicted_ratings  actual_ratings  ratings  \\\n",
       "1            symmetry-singapore                4.0             4.0      4.0   \n",
       "3   dean-and-deluca-singapore-4                4.0             4.0      4.0   \n",
       "15     krispy-kreme-singapore-2                4.0             3.0      3.0   \n",
       "\n",
       "    prediction_rounded  prediction  final_rating_predictions  \\\n",
       "1                  4.0    3.922104                  3.958285   \n",
       "3                  4.0    3.934483                  3.964914   \n",
       "15                 3.0    2.933910                  3.429088   \n",
       "\n",
       "    final_rating_predictions_rd  \n",
       "1                           4.0  \n",
       "3                           4.0  \n",
       "15                          3.0  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the first few rows of the df containing the rounded prediction column...\n",
    "con_collab_2043_tst.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>actual_ratings</th>\n",
       "      <th>final_rating_predictions_rd</th>\n",
       "      <th>final_rating_predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>d-good-cafe-singapore-3</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.953647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>mr-teh-tarik-eating-house-singapore-5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.944909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>jewel-coffee-singapore-13</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.941013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>meidi-ya-singapore-2</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.939960</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>192</th>\n",
       "      <td>isle-eating-house-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.939903</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     shops  actual_ratings  \\\n",
       "170                d-good-cafe-singapore-3             5.0   \n",
       "127  mr-teh-tarik-eating-house-singapore-5             5.0   \n",
       "114              jewel-coffee-singapore-13             5.0   \n",
       "98                    meidi-ya-singapore-2             5.0   \n",
       "192            isle-eating-house-singapore             5.0   \n",
       "\n",
       "     final_rating_predictions_rd  final_rating_predictions  \n",
       "170                          5.0                  4.953647  \n",
       "127                          5.0                  4.944909  \n",
       "114                          5.0                  4.941013  \n",
       "98                           5.0                  4.939960  \n",
       "192                          5.0                  4.939903  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#getting a sense of the top 5 recommendations from this hybrid system; seems like the hybrid system's predictions are identical to the actual ratings for the top 5 recommendations!\n",
    "con_collab_2043_tst[['shops','actual_ratings','final_rating_predictions_rd','final_rating_predictions']].sort_values('final_rating_predictions',ascending=False).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         2.0       0.33      1.00      0.50         1\n",
      "         3.0       1.00      1.00      1.00         4\n",
      "         4.0       1.00      0.94      0.97        16\n",
      "         5.0       1.00      0.93      0.97        15\n",
      "\n",
      "    accuracy                           0.94        36\n",
      "   macro avg       0.83      0.97      0.86        36\n",
      "weighted avg       0.98      0.94      0.96        36\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#however, the hybrid system is rather weak in predicting rating 2...but at least it performed well for the other 3 rating classes (3,4,5)\n",
    "print(classification_report(con_collab_2043_tst['actual_ratings'],con_collab_2043_tst['final_rating_predictions_rd']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>predicted_ratings</th>\n",
       "      <th>actual_ratings</th>\n",
       "      <th>ratings</th>\n",
       "      <th>prediction_rounded</th>\n",
       "      <th>prediction</th>\n",
       "      <th>final_rating_predictions</th>\n",
       "      <th>final_rating_predictions_rd</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [shops, predicted_ratings, actual_ratings, ratings, prediction_rounded, prediction, final_rating_predictions, final_rating_predictions_rd]\n",
       "Index: []"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#appears there were no rating 1 in the 36 rows of common outlets between content-based filtering and collaborative filtering..\n",
    "con_collab_2043_tst[con_collab_2043_tst['actual_ratings']==1.0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining functions for evaluation of model\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#defining function for obtaining tn, fp, fn, tp for each rating class for feeding into micro-avg precision and recall functions defined below\n",
    "def cm_spec(y_true,y_pred,rating,state):\n",
    "    if state=='tn':\n",
    "        return multilabel_confusion_matrix(y_true,y_pred)[rating-2][0][0]\n",
    "    elif state=='fp':\n",
    "        return multilabel_confusion_matrix(y_true,y_pred)[rating-2][0][1]\n",
    "    elif state=='fn':\n",
    "        return multilabel_confusion_matrix(y_true,y_pred)[rating-2][1][0]\n",
    "    else:\n",
    "        return multilabel_confusion_matrix(y_true,y_pred)[rating-2][1][1]\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "#defining function for obtaining micro-avg precision\n",
    "def micro_avg_precision(y_true,y_pred):\n",
    "    return ((cm_spec(y_true,y_pred,1,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,2,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,3,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,4,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,5,'tp'))/(\n",
    "                                                cm_spec(y_true,y_pred,1,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,2,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,3,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,4,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,5,'tp')+\n",
    "                                                cm_spec(y_true,y_pred,1,'fp')+\n",
    "                                                 cm_spec(y_true,y_pred,2,'fp')+\n",
    "                                                 cm_spec(y_true,y_pred,3,'fp')+\n",
    "                                                 cm_spec(y_true,y_pred,4,'fp')+\n",
    "                                                 cm_spec(y_true,y_pred,5,'fp')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#defining function for obtaining micro-avg recall\n",
    "def micro_avg_recall(y_true,y_pred):\n",
    "    return ((cm_spec(y_true,y_pred,1,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,2,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,3,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,4,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,5,'tp'))/(\n",
    "                                                cm_spec(y_true,y_pred,1,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,2,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,3,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,4,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,5,'tp')+\n",
    "                                                cm_spec(y_true,y_pred,1,'fn')+\n",
    "                                                 cm_spec(y_true,y_pred,2,'fn')+\n",
    "                                                 cm_spec(y_true,y_pred,3,'fn')+\n",
    "                                                 cm_spec(y_true,y_pred,4,'fn')+\n",
    "                                                 cm_spec(y_true,y_pred,5,'fn')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "#defining function for obtaining micro_avg_f1\n",
    "def micro_avg_f1(y_true,y_pred):\n",
    "    return 2 * ((micro_avg_precision(y_true,y_pred) * micro_avg_recall(y_true,y_pred))/(micro_avg_precision(y_true,y_pred) + micro_avg_recall(y_true,y_pred)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#function to print out confusion matrix breakdown for each rating class\n",
    "def confusion_breakdown(y_true,y_pred,rating):\n",
    "    print(\"True negatives for rating {}: {}\".format(\n",
    "        rating,multilabel_confusion_matrix(y_true,y_pred)[rating-2][0][0]))\n",
    "    print(\"False positives for rating {}: {}\".format(\n",
    "        rating,multilabel_confusion_matrix(y_true,y_pred)[rating-2][0][1]))\n",
    "    print(\"False negatives for rating {}: {}\".format(\n",
    "        rating,multilabel_confusion_matrix(y_true,y_pred)[rating-2][1][0]))\n",
    "    print(\"True positives for rating {}: {}\".format(\n",
    "        rating,multilabel_confusion_matrix(y_true,y_pred)[rating-2][1][1]))\n",
    "    return \"******************************************\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True negatives for rating 2: 33\n",
      "False positives for rating 2: 2\n",
      "False negatives for rating 2: 0\n",
      "True positives for rating 2: 1\n",
      "******************************************\n",
      "True negatives for rating 3: 32\n",
      "False positives for rating 3: 0\n",
      "False negatives for rating 3: 0\n",
      "True positives for rating 3: 4\n",
      "******************************************\n",
      "True negatives for rating 4: 20\n",
      "False positives for rating 4: 0\n",
      "False negatives for rating 4: 1\n",
      "True positives for rating 4: 15\n",
      "******************************************\n",
      "True negatives for rating 5: 21\n",
      "False positives for rating 5: 0\n",
      "False negatives for rating 5: 1\n",
      "True positives for rating 5: 14\n",
      "******************************************\n"
     ]
    }
   ],
   "source": [
    "print(confusion_breakdown(con_collab_2043_tst['actual_ratings'],con_collab_2043_tst['final_rating_predictions_rd'],2))\n",
    "print(confusion_breakdown(con_collab_2043_tst['actual_ratings'],con_collab_2043_tst['final_rating_predictions_rd'],3))\n",
    "print(confusion_breakdown(con_collab_2043_tst['actual_ratings'],con_collab_2043_tst['final_rating_predictions_rd'],4))\n",
    "print(confusion_breakdown(con_collab_2043_tst['actual_ratings'],con_collab_2043_tst['final_rating_predictions_rd'],5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hybrid recommender yielded accuracy:  0.9722222222222222\n"
     ]
    }
   ],
   "source": [
    "print(\"Hybrid recommender yielded accuracy: \", (33+32+20+21+1+4+15+14)/(33+2+1+32+4+20+1+15+21+1+14))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hybrid recommender yielded micro-averaged precision:  0.96\n"
     ]
    }
   ],
   "source": [
    "print(\"Hybrid recommender yielded micro-averaged precision: \", micro_avg_precision(con_collab_2043_tst['actual_ratings'],con_collab_2043_tst['final_rating_predictions_rd']))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hybrid recommender yielded micro-averaged recall:  0.9411764705882353\n"
     ]
    }
   ],
   "source": [
    "print(\"Hybrid recommender yielded micro-averaged recall: \", micro_avg_recall(con_collab_2043_tst['actual_ratings'],con_collab_2043_tst['final_rating_predictions_rd']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hybrid recommender yielded micro_avg_f1 of  0.9504950495049505\n"
     ]
    }
   ],
   "source": [
    "print(\"Hybrid recommender yielded micro_avg_f1 of \", micro_avg_f1(con_collab_2043_tst['actual_ratings'],con_collab_2043_tst['final_rating_predictions_rd']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         2.0       0.33      1.00      0.50         1\n",
      "         3.0       1.00      1.00      1.00         4\n",
      "         4.0       1.00      0.94      0.97        16\n",
      "         5.0       1.00      0.93      0.97        15\n",
      "\n",
      "    accuracy                           0.94        36\n",
      "   macro avg       0.83      0.97      0.86        36\n",
      "weighted avg       0.98      0.94      0.96        36\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#precision, recall, f1 of all rating classes show good performance except for rating 2, which shows quite a strong false positive count...\n",
    "print(classification_report(con_collab_2043_tst['actual_ratings'],con_collab_2043_tst['final_rating_predictions_rd']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hybrid Model Evaluation Result Interpretation\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- The hybrid recommender performed pretty well in terms of predicting the ratings of userid 2043 with an $F_1$ score of 0.95, although the classification report suggests a rather strong false positive count for rating 2... and because there is no rating 1 instance (we could only evaluate on 36 common outlets between content-based and collaborative filtering and the 1 outlet userid 2043 happened to gave a rating of 1.0 to was not included among the 36), the hybrid model could not predict rating 1. This could potentially mean that this hybrid model may wrongly include outlets, that would have been poorly rated by a user, among the top recommendations (false positives for ratings 1 and 2)...\n",
    "\n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- Since we have improvised here by manually converting a continuous predictive output of ALS into a discrete class predictive output so that $F_1$ score could be obtained, it is not possible to use ```.predict_proba()``` or any ```decision_function()``` to provide scorings for the target rating classes (have checked in ALS Spark documentation and there is no mention of such scoring methods in the algorithm so it seems like it is a pure regressor) to eventually obtain the micro-averaged ROC AUC and so we will not use the ROC AUC metric here.\n",
    "    \n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- Accuracy of 0.97, Micro-Averaged precision of 0.96, Micro-Averaged recall of 0.94, Micro-Averaged $F_1$ of 0.95\n",
    "    \n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- Let's now move on to producing some actual recommendations (in the next sub-notebook)!\n",
    "    \n",
    "</ul>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}