Part 9_Simple Hybrid Recommender Trial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 9: Hybrid Recommender Recommendation Trial_userid2552\n",
    "---\n",
    "\n",
    "- Please follow the steps below to download and install all the relevant libraries and dependencies BEFORE running this notebook to avoid encountering any errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- First up, navigate to your home directory and create a new directory called ```server```:\n",
    "    ```cd ~```\n",
    "    ```mkdir server```\n",
    "  Make sure that the stuff below will be downloaded into this ```server``` folder.\n",
    "  \n",
    "- In order to run spark and pyspark on your local machine, kindly ensure that you already have Java installed with the following command in your Terminal or Windows-equivalent in command prompt: ```java -version``` If nothing comes out of this, navigate to this [link](https://java.com/en/download/help/download_options.xml) to download java for mac or windows. You may need to restart your system after installation for java to take effect.\n",
    "\n",
    "- Next, navigate to this [link](https://www.oracle.com/java/technologies/javase-jdk8-downloads.html) to download the java development kit and then install it.\n",
    "\n",
    "- Check if scala is installed by executing this command in your Terminal or Windows-equivalent command prompt: ```scala -version```. If nothing comes out, navigate to this [link](https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz) to download and install scala as well as this [link](https://github.com/sbt/sbt/releases/download/v0.13.17/sbt-0.13.17.tgz) to download and install sbt-0.13.17.tgz.\n",
    "\n",
    "- Navigate to this [link](https://spark.apache.org/downloads.html) to download Apache Spark. Select the options like the screenshot below and click on \"spark-2.4.5-bin-hadoop2.7.tgz\" under point 3 to download spark and install it.\n",
    "<img src=\"yelp_data/spark_dl.png\"/>\n",
    "\n",
    "- The following should be the directory paths of the software you have downloaded and installed above, where ```HOMEDIRECTORY``` is your home directory's name:\n",
    "    JDK: ```/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk```\n",
    "    Sbt: ```/Users/HOMEDIRECTORY/server/sbt```\n",
    "    Scala: ```/Users/HOMEDIRECTORY/server/scala-2.11.12```\n",
    "    Spark: ```/Users/HOMEDIRECTORY/server/spark-2.4.5-bin-hadoop2.7```\n",
    "\n",
    "- After all of the above have been installed, set up a ```.bash_profile``` file in your home directory. For Mac users, if you do not already have a ```.bash_profile``` file, navigate to your home directory and create one by executing the following commands:\n",
    "    ```cd ~```\n",
    "    ```touch .bash_profile```\n",
    "    After which, open it with a text editor of your choice and add the following lines of code at the top of the ```.bash_profile``` file, replacing ```HOMEDIRECTORY``` with the name of your home directory:\n",
    "\n",
    "<img src=\"yelp_data/spark_bash_profile.png\"/>\n",
    "    \n",
    "       \n",
    "- Save and close the ```.bash_profile``` file and execute ```source ~/.bash_profile``` in your Terminal or Windows-equivalent command prompt.\n",
    "\n",
    "- Completely quit your Terminal and command prompt.\n",
    "\n",
    "- Now you may proceed to run the rest of the following code.\n",
    "\n",
    "\n",
    "- ***KINDLY NOTE THAT IF YOU HAVE ENCOUNTERED A CONNECTION REFUSED ERROR OR A JAVA ERROR WHERE IT IS TRYING TO CONNECT TO YOUR IP ADDRESS BUT FAILED WHEN RUNNING ANY PYSPARK-RELATED CELL, KINDLY JUST COPY ALL THE CELLS IN THE NOTEBOOK (HIGHLIGHT THE TOP CELL AND CMD(FOR MAC)/CTRL(FOR WINDOWS) + SHIFT + HIGHLIGHT THE LAST CELL), COPY AND PASTE INTO A FRESH NOTEBOOK AND RUN THEM THERE INSTEAD***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#findspark allows pyspark to be run in jupyter notebook\n",
    "!pip install pyspark\n",
    "!pip install findspark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jasonchia/opt/anaconda3/lib/python3.7/site-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.feature_extraction.stop_words module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.feature_extraction.text. Anything that cannot be imported from sklearn.feature_extraction.text is now part of the private API.\n",
      "  warnings.warn(message, FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "import findspark\n",
    "findspark.init()\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.ml.evaluation import RegressionEvaluator, MulticlassClassificationEvaluator\n",
    "from pyspark.ml.recommendation import ALS\n",
    "from pyspark.ml.feature import StringIndexer, IndexToString\n",
    "from pyspark.ml import Pipeline as PL\n",
    "from pyspark.sql.functions import col\n",
    "from pyspark.ml.feature import StandardScaler as sparkscaler\n",
    "from pyspark.ml.tuning import CrossValidator, ParamGridBuilder\n",
    "from pyspark.sql.functions import max\n",
    "import pyspark.sql.functions as func\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import confusion_matrix, classification_report, multilabel_confusion_matrix, roc_auc_score\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import joblib\n",
    "from sklearn.feature_extraction import stop_words\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating a new spark session\n",
    "newspark = SparkSession.builder.appName('hybrid_rec').getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reading in prepped dataset for model-based collaborative filtering recommendation\n",
    "mbcf = newspark.read.csv('yelp_data/mbcf.csv', header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+-------+-------+\n",
      "|              shops|ratings|userids|\n",
      "+-------------------+-------+-------+\n",
      "|hustle-co-singapore|    5.0|    532|\n",
      "|hustle-co-singapore|    5.0|   1397|\n",
      "|hustle-co-singapore|    5.0|     80|\n",
      "+-------------------+-------+-------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#checking out the first few rows of the mbcf df\n",
    "mbcf.show(3,truncate=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+\n",
      "|max(userids)|\n",
      "+------------+\n",
      "|        2551|\n",
      "+------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#finding out the max userid so that one knows what new userid should be assigned to the new user of interest. I.e. the new user's userid should be max(userid) + 1\n",
    "mbcf.select(max(\"userids\")).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#making a copy of the spark mbcf df for experimentation/trial\n",
    "mbcf_try = mbcf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7076"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out how many rows the spark mbcf df has.\n",
    "mbcf.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['shops', 'ratings', 'userids']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the columns...\n",
    "mbcf_try.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#new user (userid 2552; me)'s arbitrary ratings given to 10 outlets in the list of 987\n",
    "vals = [('tai-chong-coffee-stall-singapore',1.0,2552),\n",
    "        ('the-stamford-brasserie-singapore',1.0,2552),\n",
    "        ('nanyang-old-coffee-singapore-3',2.0,2552),\n",
    "        ('hanis-singapore-2',2.0,2552),\n",
    "        ('dr-cafe-coffee-singapore',3.0,2552),\n",
    "        ('muzium-cafe-singapore',3.0,2552),\n",
    "        ('heavenly-wang-singapore-9',4.0,2552),\n",
    "        ('paris-baguette-singapore-5',4.0,2552),\n",
    "        ('food-for-thought-singapore',5.0,2552),\n",
    "        ('starbucks-singapore-158',5.0,2552),]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- The above cell represent new user (userid 2552, AKA me)'s arbitrary ratings given to 10 outlets in the list of 987 that I have visited (I gave ratings based on my impression of the name of the outlet alone, as I seldom visit coffee-drinking places - fun fact: I don't drink coffee!)(By right, users should give ratings only for those outlets they have actually dined in/patronized; as such, the recommendations generated may not be logical-imagine rubbish in, rubbish out...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pyspark's convention to adding new rows to the end of an existing spark dataframe-1\n",
    "newRows = newspark.createDataFrame(vals,mbcf_try.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pyspark's convention to adding new rows to the end of an existing spark dataframe-2\n",
    "mbcf_try = mbcf_try.union(newRows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7086"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the number of rows of the df upon adding those 10 new rating rows for userid 2552\n",
    "mbcf_try.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7076"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#indeed, the above step of assigning the original mbcf to a new variable makes a copy of that-just want to make sure it works with this novel language-pyspark\n",
    "#original mbcf df has 7076 rows...\n",
    "mbcf.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#converting df to pandas df for easier manipulation later on...\n",
    "mbcf_try_pd = mbcf_try.toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>ratings</th>\n",
       "      <th>userids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7076</th>\n",
       "      <td>tai-chong-coffee-stall-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7077</th>\n",
       "      <td>the-stamford-brasserie-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7078</th>\n",
       "      <td>nanyang-old-coffee-singapore-3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7079</th>\n",
       "      <td>hanis-singapore-2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7080</th>\n",
       "      <td>dr-cafe-coffee-singapore</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7081</th>\n",
       "      <td>muzium-cafe-singapore</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7082</th>\n",
       "      <td>heavenly-wang-singapore-9</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7083</th>\n",
       "      <td>paris-baguette-singapore-5</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7084</th>\n",
       "      <td>food-for-thought-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7085</th>\n",
       "      <td>starbucks-singapore-158</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  ratings  userids\n",
       "7076  tai-chong-coffee-stall-singapore      1.0     2552\n",
       "7077  the-stamford-brasserie-singapore      1.0     2552\n",
       "7078    nanyang-old-coffee-singapore-3      2.0     2552\n",
       "7079                 hanis-singapore-2      2.0     2552\n",
       "7080          dr-cafe-coffee-singapore      3.0     2552\n",
       "7081             muzium-cafe-singapore      3.0     2552\n",
       "7082         heavenly-wang-singapore-9      4.0     2552\n",
       "7083        paris-baguette-singapore-5      4.0     2552\n",
       "7084        food-for-thought-singapore      5.0     2552\n",
       "7085           starbucks-singapore-158      5.0     2552"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#getting a look again at the outlets and ratings provided by userid2552 so we know which outlets to exclude in recommending outlets to userid2552 later on...\n",
    "user_item_2552 = mbcf_try_pd[mbcf_try_pd['userids']==2552]\n",
    "user_item_2552"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+-------+-------+-------------+-----------+\n",
      "|              shops|ratings|userids|userids_index|shops_index|\n",
      "+-------------------+-------+-------+-------------+-----------+\n",
      "|hustle-co-singapore|    5.0|    532|         50.0|      324.0|\n",
      "|hustle-co-singapore|    5.0|   1397|         56.0|      324.0|\n",
      "|hustle-co-singapore|    5.0|     80|       1678.0|      324.0|\n",
      "+-------------------+-------+-------+-------------+-----------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#as part of ALS requirements for the feature columns to be in numerical format, am converting both shops and userids to the double precision format just in case (even though userids is already in a float format)\n",
    "indexer_try = [StringIndexer(inputCol=column, outputCol=column+\"_index\") for column in list(set(mbcf_try.columns)-set(['ratings']))]\n",
    "pipeline_try = PL(stages=indexer_try)\n",
    "transformed_try = pipeline_try.fit(mbcf_try).transform(mbcf_try)\n",
    "transformed_try.show(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "981"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#there are 981 unique shops.\n",
    "transformed_try.select(\"shops\").distinct().count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loading a previously saved and tuned prefitted ALS model to generate predictions here.\n",
    "alsmod = ALS.load(\"yelp_data/als_rec_prefitted.model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#training the dataset containing the new user's ratings...\n",
    "als_model_rec = alsmod.fit(transformed_try)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jasonchia/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:27: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>recommendations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>[(fun-toast-singapore-4, 4.942459583282471), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>[(group-therapy-cafe-singapore, 3.950648307800...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>[(ya-kun-kaya-toast-singapore-22, 3.9292604923...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>[(drips-singapore, 3.9600508213043213), (starb...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>[(the-stamford-brasserie-singapore, 3.74257159...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID                                    recommendations\n",
       "0       0  [(fun-toast-singapore-4, 4.942459583282471), (...\n",
       "1       1  [(group-therapy-cafe-singapore, 3.950648307800...\n",
       "2       2  [(ya-kun-kaya-toast-singapore-22, 3.9292604923...\n",
       "3       3  [(drips-singapore, 3.9600508213043213), (starb...\n",
       "4       4  [(the-stamford-brasserie-singapore, 3.74257159..."
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#making recommendations for model-based collaborative filtering alone first, passing in all 981 outlets so as to ensure as much overlap between collaborative filtering and content-based filtering in the outlets that they generate rating predictions for\n",
    "recs=als_model_rec.recommendForAllUsers(981).toPandas()\n",
    "nrecs=recs.recommendations.apply(pd.Series) \\\n",
    "            .merge(recs, right_index = True, left_index = True) \\\n",
    "            .drop([\"recommendations\"], axis = 1) \\\n",
    "            .melt(id_vars = ['userids_index'], value_name = \"recommendation\") \\\n",
    "            .drop(\"variable\", axis = 1) \\\n",
    "            .dropna() \n",
    "nrecs=nrecs.sort_values('userids_index')\n",
    "nrecs=pd.concat([nrecs['recommendation'].apply(pd.Series), nrecs['userids_index']], axis = 1)\n",
    "nrecs.columns = [\n",
    "        \n",
    "        'Shop_index',\n",
    "        'Rating',\n",
    "        'UserID_index'\n",
    "       \n",
    "     ]\n",
    "md=transformed_try.select(transformed_try['userids'],transformed_try['userids_index'],transformed_try['shops'],transformed_try['shops_index'])\n",
    "md=md.toPandas()\n",
    "dict1=dict(zip(md['userids_index'],md['userids']))\n",
    "dict2=dict(zip(md['shops_index'],md['shops']))\n",
    "nrecs['UserID']=nrecs['UserID_index'].map(dict1)\n",
    "nrecs['shops']=nrecs['Shop_index'].map(dict2)\n",
    "nrecs=nrecs.sort_values('UserID')\n",
    "nrecs.reset_index(drop=True, inplace=True)\n",
    "new=nrecs[['UserID','shops','Rating']]\n",
    "new['recommendations'] = list(zip(new.shops, new.Rating))\n",
    "res=new[['UserID','recommendations']]  \n",
    "res_new=res['recommendations'].groupby([res.UserID]).apply(list).reset_index()\n",
    "res_new.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- The above cell may take some time to load, which may result in slow response in future when this is deployed for A/B testing..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>starbucks-singapore-158</th>\n",
       "      <td>4.761548</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>food-for-thought-singapore</th>\n",
       "      <td>3.971327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>paris-baguette-singapore-5</th>\n",
       "      <td>3.928540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>heavenly-wang-singapore-9</th>\n",
       "      <td>3.873037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>singapore-swimming-club-singapore</th>\n",
       "      <td>3.686428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>starbucks-singapore-160</th>\n",
       "      <td>3.676324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ogopogo-singapore</th>\n",
       "      <td>3.665799</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mahota-commune-singapore</th>\n",
       "      <td>3.664798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>collective-brewers-singapore</th>\n",
       "      <td>3.659645</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gram-singapore-2</th>\n",
       "      <td>3.656865</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          0\n",
       "starbucks-singapore-158            4.761548\n",
       "food-for-thought-singapore         3.971327\n",
       "paris-baguette-singapore-5         3.928540\n",
       "heavenly-wang-singapore-9          3.873037\n",
       "singapore-swimming-club-singapore  3.686428\n",
       "starbucks-singapore-160            3.676324\n",
       "ogopogo-singapore                  3.665799\n",
       "mahota-commune-singapore           3.664798\n",
       "collective-brewers-singapore       3.659645\n",
       "gram-singapore-2                   3.656865"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#creating a new df for userid2552's collaborative filtering-derived recommendations\n",
    "collab_rec_2552 = pd.DataFrame(dict(res_new[res_new[\"UserID\"]==2552]['recommendations'].tolist()[0]),index=[0]).T.sort_values(0,ascending=False)\n",
    "collab_rec_2552.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating a list of outlets userid2552 has rated earlier on\n",
    "rated_2552 = mbcf_try_pd[mbcf_try_pd['userids']==2552]['shops'].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "singapore-swimming-club-singapore    3.686428\n",
       "starbucks-singapore-160              3.676324\n",
       "ogopogo-singapore                    3.665799\n",
       "mahota-commune-singapore             3.664798\n",
       "collective-brewers-singapore         3.659645\n",
       "Name: 0, dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#filtering out those 10 outlets userid2552 has rated initially from the collaborative filtering recommendation list...\n",
    "collab_rankedrecs_2552 = collab_rec_2552.loc[[shop for shop in collab_rec_2552.index if shop not in rated_2552],0]\n",
    "collab_rankedrecs_2552.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>recommendations</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>singapore-swimming-club-singapore</th>\n",
       "      <td>singapore-swimming-club-singapore</td>\n",
       "      <td>3.686428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>starbucks-singapore-160</th>\n",
       "      <td>starbucks-singapore-160</td>\n",
       "      <td>3.676324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ogopogo-singapore</th>\n",
       "      <td>ogopogo-singapore</td>\n",
       "      <td>3.665799</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     recommendations  \\\n",
       "singapore-swimming-club-singapore  singapore-swimming-club-singapore   \n",
       "starbucks-singapore-160                      starbucks-singapore-160   \n",
       "ogopogo-singapore                                  ogopogo-singapore   \n",
       "\n",
       "                                   collab_filter_predicted_ratings  \n",
       "singapore-swimming-club-singapore                         3.686428  \n",
       "starbucks-singapore-160                                   3.676324  \n",
       "ogopogo-singapore                                         3.665799  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#organizing the above series column into a df of recommendations and collaborative filtering rating predictions\n",
    "collab_2552_df = pd.DataFrame({'recommendations':collab_rankedrecs_2552.index,\n",
    "              'collab_filter_predicted_ratings':collab_rankedrecs_2552})\n",
    "collab_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- Now, for content-based filtering aspect..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reading in the previously prepped df meant for content-based filtering here for content-based filtering recommendations..\n",
    "content_f = pd.read_csv('yelp_data/content_based_df_nouser.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>reviews</th>\n",
       "      <th>category_alias</th>\n",
       "      <th>review_count</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183-rojak-singapore</td>\n",
       "      <td>Opening a rojak stall in Toa Payoh isn't that ...</td>\n",
       "      <td>food</td>\n",
       "      <td>2</td>\n",
       "      <td>3.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>Located in the first basement level at MBS, cl...</td>\n",
       "      <td>singaporean coffee foodstands</td>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2am-dessert-bar-singapore</td>\n",
       "      <td>Creative desserts with several layers of flavo...</td>\n",
       "      <td>bars desserts coffee</td>\n",
       "      <td>38</td>\n",
       "      <td>3.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2nd-mini-steamboat-delight-singapore</td>\n",
       "      <td>Fantastic Authentic Place!!!What a treat and a...</td>\n",
       "      <td>cafes</td>\n",
       "      <td>2</td>\n",
       "      <td>4.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>365-fruit-juice-and-smoothie-singapore</td>\n",
       "      <td>Real juice, with real fruits and vegetables, a...</td>\n",
       "      <td>juicebars</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    shops  \\\n",
       "0                     183-rojak-singapore   \n",
       "1     1983-a-taste-of-nanyang-singapore-2   \n",
       "2               2am-dessert-bar-singapore   \n",
       "3    2nd-mini-steamboat-delight-singapore   \n",
       "4  365-fruit-juice-and-smoothie-singapore   \n",
       "\n",
       "                                             reviews  \\\n",
       "0  Opening a rojak stall in Toa Payoh isn't that ...   \n",
       "1  Located in the first basement level at MBS, cl...   \n",
       "2  Creative desserts with several layers of flavo...   \n",
       "3  Fantastic Authentic Place!!!What a treat and a...   \n",
       "4  Real juice, with real fruits and vegetables, a...   \n",
       "\n",
       "                  category_alias  review_count  rating  \n",
       "0                           food             2     3.5  \n",
       "1  singaporean coffee foodstands             4     4.0  \n",
       "2           bars desserts coffee            38     3.5  \n",
       "3                          cafes             2     4.5  \n",
       "4                      juicebars             1     4.0  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out how the first few rows of the df look like.\n",
    "content_f.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(981, 5)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the dimensions of the df...\n",
    "content_f.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#merging userid2552's info with the df meant for content-based filtering so that rcontent-based filtering can make recommendations via rating predictions for userid 2552 later on...\n",
    "content_2552 = pd.merge(content_f,user_item_2552,how='left',on='shops')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>reviews</th>\n",
       "      <th>category_alias</th>\n",
       "      <th>review_count</th>\n",
       "      <th>rating</th>\n",
       "      <th>ratings</th>\n",
       "      <th>userids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183-rojak-singapore</td>\n",
       "      <td>Opening a rojak stall in Toa Payoh isn't that ...</td>\n",
       "      <td>food</td>\n",
       "      <td>2</td>\n",
       "      <td>3.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>Located in the first basement level at MBS, cl...</td>\n",
       "      <td>singaporean coffee foodstands</td>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2am-dessert-bar-singapore</td>\n",
       "      <td>Creative desserts with several layers of flavo...</td>\n",
       "      <td>bars desserts coffee</td>\n",
       "      <td>38</td>\n",
       "      <td>3.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  \\\n",
       "0                  183-rojak-singapore   \n",
       "1  1983-a-taste-of-nanyang-singapore-2   \n",
       "2            2am-dessert-bar-singapore   \n",
       "\n",
       "                                             reviews  \\\n",
       "0  Opening a rojak stall in Toa Payoh isn't that ...   \n",
       "1  Located in the first basement level at MBS, cl...   \n",
       "2  Creative desserts with several layers of flavo...   \n",
       "\n",
       "                  category_alias  review_count  rating  ratings  userids  \n",
       "0                           food             2     3.5      NaN      NaN  \n",
       "1  singaporean coffee foodstands             4     4.0      NaN      NaN  \n",
       "2           bars desserts coffee            38     3.5      NaN      NaN  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the first few rows of which...\n",
    "content_2552.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "#getting dummies for categorical columns...\n",
    "content_2552_wdummies = pd.get_dummies(content_2552, columns=['shops','category_alias'], drop_first=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "#setting feature and target\n",
    "X = content_2552_wdummies.drop(['ratings'], axis=1)\n",
    "y = content_2552_wdummies['ratings']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#collating dummified columns\n",
    "shops_cats_list = [col for col in content_2552_wdummies.columns if (col.startswith('shops')) or (col.startswith('category'))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "#extending with review_count and rating\n",
    "shops_cats_list.extend(['review_count','rating','userids'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#as tfidf can only work on one column of texts at a time, am separating features as below...\n",
    "X1 = X['reviews']\n",
    "X2 = X[shops_cats_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    Opening a rojak stall in Toa Payoh isn't that ...\n",
       "1    Located in the first basement level at MBS, cl...\n",
       "2    Creative desserts with several layers of flavo...\n",
       "Name: reviews, dtype: object"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the first few rows of reviews prior to preprocessing...\n",
    "X1.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Assigning a new variable name to X1 for processing.\n",
    "rev = X1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating customized stop words' list\n",
    "cust_stop_words = [word for word in stop_words.ENGLISH_STOP_WORDS]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "#adding on to the above list based on preliminary word cloud EDA\n",
    "cust_stop_words.extend([\"wa\",\"ha\",\"just\",\"ve\",\"did\",\"got\",\"quite\"]) #adding \"wa\" into the list of stop words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "#preprocessing text in reviews by defining a function to do so\n",
    "lemm = WordNetLemmatizer()\n",
    "\n",
    "def text_processer(raw_text):\n",
    "    # Function to convert a raw string of text to a string of words\n",
    "    # The input is a single string (a raw unprocessed text), and \n",
    "    # the output is a single string (a preprocessed text)\n",
    "    \n",
    "    # 1. Remove http urls.\n",
    "    review_text = re.sub(\"\\(http.+\\)\", \" \", raw_text)\n",
    "    \n",
    "    # 2. Remove non-letters.\n",
    "    letters_only = re.sub(\"[^a-zA-Z]\", \" \", review_text)\n",
    "    \n",
    "    # 3. Convert to lower case, split into individual words.\n",
    "    words = letters_only.lower().split()\n",
    "    \n",
    "    # 4. Lemmatize words.\n",
    "    lemmed_words = [lemm.lemmatize(i) for i in words]\n",
    "    \n",
    "    # 5. Remove stop words.\n",
    "    \n",
    "    meaningful_words = [w for w in lemmed_words if not w in cust_stop_words]\n",
    "    \n",
    "    # 6. Join the words back into one string separated by space, \n",
    "    # and return the result.\n",
    "    return(\" \".join(meaningful_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    opening rojak stall toa payoh isn t easy tough...\n",
       "1    located basement level mb close sw entrance es...\n",
       "2    creative dessert layer flavor enjoyed looking ...\n",
       "dtype: object"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#showing how the processed reviews look like\n",
    "rev_processed = pd.Series([text_processer(text) for text in rev])\n",
    "rev_processed[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(981, 18216)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#using tfidf vectorizer to convert the reviews into term frequency columns...\n",
    "tvec_naive = TfidfVectorizer(stop_words = cust_stop_words)  #instantiating TfidfVectorizer with customized stop words\n",
    "\n",
    "X1_tvec_naive = tvec_naive.fit_transform(rev_processed).todense()   #fitting tvec and transforming the processed reviews\n",
    "X1_tvec_naive_df = pd.DataFrame(X1_tvec_naive, columns = tvec_naive.get_feature_names())  #converting it into a dataframe for easy lookup.\n",
    "X1_tvec_naive_df.shape #checking how array's dimension has changed with the vectorization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(981, 19498)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#combining tvec-df with the rest of the features for rating prediction for userid 2552 later on...\n",
    "X_legit = pd.concat([X1_tvec_naive_df,X2], axis=1)\n",
    "X_legit.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "#adding back the column of ratings so that it can be dropped below-sorry sometimes my train of thought may sound illogical\n",
    "X_legit['ratings'] = y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating X_train manually for userid 2552\n",
    "X_train_2552 = X_legit[X_legit['userids']==2552].drop(['ratings','userids'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating y_train manually for userid 2552\n",
    "y_train_2552 = X_legit[X_legit['userids']==2552]['ratings']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating X_test manually for userid 2552 which contains all outlets that have not been rated by userid 2552\n",
    "X_test_2552 = X_legit[X_legit['userids']!=2552].drop(['ratings','userids'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiate scaler since not all of the features are of the same scale, eg. review_count and rating\n",
    "ss= StandardScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "#fitting the train and transforming both the train and test sets\n",
    "X_train_2552_sc = ss.fit_transform(X_train_2552)\n",
    "X_test_2552_sc = ss.transform(X_test_2552)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "#this is to try if tuning the model every time a user feeds ratings into the recommender would produce a better outcome\n",
    "#params = {\n",
    "#    'max_depth':[5,7,10],\n",
    "#    'min_samples_split':[10, 15, 20],\n",
    "#    'min_samples_leaf':[3, 4, 5],\n",
    "#    'class_weight':['balanced']\n",
    "#    } \n",
    "\n",
    "\n",
    "#dtc_gs = GridSearchCV(DecisionTreeClassifier(), params, cv = 2, verbose = 1, n_jobs = -1)\n",
    "#dtc_gs.fit(X_train_2552_sc, y_train_2552)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print('Gridsearch best score: ', dtc_gs.best_score_)\n",
    "#print('Gridsearch best estimator: ', dtc_gs.best_estimator_)\n",
    "#print('Gridsearch best params: ',dtc_gs.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- Terrible gridsearchcv results of 0.2 accuracy score when user only provides 2 ratings for each rating bracket (total 10 ratings) as that would have forced the number of cross-validation folds to be maximally 2-meaning 50/50 split and testing twice only for each of the hyperparameter combination to be searched upon... This would mean that the user would need to input at least 10 ratings in each rating bracket, which would total to 50 ratings in order to provide a representative picture of all rating brackets (1 - 5), which would be unduly onerous for the user!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loading a previously tuned content-based filtering model for one user (userid 2043) here to see the kind of rating predictions it can generate for a different user, userid 2552...\n",
    "loaded_model = joblib.load('yelp_data/dtc_gs_model.sav')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(ccp_alpha=0.0, class_weight='balanced', criterion='gini',\n",
       "                       max_depth=7, max_features=None, max_leaf_nodes=None,\n",
       "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                       min_samples_leaf=4, min_samples_split=15,\n",
       "                       min_weight_fraction_leaf=0.0, presort='deprecated',\n",
       "                       random_state=None, splitter='best')"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#training the loaded model on the dataset containing the new user, userid 2552's ratings.\n",
    "loaded_model.best_estimator_.fit(X_train_2552_sc, y_train_2552)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "#stacking X_test_2552 as first step in regenerating the shops column for predictions\n",
    "trial = X_test_2552.stack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating loop to re-generate original X_test_2552 order of shops\n",
    "index_lst = []\n",
    "outlets_lst = []\n",
    "for n in range(len(trial.index)):\n",
    "    if trial.index[n][1].startswith('shops_') and trial[n]!=0:\n",
    "        index_lst.append(str(trial.index[n][0]))\n",
    "        outlets_lst.append(trial.index[n][1])\n",
    "index_lst = [int(x) for x in index_lst]\n",
    "reconstructed_X_test_2552 = pd.DataFrame({'shops':outlets_lst}, index=index_lst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "#generating content-based filtering rating predictions for userid 2552\n",
    "rating_predictions = loaded_model.best_estimator_.predict(X_test_2552_sc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "#adding new column of rating predictions into the reconstructed X_test_2552\n",
    "reconstructed_X_test_2552['predicted_ratings']=rating_predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(971, 2)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#giving the reconstructed df a more easily understood name for distinction from the collaborative filtering df dealt with above\n",
    "content_2552_df = reconstructed_X_test_2552\n",
    "content_2552_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(971, 2)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the dimensions of the collaborative filtering df\n",
    "collab_2552_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#ensuring that the content-based filtering df for userid 2552 has no null values\n",
    "sum(content_2552_df.isnull().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#ensuring that the collaborative filtering-based df for userid 2552 has no null values\n",
    "sum(collab_2552_df.isnull().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>shops_183-rojak-singapore</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>shops_1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>shops_2am-dessert-bar-singapore</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>shops_2nd-mini-steamboat-delight-singapore</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>shops_365-fruit-juice-and-smoothie-singapore</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          shops  predicted_ratings\n",
       "0                     shops_183-rojak-singapore                1.0\n",
       "1     shops_1983-a-taste-of-nanyang-singapore-2                1.0\n",
       "2               shops_2am-dessert-bar-singapore                1.0\n",
       "3    shops_2nd-mini-steamboat-delight-singapore                1.0\n",
       "4  shops_365-fruit-juice-and-smoothie-singapore                1.0"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#having a look at some of the ratings predicted by content-based filtering for userid 2552. Oh no, looks like content-based filtering was only able to predict all rating 1s! Let's see if the collaborative filtering can compensate for this shortcoming...\n",
    "content_2552_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "#trimming off the shops' prefixes so that they can eventually be merged with the collaborative filtering df\n",
    "content_2552_df['shops'] = content_2552_df['shops'].apply(lambda x: x[6:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183-rojak-singapore</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2am-dessert-bar-singapore</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  predicted_ratings\n",
       "0                  183-rojak-singapore                1.0\n",
       "1  1983-a-taste-of-nanyang-singapore-2                1.0\n",
       "2            2am-dessert-bar-singapore                1.0"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#shops' prefixes have been trimmed off!\n",
    "content_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "#renaming the column of rating predictions to distinguish from collaborative filtering's prediction column later on when both dfs are merged.\n",
    "content_2552_df.rename(columns={'predicted_ratings':'content_filter_predicted_ratings'},inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>content_filter_predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183-rojak-singapore</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2am-dessert-bar-singapore</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  content_filter_predicted_ratings\n",
       "0                  183-rojak-singapore                               1.0\n",
       "1  1983-a-taste-of-nanyang-singapore-2                               1.0\n",
       "2            2am-dessert-bar-singapore                               1.0"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the renamed column...\n",
    "content_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>recommendations</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>singapore-swimming-club-singapore</th>\n",
       "      <td>singapore-swimming-club-singapore</td>\n",
       "      <td>3.686428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>starbucks-singapore-160</th>\n",
       "      <td>starbucks-singapore-160</td>\n",
       "      <td>3.676324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ogopogo-singapore</th>\n",
       "      <td>ogopogo-singapore</td>\n",
       "      <td>3.665799</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mahota-commune-singapore</th>\n",
       "      <td>mahota-commune-singapore</td>\n",
       "      <td>3.664798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>collective-brewers-singapore</th>\n",
       "      <td>collective-brewers-singapore</td>\n",
       "      <td>3.659645</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     recommendations  \\\n",
       "singapore-swimming-club-singapore  singapore-swimming-club-singapore   \n",
       "starbucks-singapore-160                      starbucks-singapore-160   \n",
       "ogopogo-singapore                                  ogopogo-singapore   \n",
       "mahota-commune-singapore                    mahota-commune-singapore   \n",
       "collective-brewers-singapore            collective-brewers-singapore   \n",
       "\n",
       "                                   collab_filter_predicted_ratings  \n",
       "singapore-swimming-club-singapore                         3.686428  \n",
       "starbucks-singapore-160                                   3.676324  \n",
       "ogopogo-singapore                                         3.665799  \n",
       "mahota-commune-singapore                                  3.664798  \n",
       "collective-brewers-singapore                              3.659645  "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the first few rows of the collaborative filtering df for userid 2552\n",
    "collab_2552_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "#renaming collaborative filtering df's recommendations' column so that it can be merged with the content-based filtering df.\n",
    "collab_2552_df.rename(columns={'recommendations':'shops'},inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>singapore-swimming-club-singapore</th>\n",
       "      <td>singapore-swimming-club-singapore</td>\n",
       "      <td>3.686428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>starbucks-singapore-160</th>\n",
       "      <td>starbucks-singapore-160</td>\n",
       "      <td>3.676324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ogopogo-singapore</th>\n",
       "      <td>ogopogo-singapore</td>\n",
       "      <td>3.665799</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                               shops  \\\n",
       "singapore-swimming-club-singapore  singapore-swimming-club-singapore   \n",
       "starbucks-singapore-160                      starbucks-singapore-160   \n",
       "ogopogo-singapore                                  ogopogo-singapore   \n",
       "\n",
       "                                   collab_filter_predicted_ratings  \n",
       "singapore-swimming-club-singapore                         3.686428  \n",
       "starbucks-singapore-160                                   3.676324  \n",
       "ogopogo-singapore                                         3.665799  "
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the renamed column\n",
    "collab_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reseting the index in the collaborative filtering df so that the index is numerical again\n",
    "collab_2552_df.reset_index(drop=True,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>singapore-swimming-club-singapore</td>\n",
       "      <td>3.686428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>starbucks-singapore-160</td>\n",
       "      <td>3.676324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ogopogo-singapore</td>\n",
       "      <td>3.665799</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               shops  collab_filter_predicted_ratings\n",
       "0  singapore-swimming-club-singapore                         3.686428\n",
       "1            starbucks-singapore-160                         3.676324\n",
       "2                  ogopogo-singapore                         3.665799"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the reset index of the collaborative filtering df\n",
    "collab_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "#merging both content-based filtering and collaborating filtering df to prepare to make hybrid recommendations for userid 2552\n",
    "content_collab_2552_df = pd.merge(content_2552_df,collab_2552_df,how='inner',on='shops')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>content_filter_predicted_ratings</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183-rojak-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.847094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.901927</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2am-dessert-bar-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.623929</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  content_filter_predicted_ratings  \\\n",
       "0                  183-rojak-singapore                               1.0   \n",
       "1  1983-a-taste-of-nanyang-singapore-2                               1.0   \n",
       "2            2am-dessert-bar-singapore                               1.0   \n",
       "\n",
       "   collab_filter_predicted_ratings  \n",
       "0                         2.847094  \n",
       "1                         2.901927  \n",
       "2                         3.623929  "
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#looking at the first few rows of the combined df for userid 2552\n",
    "content_collab_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "#as mentioned in the previous sub-notebook on this hybrid recommender's evaluation, the following are the content-based and collaborative filtering's ratings' weights\n",
    "con_wt = 0.85 / (0.85 + 0.98)\n",
    "collab_wt = 0.98 / (0.85 + 0.98)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "#feature engineering to add hybrid recommender's rating predictions into the combined df by multiplying the respective rating predictions by weights based on both models' f1 scores derived from prior evaluation and summing them up to yield hybrid predictions\n",
    "content_collab_2552_df['final_weighted_rating_predictions'] = (content_collab_2552_df['content_filter_predicted_ratings']*con_wt) + (content_collab_2552_df['collab_filter_predicted_ratings']*collab_wt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>content_filter_predicted_ratings</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "      <th>final_weighted_rating_predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183-rojak-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.847094</td>\n",
       "      <td>1.989154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.901927</td>\n",
       "      <td>2.018518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2am-dessert-bar-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.623929</td>\n",
       "      <td>2.405164</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  content_filter_predicted_ratings  \\\n",
       "0                  183-rojak-singapore                               1.0   \n",
       "1  1983-a-taste-of-nanyang-singapore-2                               1.0   \n",
       "2            2am-dessert-bar-singapore                               1.0   \n",
       "\n",
       "   collab_filter_predicted_ratings  final_weighted_rating_predictions  \n",
       "0                         2.847094                           1.989154  \n",
       "1                         2.901927                           2.018518  \n",
       "2                         3.623929                           2.405164  "
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the first few rows of the final hybrid df\n",
    "content_collab_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>content_filter_predicted_ratings</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "      <th>final_weighted_rating_predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>666</th>\n",
       "      <td>singapore-swimming-club-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.686428</td>\n",
       "      <td>2.438634</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>699</th>\n",
       "      <td>starbucks-singapore-160</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.676324</td>\n",
       "      <td>2.433223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>539</th>\n",
       "      <td>ogopogo-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.665799</td>\n",
       "      <td>2.427587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>459</th>\n",
       "      <td>mahota-commune-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.664798</td>\n",
       "      <td>2.427050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>collective-brewers-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.659645</td>\n",
       "      <td>2.424291</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  content_filter_predicted_ratings  \\\n",
       "666  singapore-swimming-club-singapore                               1.0   \n",
       "699            starbucks-singapore-160                               1.0   \n",
       "539                  ogopogo-singapore                               1.0   \n",
       "459           mahota-commune-singapore                               1.0   \n",
       "176       collective-brewers-singapore                               1.0   \n",
       "\n",
       "     collab_filter_predicted_ratings  final_weighted_rating_predictions  \n",
       "666                         3.686428                           2.438634  \n",
       "699                         3.676324                           2.433223  \n",
       "539                         3.665799                           2.427587  \n",
       "459                         3.664798                           2.427050  \n",
       "176                         3.659645                           2.424291  "
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#top 5 coffee-drinking outlet recommendations for userid 2552 (me!) based on my ratings given rather randomly to 10 of the outlets earlier on...\n",
    "content_collab_2552_df.sort_values('final_weighted_rating_predictions',ascending=False).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "971.0"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#content-filtering could only predict 1.0 for all of the other outlets that userid 2552 (me!) has not rated...\n",
    "content_collab_2552_df['content_filter_predicted_ratings'].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpretation of trial's results\n",
    "---\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- Interestingly enough, content-based filtering failed to predict variations in ratings across all 971 outlets that I have not rated...fortunately, I have the model-based collaborative filtering as a backup to compensate where content-based filtering has failed, in this hybrid recommender...\n",
    "\n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- This hybrid recommender has shown that having the user provide an even distribution of ratings (2 each of ratings 1-5) for 10 different outlets he/she has visited from the getgo out of 980+ outlets (basically training only on about 1% of the data and testing on 99%) is not a viable solution for a recommender that solely relies on a model-inclined content-based filtering system. The model-based collaborative filtering with ALS managed to salvage a little by providing a variation of ratings at least, suggesting that it is indeed robust to missing data-the 971 outlets not rated by userid 2552.\n",
    "\n",
    "</ul>    \n",
    "    \n",
    "<ul>\n",
    "\n",
    "- As mentioned earlier, the 10 initial ratings provided by me were arbitrary and perhaps rather random (not based on actual experience having visited the outlets that I have rated, but purely based instead on my impression of the name of the outlets). As such, it may not be surprising that the user ratings predicted by the content-based filtering algorithm do not make sense. Perhaps I can deploy this proper on a platform for actual coffee lovers to try out and validate it in what is called an A/B testing as part of the future plans of this project.\n",
    "    \n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models' Summary\n",
    "---\n",
    "\n",
    "<ul>\n",
    "\n",
    "- <font size='3'>__Content-based Filtering (baseline accuracy: 0.48)__</font>:\n",
    "    \n",
    "    </ul>\n",
    "\n",
    "|<center><font size='2'>Model<center>|<center><font size='2'>Accuracy<center>|<center><font size='2'>Micro-Average<br>Precision<center>|<center><font size='2'>Micro-Average<br>Recall<center>|<center><font size='2'>Micro-Average<br>$F_1$ score<center>|<center><font size='2'>Micro-Average<br>ROC AUC<center>|<center><font size='2'>Prevalence-Weighted<br>ROC AUC<center>|\n",
    "|---|---|---|---|---|---|---|\n",
    "|<center><font size='1'>*Logistic Regression<br>with<br>TfidfVectorizer*<center>|<center>0.81<center>|<center>0.81<center>|<center>0.81<center>|<center>0.81<center>|<center>0.88<center>|<center>0.71<center>|\n",
    "|<center><font size='1'>*Logistic Regression<br>with<br>TfidfVectorizer<br>with<br>PCA*<center>|<center>0.50<center>|<center>0.50<center>|<center>0.50<center>|<center>0.50<center>|<center>0.65<center>|<center>0.57<center>|\n",
    "|<center><font size='1'>***Decision Tree Classifier<br>with<br>TfidfVectorizer (chosen)***<center>|<center>***0.85***<center>|<center>***0.85***<center>|<center>***0.85***<center>|<center>***0.85***<center>|<center>***0.94***<center>|<center>***0.90***<center>|\n",
    "|<center><font size='1'>*Decision Tree Classifier<br>with<br>TfidfVectorizer<br>with<br>PCA*<center>|<center>0.43<center>|<center>0.43<center>|<center>0.43<center>|<center>0.43<center>|<center>0.62<center>|<center>0.47<center>|\n",
    "|<center><font size='1'>*Random Forest Classifier<br>with<br>TfidfVectorizer*<center>|<center>0.61<center>|<center>0.61<center>|<center>0.61<center>|<center>0.61<center>|<center>0.92<center>|<center>0.81<center>|\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- <font size='3'>__Model-based Collaborative Filtering (baseline accuracy: 0.47)__</font>:\n",
    "    \n",
    "    </ul>\n",
    "\n",
    "|<center><font size='2'>Model<center>|<center><font size='2'>Accuracy<center>|<center><font size='2'>Micro-Average<br>Precision<center>|<center><font size='2'>Micro-Average<br>Recall<center>|<center><font size='2'>Micro-Average<br>$F_1$ score<center>|\n",
    "|---|---|---|---|---|\n",
    "|<center><font size='1'>*Alternating Least Squares (ALS)*<center>|<center>0.97<center>|<center>0.97<center>|<center>0.97<center>|<center>0.97<center>|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- <font size='3'>__Hybrid Recommender (baseline accuracy: 0.48)__</font>:\n",
    "    \n",
    "    </ul>\n",
    "\n",
    "|<center><font size='2'>Model<center>|<center><font size='2'>Accuracy<center>|<center><font size='2'>Micro-Average<br>Precision<center>|<center><font size='2'>Micro-Average<br>Recall<center>|<center><font size='2'>Micro-Average<br>$F_1$ score<center>|\n",
    "|---|---|---|---|---|\n",
    "|<center><font size='1'>*Hybrid Recommender<br>(ALS and DecisionTreeClassifier)*<center>|<center>0.97<center>|<center>0.96<center>|<center>0.94<center>|<center>0.95<center>|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Limitations and Potential Improvements\n",
    "---\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- One major limitation is that the content-based filtering was only trained on a single user's ratings (even though that particular user rated almost every outlet scraped from Yelp) and so it may not be representative of the opinion of the general community of people who visit local coffee-drinking outlets for coffee. As such, the bias that results could have skewed the evaluation outcome (exaggerating content-based filtering model performance-looking overly rosy with a high micro-averaged $F_1$ score of 0.85 and yet performing so poorly for the trial). Some ways to improve this could be to\n",
    "    \n",
    "    <ul>\n",
    "        <li>Train a content-based filtering model for each user or</li>\n",
    "    </ul>\n",
    "    \n",
    "    <ul>\n",
    "        <li>Include cases where the same user has rated the same outlet more than once and train one content-based model for users who have at least rated an outlet twice or rated 2 different outlets at least to eliminate the bias from just training on one avid user (but this would mean that the final recommendation outcome could have the same outlet showing up more than once among the top 5 or top 10 recommendations since there could be different ratings given by the same user to the same outlet on different occasions-eg. ratings given by same user to same outlet before-after renovation, as business grows, before-after a change in style due to change in management etc. but these time-dependent factors were not accounted for in this project) or</li>\n",
    "    </ul>\n",
    "    \n",
    "    <ul>\n",
    "        <li>Instead of just tuning the content-based filtering model on a specific userid 2043 who happened to have rated a large majority of outlets, include the other userids after filtering off the rows with duplicate userids and outlets and check if any error emerges from stratifying by userid instead of ratings at the ```train_test_)split``` stage. If there are no errors raised, one may carry on with the rest of the steps detailed in this capstone project to build a content-based filtering model based on a series of users. However, unlike the [Movielens](https://towardsdatascience.com/prototyping-a-recommender-system-step-by-step-part-1-knn-item-based-collaborative-filtering-637969614ea) case where each user has provided a considerable number of ratings and reviews for a large number of movies, quite a lot of users from this scraped Yelp dataset provided very few number of ratings-we are looking at a considerable number of users who only provided within a range of 1 to 5 ratings for each unique outlet per unique user (the scraped Yelp dataset has less than 10,000 reviews and ratings). Probably with more data or a more extensive dataset that is established enough for recommenders to be built upon it, the concepts applied in this project could result in the development of a hybrid recommender that performs better!</li>\n",
    "    </ul>\n",
    "    \n",
    "    <ul>\n",
    "        <li>Tune hyperparameters for a selected estimator on each new user's 10 ratings' worth of data but this might take a considerable amount of time for each usage of the hybrid recommender, and since the number of cross-validation folds cannot be more than the number of samples in each rating class, this would require the user to provide a substantial number of ratings for each rating bracket, which would take up users' time and frustrate them...</li>\n",
    "    </ul>\n",
    "\n",
    "</ul>    \n",
    "\n",
    "<ul>\n",
    "    \n",
    "- In order to evaluate ALS' rating predictions on the same grounds as the content-based filtering that I have evaluated with classification metrics (since the ratings fall into finite discrete classes), ALS' continuous rating predictions were rounded off to nearest whole numbers and those falling into incorrect rating classes such as -1.0 or 0.0 were manually coded as the nearest possible value-1.0. Even though these incorrect classes were already mis-classified and correcting them did not alter the spark evaluator-computed $F_1$ score (still 0.98 after corrections), it should be noted that the ALS predictions were \"tweaked\". There should be better, more reliable options to convert the continuous output into discrete rating classes but for now, I am making do with it and qualifying this as an \"assumption\" by which the results of my evaluation of the ALS component holds true. One way of improving this is to consider incorporating a logistic/sigmoid function ($f(x)$) into the matrix factorization loss function to automatically squeeze the range of predicted output ratings down to probabilities that range between 0 and 1 for each discrete rating class: \n",
    "\n",
    "</ul>\n",
    "\n",
    "<img src=\"yelp_data/extn_matfac.png\"/>\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- The above extension of matrix factorization can then be adapted and extended to more complex algorithms like neural networks, which are used for near state-of-the-art recommenders.\n",
    "    \n",
    "</ul>\n",
    "\n",
    "<ul>    \n",
    "    \n",
    "- Another limitation is that this hybrid model only takes into account explicit users' preferences in the form of ratings and reviews. Perhaps implicit feedback could be incorporated such as clickthrough data, or page views (which are not easily obtainable, if not impossible to obtain, just from the Yelp API token and the scraping of its websites via BeautifulSoup) to supplement the hybrid recommender.\n",
    "    \n",
    "</ul>    \n",
    "    \n",
    "<ul>\n",
    "\n",
    "- Possibly yet another limitation could be that ```TfidfVectorizer``` was not tuned and naive vectorizer was fitted instead as reviews were not the only input feature but was combined with numerical feature columns for the content-based filtering. As such, it was difficult, if not impossible, to tune the vetorizer for example, limiting the ```max_features``` hyperparameter. One possibility of improving this though, could be to use an [Auto-encoder](https://towardsdatascience.com/creating-a-hybrid-content-collaborative-movie-recommender-using-deep-learning-cc8b431618af) which leverages on a neural network to decompose the multi-dimensional tfidf matrix down to its crucial components.\n",
    "\n",
    "</ul>        \n",
    "\n",
    "<ul>\n",
    "    \n",
    "- Consider mean normalization and NaN or null ratings where instead of eliminating NaNs or null ratings (user had not rated outlet), mean normalize them such that NaNs are converted into zero first and then eventually add the average outlet rating to these zeros to yield the predicted ratings as in [here](https://www.coursera.org/lecture/machine-learning/implementational-detail-mean-normalization-Adk8G) for those previously null rating values. This could add value by generating rating predictions for users who opt not to rate any outlets at all when using this hybrid recommender system, and therefore partially address the cold-start problem common to collaborative filtering systems.\n",
    "    \n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- The scraped data may be incomplete or its quality may not be optimal as the structuring of the html source code behind Yelp's webpage is such that it is hard to ensure that all userids are aligned to all ratings - there may be some pages with an extra rating and review with no userid attached as that was an old review by the userid who posted an updated post right above the old one (it is difficult to write a code with BeautifulSoup to distinguish that and scrape the relevant details accordingly). Fortunately, this scenario is not extremely common on Yelp. Perhaps increased data quality and more of it will go a long way in improving this hybrid recommender system.\n",
    "    \n",
    "    </ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions and Future Plans\n",
    "---\n",
    "\n",
    "<ul>\n",
    "  \n",
    "- If there is more time allowance, will allow XGBClassifier and GradientBoostClassifier to tune to completion for Content-based Filtering since both of their ```cross_val_score``` were at least 0.80 (compared to DecisionTreeClassifier's 0.79) which implies that they just might outperform DecisionTreeClassifier after being tuned with XGBClassifier emerging top!\n",
    "\n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- There is a potential that if more time is provided to tune certain algorithms like DecisionTreeClassifier with tfidf and PCA as well as RandomForestClassifier with tfidf etc, they might be able to perform better than the chosen DecisionTreeClassifier with only tfidf since for some of these models, their tuned hyperparameters fall on either edge of the search space provided for the hyperparameters in the parameter grid - this suggests that if tuned a couple more times, providing different hyperparameter values beyond the previous search grid each time, a more optimal model could be achieved eventually for each of these algorithms that had not been chosen. Furthermore, ```PCA``` was not tuned as well - this could help reduce the dimensionality of the data and reduce model variance while maintaining all features which is crucial since all the outlets are coded as features and they should not be removed so that there is at least a predicted rating for all outlets - we do not want to miss out on possible outlets the user could be interested in from the list of recommendations (beyond the top 5 or 10). Perhaps the ```PCA``` should be fitted onto just ```X_train``` instead of ```X_train_sc``` as I had casually experimented on the former near the end of the capstone period with some multiclass ROC AUC plots and scores and it churned out better micro-average ROC AUCs and weighted-by-prevalence ROC AUCs. That said, there has been [advice online](https://stackoverflow.com/questions/39685740/calculate-sklearn-roc-auc-score-for-multi-class) that multiclass classification should better be assessed by its usual methods-confusion matrix-instead...\n",
    "\n",
    "</ul>\n",
    "\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- Plan on incorporating the above code into a ```.py``` script which will be a part of the service script of a Flask engine for deployment where the user can scroll through a list of the 987 coffee-drinking outlets and look at some of the associated info such as category_alias, review_count, average outlet rating etc and rate at least 10 of the outlets by providing at least 2 ratings from each rating bracket (1 to 5). These user input will then be added to the database which will then undergo the above code and churn out the top 5 outlet recommendations to drink coffee as output back to the user.\n",
    "\n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "\n",
    "- If the above is possible, might look into deploying it on Heroku too.\n",
    "\n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "\n",
    "- The above deployment plans could serve to put this hybrid recommender out there to see if it can stand up to public scrutiny. As mentioned under interpretations above, this could constitute a form of A/B testing to really gauge the recommender's performance with actual users' opinion/sentiments.\n",
    "\n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "\n",
    "- All in all, this capstone project is \"bootstrapp-ish\" as almost every component was cobbled together from its 'raw materials': From the data-scraping phase to constructing the content-based filtering model and combining it with a more-or-less well-established collaborative filtering model, and then aggregating rating predictions from both to yield the final hybrid prototype. There is no tried-and-tested method, gold-standard or novice-friendly digestible guide for constructing a full hybrid recommendation system out there. Although this hybrid system cannot be compared to commercial ones like Netflix, it serves as a peek into the nuts and bolts that go into creating a simple hybrid recommender from scratch for beginners. \n",
    "\n",
    "</ul>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sources\n",
    "---\n",
    "\n",
    "- https://java.com/en/download/help/download_options.xml\n",
    "- https://www.oracle.com/java/technologies/javase-jdk8-downloads.html\n",
    "- https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz\n",
    "- https://github.com/sbt/sbt/releases/download/v0.13.17/sbt-0.13.17.tgz\n",
    "- https://spark.apache.org/downloads.html\n",
    "- https://medium.com/luckspark/installing-spark-2-3-0-on-macos-high-sierra-276a127b8b85\n",
    "- https://towardsdatascience.com/creating-a-hybrid-content-collaborative-movie-recommender-using-deep-learning-cc8b431618af\n",
    "- https://towardsdatascience.com/prototyping-a-recommender-system-step-by-step-part-1-knn-item-based-collaborative-filtering-637969614ea\n",
    "- https://www.coursera.org/lecture/machine-learning/implementational-detail-mean-normalization-Adk8G\n",
    "- https://stackoverflow.com/questions/39685740/calculate-sklearn-roc-auc-score-for-multi-class\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}