Simple Hybrid Recommender Trial_extn with XGB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 9: Hybrid Recommender Recommendation Trial_userid2552\n",
    "---\n",
    "\n",
    "- Please follow the steps below to download and install all the relevant libraries and dependencies BEFORE running this notebook to avoid encountering any errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- First up, navigate to your home directory and create a new directory called ```server```:\n",
    "    ```cd ~```\n",
    "    ```mkdir server```\n",
    "  Make sure that the stuff below will be downloaded into this ```server``` folder.\n",
    "  \n",
    "- In order to run spark and pyspark on your local machine, kindly ensure that you already have Java installed with the following command in your Terminal or Windows-equivalent in command prompt: ```java -version``` If nothing comes out of this, navigate to this [link](https://java.com/en/download/help/download_options.xml) to download java for mac or windows. You may need to restart your system after installation for java to take effect.\n",
    "\n",
    "- Next, navigate to this [link](https://www.oracle.com/java/technologies/javase-jdk8-downloads.html) to download the java development kit and then install it.\n",
    "\n",
    "- Check if scala is installed by executing this command in your Terminal or Windows-equivalent command prompt: ```scala -version```. If nothing comes out, navigate to this [link](https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz) to download and install scala as well as this [link](https://github.com/sbt/sbt/releases/download/v0.13.17/sbt-0.13.17.tgz) to download and install sbt-0.13.17.tgz.\n",
    "\n",
    "- Navigate to this [link](https://spark.apache.org/downloads.html) to download Apache Spark. Select the options like the screenshot below and click on \"spark-2.4.5-bin-hadoop2.7.tgz\" under point 3 to download spark and install it.\n",
    "<img src=\"yelp_data/spark_dl.png\"/>\n",
    "\n",
    "- The following should be the directory paths of the software you have downloaded and installed above, where ```HOMEDIRECTORY``` is your home directory's name:\n",
    "    JDK: ```/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk```\n",
    "    Sbt: ```/Users/HOMEDIRECTORY/server/sbt```\n",
    "    Scala: ```/Users/HOMEDIRECTORY/server/scala-2.11.12```\n",
    "    Spark: ```/Users/HOMEDIRECTORY/server/spark-2.4.5-bin-hadoop2.7```\n",
    "\n",
    "- After all of the above have been installed, set up a ```.bash_profile``` file in your home directory. For Mac users, if you do not already have a ```.bash_profile``` file, navigate to your home directory and create one by executing the following commands:\n",
    "    ```cd ~```\n",
    "    ```touch .bash_profile```\n",
    "    After which, open it with a text editor of your choice and add the following lines of code at the top of the ```.bash_profile``` file, replacing ```HOMEDIRECTORY``` with the name of your home directory:\n",
    "\n",
    "<img src=\"yelp_data/spark_bash_profile.png\"/>\n",
    "    \n",
    "       \n",
    "- Save and close the ```.bash_profile``` file and execute ```source ~/.bash_profile``` in your Terminal or Windows-equivalent command prompt.\n",
    "\n",
    "- Completely quit your Terminal and command prompt.\n",
    "\n",
    "- Now you may proceed to run the rest of the following code.\n",
    "\n",
    "\n",
    "- ***KINDLY NOTE THAT IF YOU HAVE ENCOUNTERED A CONNECTION REFUSED ERROR OR A JAVA ERROR WHERE IT IS TRYING TO CONNECT TO YOUR IP ADDRESS BUT FAILED WHEN RUNNING ANY PYSPARK-RELATED CELL, KINDLY JUST COPY ALL THE CELLS IN THE NOTEBOOK (HIGHLIGHT THE TOP CELL AND CMD(FOR MAC)/CTRL(FOR WINDOWS) + SHIFT + HIGHLIGHT THE LAST CELL), COPY AND PASTE INTO A FRESH NOTEBOOK AND RUN THEM THERE INSTEAD***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#this is required to allow pyspark to run in a jupyter notebook\n",
    "#!pip install findspark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jasonchia/opt/anaconda3/lib/python3.7/site-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.feature_extraction.stop_words module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.feature_extraction.text. Anything that cannot be imported from sklearn.feature_extraction.text is now part of the private API.\n",
      "  warnings.warn(message, FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "import findspark\n",
    "findspark.init()\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.ml.recommendation import ALS\n",
    "from pyspark.ml.feature import StringIndexer, IndexToString\n",
    "from pyspark.ml import Pipeline as PL\n",
    "from pyspark.sql.functions import col\n",
    "\n",
    "from pyspark.ml.tuning import CrossValidator, ParamGridBuilder\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "from xgboost import XGBClassifier\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.feature_extraction import stop_words\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating a new spark session\n",
    "newspark = SparkSession.builder.appName('hybrid_rec').getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reading in prepped dataset for model-based collaborative filtering recommendation\n",
    "mbcf = newspark.read.csv('yelp_data/mbcf.csv', header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+-------+-------+\n",
      "|              shops|ratings|userids|\n",
      "+-------------------+-------+-------+\n",
      "|hustle-co-singapore|    5.0|    532|\n",
      "|hustle-co-singapore|    5.0|   1397|\n",
      "|hustle-co-singapore|    5.0|     80|\n",
      "+-------------------+-------+-------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#checking out the first few rows of the mbcf df\n",
    "mbcf.show(3,truncate=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#finding out the max userid so that one knows what new userid should be assigned to the new user of interest. I.e. the new user's userid should be max(userid) + 1\n",
    "#mbcf.select(max(\"userids\")).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#making a copy of the spark mbcf df for experimentation/trial\n",
    "mbcf_try = mbcf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7076"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out how many rows the spark mbcf df has.\n",
    "mbcf.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['shops', 'ratings', 'userids']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the columns...\n",
    "mbcf_try.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#new user (userid 2552; me)'s arbitrary ratings given to 10 outlets in the list of 987\n",
    "vals = [('tai-chong-coffee-stall-singapore',1.0,2552),\n",
    "        ('the-stamford-brasserie-singapore',1.0,2552),\n",
    "        ('nanyang-old-coffee-singapore-3',2.0,2552),\n",
    "        ('hanis-singapore-2',2.0,2552),\n",
    "        ('dr-cafe-coffee-singapore',3.0,2552),\n",
    "        ('muzium-cafe-singapore',3.0,2552),\n",
    "        ('heavenly-wang-singapore-9',4.0,2552),\n",
    "        ('paris-baguette-singapore-5',4.0,2552),\n",
    "        ('food-for-thought-singapore',5.0,2552),\n",
    "        ('starbucks-singapore-158',5.0,2552),]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- The above cell represent new user (userid 2552, AKA me)'s arbitrary ratings given to 10 outlets in the list of 987 that I have visited (I gave ratings based on my impression of the name of the outlet alone, as I seldom visit coffee-drinking places - fun fact: I don't drink coffee!)(By right, users should give ratings only for those outlets they have actually dined in/patronized; as such, the recommendations generated may not be logical-imagine rubbish in, rubbish out...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pyspark's convention to adding new rows to the end of an existing spark dataframe-1\n",
    "newRows = newspark.createDataFrame(vals,mbcf_try.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pyspark's convention to adding new rows to the end of an existing spark dataframe-2\n",
    "mbcf_try = mbcf_try.union(newRows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7086"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the number of rows of the df upon adding those 10 new rating rows for userid 2552\n",
    "mbcf_try.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7076"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#indeed, the above step of assigning the original mbcf to a new variable makes a copy of that-just want to make sure it works with this novel language-pyspark\n",
    "#original mbcf df has 7076 rows...\n",
    "mbcf.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#converting df to pandas df for easier manipulation later on...\n",
    "mbcf_try_pd = mbcf_try.toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>ratings</th>\n",
       "      <th>userids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7076</th>\n",
       "      <td>tai-chong-coffee-stall-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7077</th>\n",
       "      <td>the-stamford-brasserie-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7078</th>\n",
       "      <td>nanyang-old-coffee-singapore-3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7079</th>\n",
       "      <td>hanis-singapore-2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7080</th>\n",
       "      <td>dr-cafe-coffee-singapore</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7081</th>\n",
       "      <td>muzium-cafe-singapore</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7082</th>\n",
       "      <td>heavenly-wang-singapore-9</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7083</th>\n",
       "      <td>paris-baguette-singapore-5</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7084</th>\n",
       "      <td>food-for-thought-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7085</th>\n",
       "      <td>starbucks-singapore-158</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2552</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  ratings  userids\n",
       "7076  tai-chong-coffee-stall-singapore      1.0     2552\n",
       "7077  the-stamford-brasserie-singapore      1.0     2552\n",
       "7078    nanyang-old-coffee-singapore-3      2.0     2552\n",
       "7079                 hanis-singapore-2      2.0     2552\n",
       "7080          dr-cafe-coffee-singapore      3.0     2552\n",
       "7081             muzium-cafe-singapore      3.0     2552\n",
       "7082         heavenly-wang-singapore-9      4.0     2552\n",
       "7083        paris-baguette-singapore-5      4.0     2552\n",
       "7084        food-for-thought-singapore      5.0     2552\n",
       "7085           starbucks-singapore-158      5.0     2552"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#getting a look again at the outlets and ratings provided by userid2552 so we know which outlets to exclude in recommending outlets to userid2552 later on...\n",
    "user_item_2552 = mbcf_try_pd[mbcf_try_pd['userids']==2552]\n",
    "user_item_2552"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+-------+-------+-------------+-----------+\n",
      "|              shops|ratings|userids|userids_index|shops_index|\n",
      "+-------------------+-------+-------+-------------+-----------+\n",
      "|hustle-co-singapore|    5.0|    532|         50.0|      324.0|\n",
      "|hustle-co-singapore|    5.0|   1397|         56.0|      324.0|\n",
      "|hustle-co-singapore|    5.0|     80|       1678.0|      324.0|\n",
      "+-------------------+-------+-------+-------------+-----------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#as part of ALS requirements for the feature columns to be in numerical format, am converting both shops and userids to the double precision format just in case (even though userids is already in a float format)\n",
    "indexer_try = [StringIndexer(inputCol=column, outputCol=column+\"_index\") for column in list(set(mbcf_try.columns)-set(['ratings']))]\n",
    "pipeline_try = PL(stages=indexer_try)\n",
    "transformed_try = pipeline_try.fit(mbcf_try).transform(mbcf_try)\n",
    "transformed_try.show(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-------+-------+-------------+-----------+\n",
      "|               shops|ratings|userids|userids_index|shops_index|\n",
      "+--------------------+-------+-------+-------------+-----------+\n",
      "|tai-chong-coffee-...|    1.0|   2552|        100.0|      525.0|\n",
      "|the-stamford-bras...|    1.0|   2552|        100.0|      348.0|\n",
      "|nanyang-old-coffe...|    2.0|   2552|        100.0|      369.0|\n",
      "|   hanis-singapore-2|    2.0|   2552|        100.0|      434.0|\n",
      "|dr-cafe-coffee-si...|    3.0|   2552|        100.0|      408.0|\n",
      "|muzium-cafe-singa...|    3.0|   2552|        100.0|      508.0|\n",
      "|heavenly-wang-sin...|    4.0|   2552|        100.0|      305.0|\n",
      "|paris-baguette-si...|    4.0|   2552|        100.0|      521.0|\n",
      "|food-for-thought-...|    5.0|   2552|        100.0|       26.0|\n",
      "|starbucks-singapo...|    5.0|   2552|        100.0|      479.0|\n",
      "+--------------------+-------+-------+-------------+-----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "transformed_try.filter(\"userids = 2552\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformed_2552 = transformed_try.filter(\"userids = 2552\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "981"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#there are 981 unique shops.\n",
    "transformed_try.select(\"shops\").distinct().count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#rank=300 and regParam=0.1 was a pair of tuned best params while retuning als with train test split stratified for userids...\n",
    "als = ALS(rank=300, regParam=0.1, maxIter=20, seed=42, userCol='userids_index',itemCol='shops_index', ratingCol='ratings',coldStartStrategy='drop')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loading a previously saved and tuned prefitted ALS model to generate predictions here.\n",
    "#alsmod = ALS.load(\"yelp_data/als_rec_prefitted.model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#training the dataset containing the new user's ratings...\n",
    "als_model_rec = als.fit(transformed_try)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jasonchia/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:27: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>recommendations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>[(apartment-coffee-singapore, 4.94189643859863...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>[(drips-singapore, 3.950673818588257), (butter...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>[(ya-kun-kaya-toast-singapore-22, 3.9299848079...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>[(40-hands-singapore, 3.96014666557312), (star...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>[(the-stamford-brasserie-singapore, 3.73784589...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID                                    recommendations\n",
       "0       0  [(apartment-coffee-singapore, 4.94189643859863...\n",
       "1       1  [(drips-singapore, 3.950673818588257), (butter...\n",
       "2       2  [(ya-kun-kaya-toast-singapore-22, 3.9299848079...\n",
       "3       3  [(40-hands-singapore, 3.96014666557312), (star...\n",
       "4       4  [(the-stamford-brasserie-singapore, 3.73784589..."
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#making recommendations for model-based collaborative filtering alone first, passing in all 981 outlets so as to ensure as much overlap between collaborative filtering and content-based filtering in the outlets that they generate rating predictions for\n",
    "recs=als_model_rec.recommendForAllUsers(981).toPandas()\n",
    "nrecs=recs.recommendations.apply(pd.Series) \\\n",
    "            .merge(recs, right_index = True, left_index = True) \\\n",
    "            .drop([\"recommendations\"], axis = 1) \\\n",
    "            .melt(id_vars = ['userids_index'], value_name = \"recommendation\") \\\n",
    "            .drop(\"variable\", axis = 1) \\\n",
    "            .dropna() \n",
    "nrecs=nrecs.sort_values('userids_index')\n",
    "nrecs=pd.concat([nrecs['recommendation'].apply(pd.Series), nrecs['userids_index']], axis = 1)\n",
    "nrecs.columns = [\n",
    "        \n",
    "        'Shop_index',\n",
    "        'Rating',\n",
    "        'UserID_index'\n",
    "       \n",
    "     ]\n",
    "md=transformed_try.select(transformed_try['userids'],transformed_try['userids_index'],transformed_try['shops'],transformed_try['shops_index'])\n",
    "md=md.toPandas()\n",
    "dict1=dict(zip(md['userids_index'],md['userids']))\n",
    "dict2=dict(zip(md['shops_index'],md['shops']))\n",
    "nrecs['UserID']=nrecs['UserID_index'].map(dict1)\n",
    "nrecs['shops']=nrecs['Shop_index'].map(dict2)\n",
    "nrecs=nrecs.sort_values('UserID')\n",
    "nrecs.reset_index(drop=True, inplace=True)\n",
    "new=nrecs[['UserID','shops','Rating']]\n",
    "new['recommendations'] = list(zip(new.shops, new.Rating))\n",
    "res=new[['UserID','recommendations']]  \n",
    "res_new=res['recommendations'].groupby([res.UserID]).apply(list).reset_index()\n",
    "res_new.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- The above cell may take some time to load, which may result in slow response in future when this is deployed for A/B testing..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>starbucks-singapore-158</th>\n",
       "      <td>4.760870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>food-for-thought-singapore</th>\n",
       "      <td>3.974101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>paris-baguette-singapore-5</th>\n",
       "      <td>3.927828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>heavenly-wang-singapore-9</th>\n",
       "      <td>3.872818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>singapore-swimming-club-singapore</th>\n",
       "      <td>3.699315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>starbucks-singapore-160</th>\n",
       "      <td>3.688314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ogopogo-singapore</th>\n",
       "      <td>3.678975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mahota-commune-singapore</th>\n",
       "      <td>3.676981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>collective-brewers-singapore</th>\n",
       "      <td>3.671374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gram-singapore-2</th>\n",
       "      <td>3.669931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>reload-singapore</th>\n",
       "      <td>3.656636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>empire-cafe-singapore</th>\n",
       "      <td>3.656636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chin-chin-restaurant-singapore</th>\n",
       "      <td>3.655279</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>little-creatures-brewing-singapore-singapore</th>\n",
       "      <td>3.654633</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>l-atelier-tiramisu-singapore</th>\n",
       "      <td>3.653682</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>the-cup-rice-and-noodle-singapore</th>\n",
       "      <td>3.653046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fine-palate-cafe-singapore</th>\n",
       "      <td>3.652950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>caffe-beviamo-tanglin-mall-singapore</th>\n",
       "      <td>3.652204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rich-and-good-cake-shop-singapore</th>\n",
       "      <td>3.651603</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>swirls-bake-shop-singapore</th>\n",
       "      <td>3.651515</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     0\n",
       "starbucks-singapore-158                       4.760870\n",
       "food-for-thought-singapore                    3.974101\n",
       "paris-baguette-singapore-5                    3.927828\n",
       "heavenly-wang-singapore-9                     3.872818\n",
       "singapore-swimming-club-singapore             3.699315\n",
       "starbucks-singapore-160                       3.688314\n",
       "ogopogo-singapore                             3.678975\n",
       "mahota-commune-singapore                      3.676981\n",
       "collective-brewers-singapore                  3.671374\n",
       "gram-singapore-2                              3.669931\n",
       "reload-singapore                              3.656636\n",
       "empire-cafe-singapore                         3.656636\n",
       "chin-chin-restaurant-singapore                3.655279\n",
       "little-creatures-brewing-singapore-singapore  3.654633\n",
       "l-atelier-tiramisu-singapore                  3.653682\n",
       "the-cup-rice-and-noodle-singapore             3.653046\n",
       "fine-palate-cafe-singapore                    3.652950\n",
       "caffe-beviamo-tanglin-mall-singapore          3.652204\n",
       "rich-and-good-cake-shop-singapore             3.651603\n",
       "swirls-bake-shop-singapore                    3.651515"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#creating a new df for userid2552's collaborative filtering-derived recommendations\n",
    "collab_rec_2552 = pd.DataFrame(dict(res_new[res_new[\"UserID\"]==2552]['recommendations'].tolist()[0]),index=[0]).T.sort_values(0,ascending=False)\n",
    "collab_rec_2552.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating a list of outlets userid2552 has rated earlier on\n",
    "rated_2552 = mbcf_try_pd[mbcf_try_pd['userids']==2552]['shops'].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "singapore-swimming-club-singapore    3.699315\n",
       "starbucks-singapore-160              3.688314\n",
       "ogopogo-singapore                    3.678975\n",
       "mahota-commune-singapore             3.676981\n",
       "collective-brewers-singapore         3.671374\n",
       "Name: 0, dtype: float64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#filtering out those 10 outlets userid2552 has rated initially from the collaborative filtering recommendation list...\n",
    "collab_rankedrecs_2552 = collab_rec_2552.loc[[shop for shop in collab_rec_2552.index if shop not in rated_2552],0]\n",
    "collab_rankedrecs_2552.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>recommendations</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>singapore-swimming-club-singapore</th>\n",
       "      <td>singapore-swimming-club-singapore</td>\n",
       "      <td>3.699315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>starbucks-singapore-160</th>\n",
       "      <td>starbucks-singapore-160</td>\n",
       "      <td>3.688314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ogopogo-singapore</th>\n",
       "      <td>ogopogo-singapore</td>\n",
       "      <td>3.678975</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     recommendations  \\\n",
       "singapore-swimming-club-singapore  singapore-swimming-club-singapore   \n",
       "starbucks-singapore-160                      starbucks-singapore-160   \n",
       "ogopogo-singapore                                  ogopogo-singapore   \n",
       "\n",
       "                                   collab_filter_predicted_ratings  \n",
       "singapore-swimming-club-singapore                         3.699315  \n",
       "starbucks-singapore-160                                   3.688314  \n",
       "ogopogo-singapore                                         3.678975  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#organizing the above series column into a df of recommendations and collaborative filtering rating predictions\n",
    "collab_2552_df = pd.DataFrame({'recommendations':collab_rankedrecs_2552.index,\n",
    "              'collab_filter_predicted_ratings':collab_rankedrecs_2552})\n",
    "collab_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- Now, for content-based filtering aspect..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reading in the previously prepped df meant for content-based filtering here for content-based filtering recommendations..\n",
    "content_f = pd.read_csv('yelp_data/content_based_df_nouser.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>reviews</th>\n",
       "      <th>category_alias</th>\n",
       "      <th>review_count</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183-rojak-singapore</td>\n",
       "      <td>Opening a rojak stall in Toa Payoh isn't that ...</td>\n",
       "      <td>food</td>\n",
       "      <td>2</td>\n",
       "      <td>3.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>Located in the first basement level at MBS, cl...</td>\n",
       "      <td>singaporean coffee foodstands</td>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2am-dessert-bar-singapore</td>\n",
       "      <td>Creative desserts with several layers of flavo...</td>\n",
       "      <td>bars desserts coffee</td>\n",
       "      <td>38</td>\n",
       "      <td>3.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2nd-mini-steamboat-delight-singapore</td>\n",
       "      <td>Fantastic Authentic Place!!!What a treat and a...</td>\n",
       "      <td>cafes</td>\n",
       "      <td>2</td>\n",
       "      <td>4.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>365-fruit-juice-and-smoothie-singapore</td>\n",
       "      <td>Real juice, with real fruits and vegetables, a...</td>\n",
       "      <td>juicebars</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    shops  \\\n",
       "0                     183-rojak-singapore   \n",
       "1     1983-a-taste-of-nanyang-singapore-2   \n",
       "2               2am-dessert-bar-singapore   \n",
       "3    2nd-mini-steamboat-delight-singapore   \n",
       "4  365-fruit-juice-and-smoothie-singapore   \n",
       "\n",
       "                                             reviews  \\\n",
       "0  Opening a rojak stall in Toa Payoh isn't that ...   \n",
       "1  Located in the first basement level at MBS, cl...   \n",
       "2  Creative desserts with several layers of flavo...   \n",
       "3  Fantastic Authentic Place!!!What a treat and a...   \n",
       "4  Real juice, with real fruits and vegetables, a...   \n",
       "\n",
       "                  category_alias  review_count  rating  \n",
       "0                           food             2     3.5  \n",
       "1  singaporean coffee foodstands             4     4.0  \n",
       "2           bars desserts coffee            38     3.5  \n",
       "3                          cafes             2     4.5  \n",
       "4                      juicebars             1     4.0  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out how the first few rows of the df look like.\n",
    "content_f.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(981, 5)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the dimensions of the df...\n",
    "content_f.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#merging userid2552's info with the df meant for content-based filtering so that rcontent-based filtering can make recommendations via rating predictions for userid 2552 later on...\n",
    "content_2552 = pd.merge(content_f,user_item_2552,how='left',on='shops')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>reviews</th>\n",
       "      <th>category_alias</th>\n",
       "      <th>review_count</th>\n",
       "      <th>rating</th>\n",
       "      <th>ratings</th>\n",
       "      <th>userids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183-rojak-singapore</td>\n",
       "      <td>Opening a rojak stall in Toa Payoh isn't that ...</td>\n",
       "      <td>food</td>\n",
       "      <td>2</td>\n",
       "      <td>3.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>Located in the first basement level at MBS, cl...</td>\n",
       "      <td>singaporean coffee foodstands</td>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2am-dessert-bar-singapore</td>\n",
       "      <td>Creative desserts with several layers of flavo...</td>\n",
       "      <td>bars desserts coffee</td>\n",
       "      <td>38</td>\n",
       "      <td>3.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  \\\n",
       "0                  183-rojak-singapore   \n",
       "1  1983-a-taste-of-nanyang-singapore-2   \n",
       "2            2am-dessert-bar-singapore   \n",
       "\n",
       "                                             reviews  \\\n",
       "0  Opening a rojak stall in Toa Payoh isn't that ...   \n",
       "1  Located in the first basement level at MBS, cl...   \n",
       "2  Creative desserts with several layers of flavo...   \n",
       "\n",
       "                  category_alias  review_count  rating  ratings  userids  \n",
       "0                           food             2     3.5      NaN      NaN  \n",
       "1  singaporean coffee foodstands             4     4.0      NaN      NaN  \n",
       "2           bars desserts coffee            38     3.5      NaN      NaN  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the first few rows of which...\n",
    "content_2552.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#getting dummies for categorical columns...\n",
    "content_2552_wdummies = pd.get_dummies(content_2552, columns=['shops','category_alias'], drop_first=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "#setting feature and target\n",
    "X = content_2552_wdummies.drop(['ratings'], axis=1)\n",
    "y = content_2552_wdummies['ratings']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#collating dummified columns\n",
    "shops_cats_list = [col for col in content_2552_wdummies.columns if (col.startswith('shops')) or (col.startswith('category'))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "#extending with review_count and rating\n",
    "shops_cats_list.extend(['review_count','rating','userids'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "#as tfidf can only work on one column of texts at a time, am separating features as below...\n",
    "X1 = X['reviews']\n",
    "X2 = X[shops_cats_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    Opening a rojak stall in Toa Payoh isn't that ...\n",
       "1    Located in the first basement level at MBS, cl...\n",
       "2    Creative desserts with several layers of flavo...\n",
       "Name: reviews, dtype: object"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the first few rows of reviews prior to preprocessing...\n",
    "X1.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Assigning a new variable name to X1 for processing.\n",
    "rev = X1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating customized stop words' list\n",
    "cust_stop_words = [word for word in stop_words.ENGLISH_STOP_WORDS]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "#adding on to the above list based on preliminary word cloud EDA\n",
    "cust_stop_words.extend([\"wa\",\"ha\",\"just\",\"ve\",\"did\",\"got\",\"quite\"]) #adding \"wa\" into the list of stop words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "#preprocessing text in reviews by defining a function to do so\n",
    "lemm = WordNetLemmatizer()\n",
    "\n",
    "def text_processer(raw_text):\n",
    "    # Function to convert a raw string of text to a string of words\n",
    "    # The input is a single string (a raw unprocessed text), and \n",
    "    # the output is a single string (a preprocessed text)\n",
    "    \n",
    "    # 1. Remove http urls.\n",
    "    review_text = re.sub(\"\\(http.+\\)\", \" \", raw_text)\n",
    "    \n",
    "    # 2. Remove non-letters.\n",
    "    letters_only = re.sub(\"[^a-zA-Z]\", \" \", review_text)\n",
    "    \n",
    "    # 3. Convert to lower case, split into individual words.\n",
    "    words = letters_only.lower().split()\n",
    "    \n",
    "    # 4. Lemmatize words.\n",
    "    lemmed_words = [lemm.lemmatize(i) for i in words]\n",
    "    \n",
    "    # 5. Remove stop words.\n",
    "    \n",
    "    meaningful_words = [w for w in lemmed_words if not w in cust_stop_words]\n",
    "    \n",
    "    # 6. Join the words back into one string separated by space, \n",
    "    # and return the result.\n",
    "    return(\" \".join(meaningful_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    opening rojak stall toa payoh isn t easy tough...\n",
       "1    located basement level mb close sw entrance es...\n",
       "2    creative dessert layer flavor enjoyed looking ...\n",
       "dtype: object"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#showing how the processed reviews look like\n",
    "rev_processed = pd.Series([text_processer(text) for text in rev])\n",
    "rev_processed[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(981, 18216)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#using tfidf vectorizer to convert the reviews into term frequency columns...\n",
    "tvec_naive = TfidfVectorizer(stop_words = cust_stop_words)  #instantiating TfidfVectorizer with customized stop words\n",
    "\n",
    "X1_tvec_naive = tvec_naive.fit_transform(rev_processed).todense()   #fitting tvec and transforming the processed reviews\n",
    "X1_tvec_naive_df = pd.DataFrame(X1_tvec_naive, columns = tvec_naive.get_feature_names())  #converting it into a dataframe for easy lookup.\n",
    "X1_tvec_naive_df.shape #checking how array's dimension has changed with the vectorization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(981, 19498)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#combining tvec-df with the rest of the features for rating prediction for userid 2552 later on...\n",
    "X_legit = pd.concat([X1_tvec_naive_df,X2], axis=1)\n",
    "X_legit.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "#adding back the column of ratings so that it can be dropped below-sorry sometimes my train of thought may sound illogical\n",
    "X_legit['ratings'] = y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating X_train manually for userid 2552\n",
    "X_train_2552 = X_legit[X_legit['userids']==2552].drop(['ratings','userids'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating y_train manually for userid 2552\n",
    "y_train_2552 = X_legit[X_legit['userids']==2552]['ratings']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating X_test manually for userid 2552 which contains all outlets that have not been rated by userid 2552\n",
    "X_test_2552 = X_legit[X_legit['userids']!=2552].drop(['ratings','userids'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiate scaler since not all of the features are of the same scale, eg. review_count and rating\n",
    "ss= StandardScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "#fitting the train and transforming both the train and test sets\n",
    "X_train_2552_sc = ss.fit_transform(X_train_2552)\n",
    "X_test_2552_sc = ss.transform(X_test_2552)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "#this is to try if tuning the model every time a user feeds ratings into the recommender would produce a better outcome\n",
    "#params = {\n",
    "#    'max_depth':[5,7,10],\n",
    "#    'min_samples_split':[10, 15, 20],\n",
    "#    'min_samples_leaf':[3, 4, 5],\n",
    "#    'class_weight':['balanced']\n",
    "#    } \n",
    "\n",
    "\n",
    "#dtc_gs = GridSearchCV(DecisionTreeClassifier(), params, cv = 2, verbose = 1, n_jobs = -1)\n",
    "#dtc_gs.fit(X_train_2552_sc, y_train_2552)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print('Gridsearch best score: ', dtc_gs.best_score_)\n",
    "#print('Gridsearch best estimator: ', dtc_gs.best_estimator_)\n",
    "#print('Gridsearch best params: ',dtc_gs.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- Terrible gridsearchcv results of 0.2 accuracy score when user only provides 2 ratings for each rating bracket (total 10 ratings) as that would have forced the number of cross-validation folds to be maximally 2-meaning 50/50 split and testing twice only for each of the hyperparameter combination to be searched upon... This would mean that the user would need to input at least 10 ratings in each rating bracket, which would total to 50 ratings in order to provide a representative picture of all rating brackets (1 - 5), which would be unduly onerous for the user!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loading a previously tuned content-based filtering model for one user (userid 2043) here to see the kind of rating predictions it can generate for a different user, userid 2552...\n",
    "#loaded_model = joblib.load('xgb_model.sav')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "#learning rate, max depth, and n_estimators' values were obtained from previously tuned xgb model - xgb model.sav\n",
    "xgb = XGBClassifier(learning_rate=0.5, max_depth=9, n_estimators=200, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "              colsample_bytree=1, gamma=0, learning_rate=0.5, max_delta_step=0,\n",
       "              max_depth=9, min_child_weight=1, missing=None, n_estimators=200,\n",
       "              n_jobs=1, nthread=None, objective='multi:softprob',\n",
       "              random_state=42, reg_alpha=0, reg_lambda=1, scale_pos_weight=1,\n",
       "              seed=None, silent=True, subsample=1)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#training the loaded model on the dataset containing the new user, userid 2552's ratings.\n",
    "xgb.fit(X_train_2552_sc, y_train_2552)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "#stacking X_test_2552 as first step in regenerating the shops column for predictions\n",
    "trial = X_test_2552.stack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating loop to re-generate original X_test_2552 order of shops\n",
    "index_lst = []\n",
    "outlets_lst = []\n",
    "for n in range(len(trial.index)):\n",
    "    if trial.index[n][1].startswith('shops_') and trial[n]!=0:\n",
    "        index_lst.append(str(trial.index[n][0]))\n",
    "        outlets_lst.append(trial.index[n][1])\n",
    "index_lst = [int(x) for x in index_lst]\n",
    "reconstructed_X_test_2552 = pd.DataFrame({'shops':outlets_lst}, index=index_lst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "#generating content-based filtering rating predictions for userid 2552\n",
    "rating_predictions = xgb.predict(X_test_2552_sc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "#adding new column of rating predictions into the reconstructed X_test_2552\n",
    "reconstructed_X_test_2552['predicted_ratings']=rating_predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(971, 2)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#giving the reconstructed df a more easily understood name for distinction from the collaborative filtering df dealt with above\n",
    "content_2552_df = reconstructed_X_test_2552\n",
    "content_2552_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(971, 2)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the dimensions of the collaborative filtering df\n",
    "collab_2552_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#ensuring that the content-based filtering df for userid 2552 has no null values\n",
    "sum(content_2552_df.isnull().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#ensuring that the collaborative filtering-based df for userid 2552 has no null values\n",
    "sum(collab_2552_df.isnull().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>shops_183-rojak-singapore</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>shops_1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>shops_2am-dessert-bar-singapore</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>shops_2nd-mini-steamboat-delight-singapore</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>shops_365-fruit-juice-and-smoothie-singapore</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          shops  predicted_ratings\n",
       "0                     shops_183-rojak-singapore                2.0\n",
       "1     shops_1983-a-taste-of-nanyang-singapore-2                2.0\n",
       "2               shops_2am-dessert-bar-singapore                1.0\n",
       "3    shops_2nd-mini-steamboat-delight-singapore                1.0\n",
       "4  shops_365-fruit-juice-and-smoothie-singapore                3.0"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#having a look at some of the ratings predicted by content-based filtering for userid 2552. Oh no, looks like content-based filtering was only able to predict all rating 1s! Let's see if the collaborative filtering can compensate for this shortcoming...\n",
    "content_2552_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "#trimming off the shops' prefixes so that they can eventually be merged with the collaborative filtering df\n",
    "content_2552_df['shops'] = content_2552_df['shops'].apply(lambda x: x[6:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183-rojak-singapore</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2am-dessert-bar-singapore</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  predicted_ratings\n",
       "0                  183-rojak-singapore                2.0\n",
       "1  1983-a-taste-of-nanyang-singapore-2                2.0\n",
       "2            2am-dessert-bar-singapore                1.0"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#shops' prefixes have been trimmed off!\n",
    "content_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "#renaming the column of rating predictions to distinguish from collaborative filtering's prediction column later on when both dfs are merged.\n",
    "content_2552_df.rename(columns={'predicted_ratings':'content_filter_predicted_ratings'},inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>content_filter_predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183-rojak-singapore</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2am-dessert-bar-singapore</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  content_filter_predicted_ratings\n",
       "0                  183-rojak-singapore                               2.0\n",
       "1  1983-a-taste-of-nanyang-singapore-2                               2.0\n",
       "2            2am-dessert-bar-singapore                               1.0"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the renamed column...\n",
    "content_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>recommendations</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>singapore-swimming-club-singapore</th>\n",
       "      <td>singapore-swimming-club-singapore</td>\n",
       "      <td>3.699315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>starbucks-singapore-160</th>\n",
       "      <td>starbucks-singapore-160</td>\n",
       "      <td>3.688314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ogopogo-singapore</th>\n",
       "      <td>ogopogo-singapore</td>\n",
       "      <td>3.678975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mahota-commune-singapore</th>\n",
       "      <td>mahota-commune-singapore</td>\n",
       "      <td>3.676981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>collective-brewers-singapore</th>\n",
       "      <td>collective-brewers-singapore</td>\n",
       "      <td>3.671374</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     recommendations  \\\n",
       "singapore-swimming-club-singapore  singapore-swimming-club-singapore   \n",
       "starbucks-singapore-160                      starbucks-singapore-160   \n",
       "ogopogo-singapore                                  ogopogo-singapore   \n",
       "mahota-commune-singapore                    mahota-commune-singapore   \n",
       "collective-brewers-singapore            collective-brewers-singapore   \n",
       "\n",
       "                                   collab_filter_predicted_ratings  \n",
       "singapore-swimming-club-singapore                         3.699315  \n",
       "starbucks-singapore-160                                   3.688314  \n",
       "ogopogo-singapore                                         3.678975  \n",
       "mahota-commune-singapore                                  3.676981  \n",
       "collective-brewers-singapore                              3.671374  "
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the first few rows of the collaborative filtering df for userid 2552\n",
    "collab_2552_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "#renaming collaborative filtering df's recommendations' column so that it can be merged with the content-based filtering df.\n",
    "collab_2552_df.rename(columns={'recommendations':'shops'},inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>singapore-swimming-club-singapore</th>\n",
       "      <td>singapore-swimming-club-singapore</td>\n",
       "      <td>3.699315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>starbucks-singapore-160</th>\n",
       "      <td>starbucks-singapore-160</td>\n",
       "      <td>3.688314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ogopogo-singapore</th>\n",
       "      <td>ogopogo-singapore</td>\n",
       "      <td>3.678975</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                               shops  \\\n",
       "singapore-swimming-club-singapore  singapore-swimming-club-singapore   \n",
       "starbucks-singapore-160                      starbucks-singapore-160   \n",
       "ogopogo-singapore                                  ogopogo-singapore   \n",
       "\n",
       "                                   collab_filter_predicted_ratings  \n",
       "singapore-swimming-club-singapore                         3.699315  \n",
       "starbucks-singapore-160                                   3.688314  \n",
       "ogopogo-singapore                                         3.678975  "
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the renamed column\n",
    "collab_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reseting the index in the collaborative filtering df so that the index is numerical again\n",
    "collab_2552_df.reset_index(drop=True,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>singapore-swimming-club-singapore</td>\n",
       "      <td>3.699315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>starbucks-singapore-160</td>\n",
       "      <td>3.688314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ogopogo-singapore</td>\n",
       "      <td>3.678975</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               shops  collab_filter_predicted_ratings\n",
       "0  singapore-swimming-club-singapore                         3.699315\n",
       "1            starbucks-singapore-160                         3.688314\n",
       "2                  ogopogo-singapore                         3.678975"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the reset index of the collaborative filtering df\n",
    "collab_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "#merging both content-based filtering and collaborating filtering df to prepare to make hybrid recommendations for userid 2552\n",
    "content_collab_2552_df = pd.merge(content_2552_df,collab_2552_df,how='inner',on='shops')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>content_filter_predicted_ratings</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183-rojak-singapore</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.856616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.912037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2am-dessert-bar-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.637525</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  content_filter_predicted_ratings  \\\n",
       "0                  183-rojak-singapore                               2.0   \n",
       "1  1983-a-taste-of-nanyang-singapore-2                               2.0   \n",
       "2            2am-dessert-bar-singapore                               1.0   \n",
       "\n",
       "   collab_filter_predicted_ratings  \n",
       "0                         2.856616  \n",
       "1                         2.912037  \n",
       "2                         3.637525  "
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#looking at the first few rows of the combined df for userid 2552\n",
    "content_collab_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "#as mentioned in the previous sub-notebook on this hybrid recommender's evaluation, the following are the content-based and collaborative filtering's ratings' weights\n",
    "con_wt = 0.97 / (0.97 + 1.0)\n",
    "collab_wt = 1.0 / (0.97 + 1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "#feature engineering to add hybrid recommender's rating predictions into the combined df by multiplying the respective rating predictions by weights based on both models' f1 scores derived from prior evaluation and summing them up to yield hybrid predictions\n",
    "content_collab_2552_df['final_weighted_rating_predictions'] = (content_collab_2552_df['content_filter_predicted_ratings']*con_wt) + (content_collab_2552_df['collab_filter_predicted_ratings']*collab_wt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>content_filter_predicted_ratings</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "      <th>final_weighted_rating_predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183-rojak-singapore</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.856616</td>\n",
       "      <td>2.434830</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1983-a-taste-of-nanyang-singapore-2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.912037</td>\n",
       "      <td>2.462963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2am-dessert-bar-singapore</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.637525</td>\n",
       "      <td>2.338845</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 shops  content_filter_predicted_ratings  \\\n",
       "0                  183-rojak-singapore                               2.0   \n",
       "1  1983-a-taste-of-nanyang-singapore-2                               2.0   \n",
       "2            2am-dessert-bar-singapore                               1.0   \n",
       "\n",
       "   collab_filter_predicted_ratings  final_weighted_rating_predictions  \n",
       "0                         2.856616                           2.434830  \n",
       "1                         2.912037                           2.462963  \n",
       "2                         3.637525                           2.338845  "
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the first few rows of the final hybrid df\n",
    "content_collab_2552_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>content_filter_predicted_ratings</th>\n",
       "      <th>collab_filter_predicted_ratings</th>\n",
       "      <th>final_weighted_rating_predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>243</th>\n",
       "      <td>empire-cafe-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.656636</td>\n",
       "      <td>4.318089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>255</th>\n",
       "      <td>fine-palate-cafe-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.652950</td>\n",
       "      <td>4.316218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>brunetti-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.650090</td>\n",
       "      <td>4.314767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>baristart-coffee-sentosa-southern-islands</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.649774</td>\n",
       "      <td>4.314606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>352</th>\n",
       "      <td>jcone-jipangyi-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.646433</td>\n",
       "      <td>4.312910</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         shops  \\\n",
       "243                      empire-cafe-singapore   \n",
       "255                 fine-palate-cafe-singapore   \n",
       "100                         brunetti-singapore   \n",
       "63   baristart-coffee-sentosa-southern-islands   \n",
       "352                   jcone-jipangyi-singapore   \n",
       "\n",
       "     content_filter_predicted_ratings  collab_filter_predicted_ratings  \\\n",
       "243                               5.0                         3.656636   \n",
       "255                               5.0                         3.652950   \n",
       "100                               5.0                         3.650090   \n",
       "63                                5.0                         3.649774   \n",
       "352                               5.0                         3.646433   \n",
       "\n",
       "     final_weighted_rating_predictions  \n",
       "243                           4.318089  \n",
       "255                           4.316218  \n",
       "100                           4.314767  \n",
       "63                            4.314606  \n",
       "352                           4.312910  "
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#top 5 coffee-drinking outlet recommendations for userid 2552 (me!) based on my ratings given rather randomly to 10 of the outlets earlier on...\n",
    "content_collab_2552_df.sort_values('final_weighted_rating_predictions',ascending=False).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>final_weighted_rating_predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>243</th>\n",
       "      <td>empire-cafe-singapore</td>\n",
       "      <td>4.318089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>255</th>\n",
       "      <td>fine-palate-cafe-singapore</td>\n",
       "      <td>4.316218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>brunetti-singapore</td>\n",
       "      <td>4.314767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>baristart-coffee-sentosa-southern-islands</td>\n",
       "      <td>4.314606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>352</th>\n",
       "      <td>jcone-jipangyi-singapore</td>\n",
       "      <td>4.312910</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         shops  \\\n",
       "243                      empire-cafe-singapore   \n",
       "255                 fine-palate-cafe-singapore   \n",
       "100                         brunetti-singapore   \n",
       "63   baristart-coffee-sentosa-southern-islands   \n",
       "352                   jcone-jipangyi-singapore   \n",
       "\n",
       "     final_weighted_rating_predictions  \n",
       "243                           4.318089  \n",
       "255                           4.316218  \n",
       "100                           4.314767  \n",
       "63                            4.314606  \n",
       "352                           4.312910  "
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_5_recs = content_collab_2552_df[['shops','final_weighted_rating_predictions']].sort_values('final_weighted_rating_predictions',ascending=False).head()\n",
    "top_5_recs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_5_recs.reset_index(drop=True,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>final_weighted_rating_predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>empire-cafe-singapore</td>\n",
       "      <td>4.318089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>fine-palate-cafe-singapore</td>\n",
       "      <td>4.316218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>brunetti-singapore</td>\n",
       "      <td>4.314767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>baristart-coffee-sentosa-southern-islands</td>\n",
       "      <td>4.314606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>jcone-jipangyi-singapore</td>\n",
       "      <td>4.312910</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       shops  \\\n",
       "0                      empire-cafe-singapore   \n",
       "1                 fine-palate-cafe-singapore   \n",
       "2                         brunetti-singapore   \n",
       "3  baristart-coffee-sentosa-southern-islands   \n",
       "4                   jcone-jipangyi-singapore   \n",
       "\n",
       "   final_weighted_rating_predictions  \n",
       "0                           4.318089  \n",
       "1                           4.316218  \n",
       "2                           4.314767  \n",
       "3                           4.314606  \n",
       "4                           4.312910  "
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_5_recs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'jcone-jipangyi-singapore'"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_5_recs.loc[4,'shops']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2773.0"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#content-filtering could only predict 1.0 for all of the other outlets that userid 2552 (me!) has not rated...\n",
    "content_collab_2552_df['content_filter_predicted_ratings'].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0    258\n",
       "1.0    214\n",
       "5.0    210\n",
       "3.0    163\n",
       "4.0    126\n",
       "Name: content_filter_predicted_ratings, dtype: int64"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "content_collab_2552_df.content_filter_predicted_ratings.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(971, 4)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "content_collab_2552_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpretation of trial's results\n",
    "---\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- Interestingly enough, content-based filtering failed to predict variations in ratings across all 971 outlets that I have not rated...fortunately, I have the model-based collaborative filtering as a backup to compensate where content-based filtering has failed, in this hybrid recommender...\n",
    "\n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- This hybrid recommender has shown that having the user provide an even distribution of ratings (2 each of ratings 1-5) for 10 different outlets he/she has visited from the getgo out of 980+ outlets (basically training only on about 1% of the data and testing on 99%) is not a viable solution for a recommender that solely relies on a model-inclined content-based filtering system. The model-based collaborative filtering with ALS managed to salvage a little by providing a variation of ratings at least, suggesting that it is indeed robust to missing data-the 971 outlets not rated by userid 2552.\n",
    "\n",
    "</ul>    \n",
    "    \n",
    "<ul>\n",
    "\n",
    "- As mentioned earlier, the 10 initial ratings provided by me were arbitrary and perhaps rather random (not based on actual experience having visited the outlets that I have rated, but purely based instead on my impression of the name of the outlets). As such, it may not be surprising that the user ratings predicted by the content-based filtering algorithm do not make sense. Perhaps I can deploy this proper on a platform for actual coffee lovers to try out and validate it in what is called an A/B testing as part of the future plans of this project.\n",
    "    \n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models' Summary\n",
    "---\n",
    "\n",
    "<ul>\n",
    "\n",
    "- <font size='3'>__Content-based Filtering (baseline accuracy: 0.48)__</font>:\n",
    "    \n",
    "    </ul>\n",
    "\n",
    "|<center><font size='2'>Model<center>|<center><font size='2'>Accuracy<center>|<center><font size='2'>Micro-Average<br>Precision<center>|<center><font size='2'>Micro-Average<br>Recall<center>|<center><font size='2'>Micro-Average<br>$F_1$ score<center>|<center><font size='2'>Micro-Average<br>ROC AUC<center>|<center><font size='2'>Prevalence-Weighted<br>ROC AUC<center>|\n",
    "|---|---|---|---|---|---|---|\n",
    "|<center><font size='1'>*Logistic Regression<br>with<br>TfidfVectorizer*<center>|<center>0.81<center>|<center>0.81<center>|<center>0.81<center>|<center>0.81<center>|<center>0.88<center>|<center>0.71<center>|\n",
    "|<center><font size='1'>*Logistic Regression<br>with<br>TfidfVectorizer<br>with<br>PCA*<center>|<center>0.50<center>|<center>0.50<center>|<center>0.50<center>|<center>0.50<center>|<center>0.65<center>|<center>0.57<center>|\n",
    "|<center><font size='1'>***XGB Classifier<br>with<br>TfidfVectorizer (chosen)***<center>|<center>***0.97***<center>|<center>***0.97***<center>|<center>***0.97***<center>|<center>***0.97***<center>|<center>***1.0***<center>|<center>***1.0***<center>|\n",
    "|<center><font size='1'>Decision Tree Classifier<br>with<br>TfidfVectorizer<center>|<center>0.85<center>|<center>0.85<center>|<center>0.85<center>|<center>0.85<center>|<center>0.94<center>|<center>0.90<center>|\n",
    "|<center><font size='1'>*Decision Tree Classifier<br>with<br>TfidfVectorizer<br>with<br>PCA*<center>|<center>0.43<center>|<center>0.43<center>|<center>0.43<center>|<center>0.43<center>|<center>0.62<center>|<center>0.47<center>|\n",
    "|<center><font size='1'>*Random Forest Classifier<br>with<br>TfidfVectorizer*<center>|<center>0.61<center>|<center>0.61<center>|<center>0.61<center>|<center>0.61<center>|<center>0.92<center>|<center>0.81<center>|\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- <font size='3'>__Model-based Collaborative Filtering (baseline accuracy: 0.47)__</font>:\n",
    "    \n",
    "    </ul>\n",
    "\n",
    "|<center><font size='2'>Model<center>|<center><font size='2'>Accuracy<center>|<center><font size='2'>Micro-Average<br>Precision<center>|<center><font size='2'>Micro-Average<br>Recall<center>|<center><font size='2'>Micro-Average<br>$F_1$ score<center>|\n",
    "|---|---|---|---|---|\n",
    "|<center><font size='1'>*Alternating Least Squares (ALS)*<center>|<center>1.0<center>|<center>1.0<center>|<center>1.0<center>|<center>1.0<center>|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- <font size='3'>__Hybrid Recommender (baseline accuracy: 0.48)__</font>:\n",
    "    \n",
    "    </ul>\n",
    "\n",
    "|<center><font size='2'>Model<center>|<center><font size='2'>Accuracy<center>|<center><font size='2'>Micro-Average<br>Precision<center>|<center><font size='2'>Micro-Average<br>Recall<center>|<center><font size='2'>Micro-Average<br>$F_1$ score<center>|\n",
    "|---|---|---|---|---|\n",
    "|<center><font size='1'>*Hybrid Recommender<br>(ALS and XGB Classifier)*<center>|<center>1.0<center>|<center>1.0<center>|<center>1.0<center>|<center>1.0<center>|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Improvements and Current Limitations\n",
    "---\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- The major limitation with the earlier phase of the project was that the content-based filtering was trained and tuned on only a single userid's ratings which may not be representative of the vast majority even though said userid rated a considerable number of outlets... In this extension, the content-based filtering was not only trained on more than 1 userid (110 to be precise), but time was spent tuning an XGB Classifier algorithm properly in an attempt to mitigate this under-representation issue. This time round, training data was restricted arbitrarily to those who have rated at least 10 outlets (only 110 userids out of 2552; the rest rated less than 10 outlets - quite a significant number rated only 1-2 outlets and including them will make the train_test_split and cross-validation aspect of the project problematic since it is important to stratify the splitting by userids in the evaluation stage).\n",
    "<ul>\n",
    "    - Indeed, the XGB Classifier did not disappoint, with a near-perfect scores of 0.97 - 1.0 in its performance.\n",
    "    </ul>\n",
    "    \n",
    "</ul>\n",
    "    \n",
    "<ul>\n",
    "    \n",
    "- In order to evaluate ALS' rating predictions on the same grounds as the content-based filtering that I have evaluated with classification metrics (since the ratings fall into finite discrete classes), ALS' continuous rating predictions were rounded off to nearest whole numbers and those falling into incorrect rating classes such as -1.0 or 0.0 were manually coded as the nearest possible value-1.0. Even though these incorrect classes were already mis-classified and correcting them did not alter the spark evaluator-computed $F_1$ score (still 0.98 after corrections), it should be noted that the ALS predictions were \"tweaked\". There should be better, more reliable options to convert the continuous output into discrete rating classes but for now, I am making do with it and qualifying this as an \"assumption\" by which the results of my evaluation of the ALS component holds true. One way of improving this is to consider incorporating a logistic/sigmoid function ($f(x)$) into the matrix factorization loss function to automatically squeeze the range of predicted output ratings down to probabilities that range between 0 and 1 for each discrete rating class: \n",
    "\n",
    "</ul>\n",
    "\n",
    "<img src=\"yelp_data/extn_matfac.png\"/>\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- The above extension of matrix factorization can then be adapted and extended to more complex algorithms like neural networks, which are used for near state-of-the-art recommenders.\n",
    "    \n",
    "</ul>\n",
    "\n",
    "<ul>    \n",
    "    \n",
    "- Another limitation is that this hybrid model only takes into account explicit users' preferences in the form of ratings and reviews. Perhaps implicit feedback could be incorporated such as clickthrough data, or page views (which are not easily obtainable, if not impossible to obtain, just from the Yelp API token and the scraping of its websites via BeautifulSoup) to supplement the hybrid recommender.\n",
    "    \n",
    "</ul>    \n",
    "    \n",
    "<ul>\n",
    "\n",
    "- Possibly yet another limitation could be that ```TfidfVectorizer``` was not tuned and naive vectorizer was fitted instead as reviews were not the only input feature but was combined with numerical feature columns for the content-based filtering. As such, it was difficult, if not impossible, to tune the vetorizer for example, limiting the ```max_features``` hyperparameter. One possibility of improving this though, could be to use an [Auto-encoder](https://towardsdatascience.com/creating-a-hybrid-content-collaborative-movie-recommender-using-deep-learning-cc8b431618af) which leverages on a neural network to decompose the multi-dimensional tfidf matrix down to its crucial components.\n",
    "\n",
    "</ul>        \n",
    "\n",
    "<ul>\n",
    "    \n",
    "- Consider mean normalization and NaN or null ratings where instead of eliminating NaNs or null ratings (user had not rated outlet), mean normalize them such that NaNs are converted into zero first and then eventually add the average outlet rating to these zeros to yield the predicted ratings as in [here](https://www.coursera.org/lecture/machine-learning/implementational-detail-mean-normalization-Adk8G) for those previously null rating values. This could add value by generating rating predictions for users who opt not to rate any outlets at all when using this hybrid recommender system, and therefore partially address the cold-start problem common to collaborative filtering systems.\n",
    "    \n",
    "</ul>\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- The scraped data may be incomplete or its quality may not be optimal as the structuring of the html source code behind Yelp's webpage is such that it is hard to ensure that all userids are aligned to all ratings - there may be some pages with an extra rating and review with no userid attached as that was an old review by the userid who posted an updated post right above the old one (it is difficult to write a code with BeautifulSoup to distinguish that and scrape the relevant details accordingly). Fortunately, this scenario is not extremely common on Yelp. Perhaps increased data quality and more of it will go a long way in improving this hybrid recommender system.\n",
    "    \n",
    "    </ul>\n",
    "        \n",
    "<ul>\n",
    "    \n",
    "- Lastly, this dataset is static and any deployed app will need to be updated regularly to remain relevant...\n",
    "    \n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions and Future Plans\n",
    "---\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- There is a potential that if more time is provided to tune certain algorithms like DecisionTreeClassifier with tfidf and PCA as well as RandomForestClassifier with tfidf etc, they might be able to perform better than the chosen DecisionTreeClassifier with only tfidf since for some of these models, their tuned hyperparameters fall on either edge of the search space provided for the hyperparameters in the parameter grid - this suggests that if tuned a couple more times, providing different hyperparameter values beyond the previous search grid each time, a more optimal model could be achieved eventually for each of these algorithms that had not been chosen. Furthermore, ```PCA``` was not tuned as well - this could help reduce the dimensionality of the data and reduce model variance while maintaining all features which is crucial since all the outlets are coded as features and they should not be removed so that there is at least a predicted rating for all outlets - we do not want to miss out on possible outlets the user could be interested in from the list of recommendations (beyond the top 5 or 10). Perhaps the ```PCA``` should be fitted onto just ```X_train``` instead of ```X_train_sc``` as I had casually experimented on the former near the end of the capstone period with some multiclass ROC AUC plots and scores and it churned out better micro-average ROC AUCs and weighted-by-prevalence ROC AUCs. That said, there has been [advice online](https://stackoverflow.com/questions/39685740/calculate-sklearn-roc-auc-score-for-multi-class) that multiclass classification should better be assessed by its usual methods-confusion matrix-instead...\n",
    "\n",
    "</ul>\n",
    "\n",
    "\n",
    "<ul>\n",
    "    \n",
    "- Built a Flask app with this hybrid recommendation system and it was implemented successfully in a local virtual environment as shown below (screenshots). However, it takes on average 15 mins or more for the recommendations to be generated... Regarding the actual deployment on a platform like Heroku, as the pyspark component is a lot more intractable and there are very few, if not no resources online on deployment of pyspark apps on Heroku, I was not able to deploy the full hybrid recommendation system and wound up only deploying the XGB Classifier-supported content-based filtering component (further screenshots below). Will keep a lookout for updates on the various deployment platforms, stackoverflow, and pyspark deployment to see if the pyspark ALS component can be incorporated into the deployment without incurring unnecessary billing, as tapping onto the [AWS S3 Bucketeer](https://elements.heroku.com/addons/bucketeer) that complements deployment of spark applications on Heroku is not complimentary. Please click [here](http://sg-coffee-recommender.herokuapp.com/) for the link to the content-based filtering deployed on Heroku. \n",
    "\n",
    "- Flask app:\n",
    "\n",
    "<img src=\"yelp_data/Flask_app_frontpage_1.png\"/>\n",
    "<img src=\"yelp_data/Flask_app_frontpage_2.png\"/>\n",
    "<img src=\"yelp_data/Flask_app_outcome.png\"/>\n",
    "    \n",
    "- Heroku App:\n",
    "\n",
    "<img src=\"yelp_data/heroku_app.png\"/>\n",
    "<img src=\"yelp_data/heroku_app_recs.png\"/>\n",
    "    \n",
    "</ul>\n",
    "\n",
    "\n",
    "<ul>\n",
    "\n",
    "- All in all, this capstone project is \"bootstrapp-ish\" as almost every component was cobbled together from its 'raw materials': From the data-scraping phase to constructing the content-based filtering model and combining it with a more-or-less well-established collaborative filtering model, and then aggregating rating predictions from both to yield the final hybrid prototype. There is no tried-and-tested method, gold-standard or novice-friendly digestible guide for constructing a full hybrid recommendation system out there. Although this hybrid system cannot be compared to commercial ones like Netflix, it serves as a peek into the nuts and bolts that go into creating a simple hybrid recommender from scratch for beginners. \n",
    "\n",
    "</ul>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sources\n",
    "---\n",
    "\n",
    "- https://java.com/en/download/help/download_options.xml\n",
    "- https://www.oracle.com/java/technologies/javase-jdk8-downloads.html\n",
    "- https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz\n",
    "- https://github.com/sbt/sbt/releases/download/v0.13.17/sbt-0.13.17.tgz\n",
    "- https://spark.apache.org/downloads.html\n",
    "- https://medium.com/luckspark/installing-spark-2-3-0-on-macos-high-sierra-276a127b8b85\n",
    "- https://towardsdatascience.com/creating-a-hybrid-content-collaborative-movie-recommender-using-deep-learning-cc8b431618af\n",
    "- https://towardsdatascience.com/prototyping-a-recommender-system-step-by-step-part-1-knn-item-based-collaborative-filtering-637969614ea\n",
    "- https://www.coursera.org/lecture/machine-learning/implementational-detail-mean-normalization-Adk8G\n",
    "- https://stackoverflow.com/questions/39685740/calculate-sklearn-roc-auc-score-for-multi-class\n",
    "- https://elements.heroku.com/addons/bucketeer\n",
    "- http://sg-coffee-recommender.herokuapp.com/\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}