Part 7_Model-based Collaborative Filtering with Alternating Least Squares (ALS)_alt_stratified splitting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 7: Model-based Collaborative Filtering with Alternating Least Squares (ALS) & Evaluation on sample userid 2043\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Foreword on Alternating Least Squares (ALS)\n",
    "---\n",
    "\n",
    "- Alternating Least Squares (ALS) algorithm is a matrix factorization technique that decomposes user-item interaction matrix (such as user-item ratings matrix for datasets with explicit feedback) into user and item latent factors where their dot product will predict user ratings. It alternates between fixing user or item latent factors to solve for the other via gradient descent at each iteration in the process of minimizing loss. In the course of doing so, it learns these latent user and item factors that predict the user ratings almost just like how a regressor learns coefficients that predict the target, based on input features.\n",
    "\n",
    "<img src=\"yelp_data/matfact_explain.png\"/>\n",
    "\n",
    "- ALS can be imported from ```implicit``` library for datasets with implicit feedback (Eg. clickthroughs and page views) or ```pyspark.ml.recommendations``` (which can be used for either implicit or explicit data). Since my data involves user ratings, I will be relying on the pyspark machine learning library.\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Please follow the steps below to download and install all the relevant libraries and dependencies BEFORE running this notebook to avoid encountering any errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- First up, navigate to your home directory and create a new directory called ```server```:\n",
    "    ```cd ~```\n",
    "    ```mkdir server```\n",
    "  Make sure that the stuff below will be downloaded into this ```server``` folder.\n",
    "  \n",
    "- In order to run spark and pyspark on your local machine, kindly ensure that you already have Java installed with the following command in your Terminal or Windows-equivalent in command prompt: ```java -version``` If nothing comes out of this, navigate to this [link](https://java.com/en/download/help/download_options.xml) to download java for mac or windows. You may need to restart your system after installation for java to take effect.\n",
    "\n",
    "- Next, navigate to this [link](https://www.oracle.com/java/technologies/javase-jdk8-downloads.html) to download the java development kit and then install it.\n",
    "\n",
    "- Check if scala is installed by executing this command in your Terminal or Windows-equivalent command prompt: ```scala -version```. If nothing comes out, navigate to this [link](https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz) to download and install scala as well as this [link](https://github.com/sbt/sbt/releases/download/v0.13.17/sbt-0.13.17.tgz) to download and install sbt-0.13.17.tgz.\n",
    "\n",
    "- Navigate to this [link](https://spark.apache.org/downloads.html) to download Apache Spark. Select the options like the screenshot below and click on \"spark-2.4.5-bin-hadoop2.7.tgz\" under point 3 to download spark and install it.\n",
    "<img src=\"yelp_data/spark_dl.png\"/>\n",
    "\n",
    "- The following should be the directory paths of the software you have downloaded and installed above, where ```HOMEDIRECTORY``` is your home directory's name:\n",
    "    JDK: ```/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk```\n",
    "    Sbt: ```/Users/HOMEDIRECTORY/server/sbt```\n",
    "    Scala: ```/Users/HOMEDIRECTORY/server/scala-2.11.12```\n",
    "    Spark: ```/Users/HOMEDIRECTORY/server/spark-2.4.5-bin-hadoop2.7```\n",
    "\n",
    "- After all of the above have been installed, set up a ```.bash_profile``` file in your home directory. For Mac users, if you do not already have a ```.bash_profile``` file, navigate to your home directory and create one by executing the following commands:\n",
    "    ```cd ~```\n",
    "    ```touch .bash_profile```\n",
    "    After which, open it with a text editor of your choice and add the following lines of code at the top of the ```.bash_profile``` file, replacing ```HOMEDIRECTORY``` with the name of your home directory:\n",
    "\n",
    "<img src=\"yelp_data/spark_bash_profile.png\"/>\n",
    "    \n",
    "       \n",
    "- Save and close the ```.bash_profile``` file and execute ```source ~/.bash_profile``` in your Terminal or Windows-equivalent command prompt.\n",
    "\n",
    "- Completely quit your Terminal and command prompt.\n",
    "\n",
    "- Now you may proceed to run the rest of the following code.\n",
    "\n",
    "\n",
    "- ***KINDLY NOTE THAT IF YOU HAVE ENCOUNTERED A CONNECTION REFUSED ERROR OR A JAVA ERROR WHERE IT IS TRYING TO CONNECT TO YOUR IP ADDRESS BUT FAILED WHEN RUNNING ANY PYSPARK-RELATED CELL, KINDLY JUST COPY ALL THE CELLS IN THE NOTEBOOK (HIGHLIGHT THE TOP CELL AND CMD(FOR MAC)/CTRL(FOR WINDOWS) + SHIFT + HIGHLIGHT THE LAST CELL), COPY AND PASTE INTO A FRESH NOTEBOOK AND RUN THEM THERE INSTEAD***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#this is required to allow pyspark to run in a jupyter notebook\n",
    "!pip install findspark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import findspark\n",
    "findspark.init()\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.ml.evaluation import RegressionEvaluator, MulticlassClassificationEvaluator\n",
    "from pyspark.ml.recommendation import ALS, ALSModel\n",
    "from pyspark.ml.feature import StringIndexer, IndexToString\n",
    "from pyspark.ml import Pipeline as PL\n",
    "from pyspark.sql.functions import col\n",
    "from pyspark.ml.tuning import CrossValidator, ParamGridBuilder, CrossValidatorModel\n",
    "from pyspark.sql.functions import max\n",
    "import pyspark.sql.functions as func\n",
    "from pyspark.ml import PipelineModel\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import confusion_matrix, classification_report, multilabel_confusion_matrix, roc_auc_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#starting the pyspark session\n",
    "spark = SparkSession.builder.appName('recommendation_system').getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reading in curated dataset for model-based collaborative filtering (mbcf)\n",
    "mbcf_hundten = spark.read.csv('yelp_data/mbcf_hundten.csv', header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+-------+-------+\n",
      "|              shops|ratings|userids|\n",
      "+-------------------+-------+-------+\n",
      "|hustle-co-singapore|    5.0|    532|\n",
      "|hustle-co-singapore|    5.0|   1397|\n",
      "|hustle-co-singapore|    5.0|   2073|\n",
      "+-------------------+-------+-------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#looking at how the mbcf looks like for a start\n",
    "mbcf_hundten.show(3,truncate=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-------+-------+\n",
      "|               shops|ratings|userids|\n",
      "+--------------------+-------+-------+\n",
      "| hustle-co-singapore|    5.0|   2043|\n",
      "|benjamin-barker-c...|    4.0|   2043|\n",
      "|the-coffee-roaste...|    3.0|   2043|\n",
      "|  koi-cafe-singapore|    5.0|   2043|\n",
      "|starbucks-singapo...|    4.0|   2043|\n",
      "+--------------------+-------+-------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#remember we want to test the ALS rating predictions on userid 2043, just as we did for content-based filtering by the various models earlier on\n",
    "mbcf_hundten.filter(\"userids = 2043\").show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "980"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#this confirmed earlier analyses (in part 1 sub-notebook) that show that userid 2043 rated 980 outlets.\n",
    "mbcf_hundten.filter(\"userids = 2043\").count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-------+-------+-----------+-------------+\n",
      "|               shops|ratings|userids|shops_index|userids_index|\n",
      "+--------------------+-------+-------+-----------+-------------+\n",
      "| hustle-co-singapore|    5.0|    532|      170.0|         50.0|\n",
      "| hustle-co-singapore|    5.0|   1397|      170.0|         56.0|\n",
      "| hustle-co-singapore|    5.0|   2073|      170.0|         90.0|\n",
      "| hustle-co-singapore|    5.0|   2043|      170.0|          0.0|\n",
      "|benjamin-barker-c...|    4.0|   2141|      452.0|         92.0|\n",
      "+--------------------+-------+-------+-----------+-------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#for ALS recommender algorithm to work, the feature columns must at least be in the numerical format - although some of the feature columns are already in the int format, let's just play safe by converting all except for the ratings' column into the double format.\n",
    "indexer = [StringIndexer(inputCol=column, outputCol=column+\"_index\") for column in list(set(mbcf_hundten.columns)-set(['ratings']))]\n",
    "pipeline = PL(stages=indexer)\n",
    "transformed = pipeline.fit(mbcf_hundten).transform(mbcf_hundten)\n",
    "transformed.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('shops', 'string'),\n",
       " ('ratings', 'double'),\n",
       " ('userids', 'int'),\n",
       " ('shops_index', 'double'),\n",
       " ('userids_index', 'double')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#the relevant dtypes are of the correct form now!\n",
    "transformed.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformed.toPandas().to_csv('pyspark_transformed_mbcf.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>ratings</th>\n",
       "      <th>userids</th>\n",
       "      <th>shops_index</th>\n",
       "      <th>userids_index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>hustle-co-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>532</td>\n",
       "      <td>170.0</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>hustle-co-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1397</td>\n",
       "      <td>170.0</td>\n",
       "      <td>56.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>hustle-co-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2073</td>\n",
       "      <td>170.0</td>\n",
       "      <td>90.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 shops  ratings  userids  shops_index  userids_index\n",
       "0  hustle-co-singapore      5.0      532        170.0           50.0\n",
       "1  hustle-co-singapore      5.0     1397        170.0           56.0\n",
       "2  hustle-co-singapore      5.0     2073        170.0           90.0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformed_pandas = pd.read_csv('yelp_data/pyspark_transformed_mbcf.csv')\n",
    "transformed_pandas.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#the difference in this alternate notebook part 8 is the train_test_split here that was stratified not by ratings but by userids instead since it is logical that the algorithm learns from data (train) on some ratings by a certain group of userids for some outlets and then predict ratings by the same group of userids for other outlets (test)\n",
    "train, test = train_test_split(transformed_pandas,test_size=0.2,random_state=42,stratify=transformed_pandas['userids'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2348, 5)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(587, 5)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>ratings</th>\n",
       "      <th>userids</th>\n",
       "      <th>shops_index</th>\n",
       "      <th>userids_index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1866</th>\n",
       "      <td>the-coconut-club-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>75</td>\n",
       "      <td>148.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           shops  ratings  userids  shops_index  userids_index\n",
       "1866  the-coconut-club-singapore      5.0       75        148.0           12.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test[(test.userids==75)&(test.shops=='the-coconut-club-singapore')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.to_csv('train_mbcf_hundten.csv',index=False)\n",
    "test.to_csv('test_mbcf_hundten.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "training = spark.read.csv('yelp_data/train_mbcf_hundten.csv', header=True, inferSchema=True)\n",
    "test = spark.read.csv('yelp_data/test_mbcf_hundten.csv', header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('shops', 'string'),\n",
       " ('ratings', 'double'),\n",
       " ('userids', 'int'),\n",
       " ('userids_index', 'double'),\n",
       " ('shops_index', 'double')]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2348"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "587"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('shops', 'string'),\n",
       " ('ratings', 'double'),\n",
       " ('userids', 'int'),\n",
       " ('userids_index', 'double'),\n",
       " ('shops_index', 'double')]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-----+\n",
      "|ratings|count|\n",
      "+-------+-----+\n",
      "|    1.0|    5|\n",
      "|    4.0| 1194|\n",
      "|    3.0|  176|\n",
      "|    2.0|   15|\n",
      "|    5.0|  958|\n",
      "+-------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#looking at the rating classes (target) in the training dataset to determine baseline accuracy...\n",
    "training.groupBy(\"ratings\").count().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- Baseline accuracy is $\\frac{1194\\ (majority\\ class)}{(5+1194+176+15+958)} = 0.51$ (rounded off to 2 decimal places)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiating the ALS model\n",
    "als = ALS(maxIter=20,userCol='userids_index',itemCol='shops_index', ratingCol='ratings',coldStartStrategy='drop')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiating the pipeline for tuning of ALS hyperparameters\n",
    "#pipeline = PL(stages=[als])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#setting up the paramGrid\n",
    "paramGrid = ParamGridBuilder() \\\n",
    "    .addGrid(als.rank, [200, 300, 400]) \\\n",
    "    .addGrid(als.regParam, [0.01, 0.1, 0.5]) \\\n",
    "    .build()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#setting up the CrossValidator object for hyperparameter tuning\n",
    "#crossval = CrossValidator(estimator=pipeline,\n",
    " #                         estimatorParamMaps=paramGrid,\n",
    "  #                        evaluator=RegressionEvaluator(predictionCol='prediction',labelCol='ratings',metricName='r2'),\n",
    "#                          numFolds=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#crossval1 = CrossValidator(estimator=ALS(maxIter=20,userCol='userids_index',itemCol='shops_index', ratingCol='ratings',coldStartStrategy='drop'),\n",
    " #                         estimatorParamMaps=paramGrid,\n",
    "  #                        evaluator=RegressionEvaluator(predictionCol='prediction',labelCol='ratings',metricName='r2'),\n",
    "   #                       numFolds=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#setting up the CrossValidator object for hyperparameter tuning\n",
    "crossval2 = CrossValidator(estimator=als,\n",
    "                          estimatorParamMaps=paramGrid,\n",
    "                          evaluator=RegressionEvaluator(predictionCol='prediction',labelCol='ratings',metricName='r2'),\n",
    "                          numFolds=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "#fitting/training on the train dataset. The following few cells or code have been commented out since this cell took a long time to tune...\n",
    "#this tuned model yielded best rank of 200 and best regParam of 0.1 as mentioned below...\n",
    "#cvModel = crossval.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cvModel1 = crossval1.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cvModel2 = crossval2.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{Param(parent='ALS_301793330495', name='rank', doc='rank of the factorization'): 300,\n",
       " Param(parent='ALS_301793330495', name='regParam', doc='regularization parameter (>= 0).'): 0.1}"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvModel2.getEstimatorParamMaps()[np.argmax(cvModel2.avgMetrics)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{Param(parent='ALS_301793330495', name='rank', doc='rank of the factorization'): 300,\n",
       " Param(parent='ALS_301793330495', name='regParam', doc='regularization parameter (>= 0).'): 0.1}"
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvModel2.getEstimatorParamMaps()[8]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [],
   "source": [
    "#saving model here\n",
    "cvModel2.save(\"als_hundten_try1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loading the saved model from above cell\n",
    "cvM1 = CrossValidatorModel.load(\"yelp_data/als_hundten_try1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{Param(parent='ALS_301793330495', name='rank', doc='rank of the factorization'): 300,\n",
       " Param(parent='ALS_301793330495', name='regParam', doc='regularization parameter (>= 0).'): 0.1}"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvModel2.getEstimatorParamMaps()[np.argmax(cvModel2.avgMetrics)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- It gets a little complicated when extracting the best tuned parameters from a pyspark object and thus some of the above code cells may not make sense or be a little messy (as I was experimenting with them) but suffice to note that the best rank and best regParam yielded from the CrossValidator above was 300 and 0.1 respectively...\n",
    "    \n",
    "    </ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "cvM1.getEstimatorParamMaps()[np.argmax(cvM1.avgMetrics)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#making predictions on the test set\n",
    "alspredictions = cvModel1.transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "#saving alspredictions to csv for retrieval to test prediction performance later on...\n",
    "alspredictions.toPandas().to_csv('yelp_data/als_predictions.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loading a prefit ALS model containing the best rank = 200 and best regParam = 0.1 derived from the crossval fitted cvModel 4 cells above. Unable to implement the code here due to saving it in the wrong format-and I don't wish to go through the above crossvalidator process again as it took close to a full day just to run it...I could not afford that time as there just wasn't enough time...\n",
    "#cvModel = ALS.load(\"yelp_data/als_rec_prefit.model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reading in alspredictions to test performance of ALS on test set.\n",
    "alspreds = spark.read.csv('yelp_data/als_predictions.csv', header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-------+-------+-----------+-------------+----------+\n",
      "|               shops|ratings|userids|shops_index|userids_index|prediction|\n",
      "+--------------------+-------+-------+-----------+-------------+----------+\n",
      "|the-coconut-club-...|    5.0|     75|      148.0|         12.0|   4.90392|\n",
      "|    wilder-singapore|    5.0|   2445|      463.0|         64.0|  4.907964|\n",
      "|the-daily-cut-sin...|    5.0|    225|      496.0|         43.0| 4.9131126|\n",
      "+--------------------+-------+-------+-----------+-------------+----------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#checking out the first few rows of the prediction df\n",
    "alspreds.show(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "#rounding the predictions to a whole number (just like the original or actual ratings) to prepare it for evaluation by MulticlassClassificationEvaluator using f1 score metric\n",
    "alspreds_df = alspreds.withColumn(\"prediction_rounded\", func.round(alspreds[\"prediction\"],0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-------+-------+-----------+-------------+----------+------------------+\n",
      "|               shops|ratings|userids|shops_index|userids_index|prediction|prediction_rounded|\n",
      "+--------------------+-------+-------+-----------+-------------+----------+------------------+\n",
      "|the-coconut-club-...|    5.0|     75|      148.0|         12.0|   4.90392|               5.0|\n",
      "|    wilder-singapore|    5.0|   2445|      463.0|         64.0|  4.907964|               5.0|\n",
      "|the-daily-cut-sin...|    5.0|    225|      496.0|         43.0| 4.9131126|               5.0|\n",
      "+--------------------+-------+-------+-----------+-------------+----------+------------------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#taking a look at the first few rows of the prediction df with the additional rounded prediction column...\n",
    "alspreds_df.show(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "497"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alspreds_df.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('shops', 'string'),\n",
       " ('ratings', 'double'),\n",
       " ('userids', 'int'),\n",
       " ('shops_index', 'double'),\n",
       " ('userids_index', 'double'),\n",
       " ('prediction', 'double'),\n",
       " ('prediction_rounded', 'double')]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the data types of the various columns\n",
    "alspreds_df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "#as MulticlassClassificationEvaluator works on double typed values, let's convert rounded predictions to double type.\n",
    "indexer_1 = StringIndexer(inputCol=\"prediction_rounded\", outputCol='prediction_rounded_dbl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "#implementing the conversion of rounded predictions to double type by fitting and transforming the test set\n",
    "alspreds_df = indexer_1.fit(alspreds_df).transform(alspreds_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------------------------------------------+-------+------------------+----------------------+\n",
      "|shops                                               |ratings|prediction_rounded|prediction_rounded_dbl|\n",
      "+----------------------------------------------------+-------+------------------+----------------------+\n",
      "|the-coconut-club-singapore                          |5.0    |5.0               |1.0                   |\n",
      "|wilder-singapore                                    |5.0    |5.0               |1.0                   |\n",
      "|the-daily-cut-singapore-3                           |5.0    |5.0               |1.0                   |\n",
      "|kim-kee-coffee-shop-tanjong-rhu-wanton-mee-singapore|4.0    |4.0               |0.0                   |\n",
      "|andiamo-gelateria-singapore                         |2.0    |2.0               |3.0                   |\n",
      "+----------------------------------------------------+-------+------------------+----------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#taking a look at the first few rows of the df with the newly added column\n",
    "alspreds_df.select(\"shops\",\"ratings\",\"prediction_rounded\",\"prediction_rounded_dbl\").show(5,False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "#in order for the double typed rounded predictions to match up with the actual ratings, \n",
    "#convert the actual ratings into double type as well for a fairer comparison\n",
    "indexer_2 = StringIndexer(inputCol=\"ratings\", outputCol='ratings_dbl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "#implementing the conversion\n",
    "alspreds_df = indexer_2.fit(alspreds_df).transform(alspreds_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------------------------------------------+-----------+------------------+----------------------+\n",
      "|shops                                               |ratings_dbl|prediction_rounded|prediction_rounded_dbl|\n",
      "+----------------------------------------------------+-----------+------------------+----------------------+\n",
      "|the-coconut-club-singapore                          |1.0        |5.0               |1.0                   |\n",
      "|wilder-singapore                                    |1.0        |5.0               |1.0                   |\n",
      "|the-daily-cut-singapore-3                           |1.0        |5.0               |1.0                   |\n",
      "|kim-kee-coffee-shop-tanjong-rhu-wanton-mee-singapore|0.0        |4.0               |0.0                   |\n",
      "|andiamo-gelateria-singapore                         |3.0        |2.0               |3.0                   |\n",
      "+----------------------------------------------------+-----------+------------------+----------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#checking out the first few rows of the prediction df again\n",
    "alspreds_df.select(\"shops\",\"ratings_dbl\",\"prediction_rounded\",\"prediction_rounded_dbl\").show(5,False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiating the MulticlassClassificationEvaluator with accuracy score as the metric to compare against baseline accuracy first. \n",
    "evaluator_b = MulticlassClassificationEvaluator(predictionCol='prediction_rounded_dbl',\n",
    "                                                labelCol='ratings_dbl',metricName='accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiating the MulticlassClassificationEvaluator with f1 score as the metric of choice. \n",
    "#F1 score was chosen as both precision and recall are important here-we wouldn't want to miss out \n",
    "#on good recommendations and neither do we want to recommend something that shouldn't have been recommended.\n",
    "evaluator_c = MulticlassClassificationEvaluator(predictionCol='prediction_rounded_dbl',\n",
    "                                                labelCol='ratings_dbl',metricName='f1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy is  1.0\n"
     ]
    }
   ],
   "source": [
    "#accuracy is 1.0!\n",
    "print(\"Accuracy is \", evaluator_b.evaluate(alspreds_df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score for rating predictions by tuned ALS model-based collaborative filtering:  1.0\n"
     ]
    }
   ],
   "source": [
    "print(\"F1 score for rating predictions by tuned ALS model-based collaborative filtering: \",evaluator_c.evaluate(alspreds_df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- The tuned ALS model performed really well with a high $F_1$ score of 1.0!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "#saving alspreds_df with selected columns for reading in again for other evaluation below\n",
    "alspreds_df.select(\"shops\",\"ratings\",\"prediction_rounded\", \"prediction\",\"userids\").toPandas().to_csv(\"yelp_data/mbcf_ALS.csv\",index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>ratings</th>\n",
       "      <th>prediction_rounded</th>\n",
       "      <th>prediction</th>\n",
       "      <th>userids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>the-coconut-club-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.903920</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>wilder-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.907964</td>\n",
       "      <td>2445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>the-daily-cut-singapore-3</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.913113</td>\n",
       "      <td>225</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        shops  ratings  prediction_rounded  prediction  \\\n",
       "0  the-coconut-club-singapore      5.0                 5.0    4.903920   \n",
       "1            wilder-singapore      5.0                 5.0    4.907964   \n",
       "2   the-daily-cut-singapore-3      5.0                 5.0    4.913113   \n",
       "\n",
       "   userids  \n",
       "0       75  \n",
       "1     2445  \n",
       "2      225  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#reading in mbcf_ALS for other evaluation\n",
    "mbcf_als_2 = pd.read_csv(\"yelp_data/mbcf_ALS.csv\")\n",
    "mbcf_als_2.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.0    253\n",
       "5.0    206\n",
       "3.0     35\n",
       "2.0      2\n",
       "1.0      1\n",
       "Name: prediction_rounded, dtype: int64"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the value_counts of rounded predictions\n",
    "mbcf_als_2['prediction_rounded'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.0    253\n",
       "5.0    206\n",
       "3.0     35\n",
       "2.0      2\n",
       "1.0      1\n",
       "Name: ratings, dtype: int64"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking out the value_counts of actual ratings\n",
    "mbcf_als_2['ratings'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "#saving as csv and then re-reading it to use spark evaluator to recalculate the F1 score...\n",
    "#mbcf_als.to_csv('yelp_data/mbcf_ALS_mod.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reading in the modified \"mbcf_ALS\" csv for evaluation of F1 using multiclassclassificationevaluator\n",
    "#df2_mod = spark.read.csv('yelp_data/mbcf_ALS_mod.csv', header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "#in order for the modified prediction_rounded and actual ratings to be recognized by the multiclassclassification evaluator, \n",
    "#convert the both columns into double type again\n",
    "#indexer_3 = StringIndexer(inputCol=\"ratings\", outputCol='ratings_double_pre')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "#implementing the conversion\n",
    "#df3_mod = indexer_3.fit(df2_mod).transform(df2_mod)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiating the indexer 4 to convert prediction_rounded column to double precision\n",
    "#indexer_4 = StringIndexer(inputCol=\"prediction_rounded\", outputCol='prediction_rounded_double_pre')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "#implementing the conversion\n",
    "#df4_mod = indexer_4.fit(df3_mod).transform(df3_mod)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiating the MulticlassClassificationEvaluator with accuracy score to compare against baseline accuracy first- see if correcting some of the incorrect rating classes have changed the accuracy score\n",
    "#evaluator_c_1 = MulticlassClassificationEvaluator(predictionCol='prediction_rounded_double_pre',\n",
    "                                               # labelCol='ratings_double_pre',metricName='accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiating the MulticlassClassificationEvaluator with f1 score as the metric of choice. \n",
    "#F1 score was chosen as both precision and recall are important here-we wouldn't want to miss out \n",
    "#on good recommendations and neither do we want to recommend something that shouldn't have been recommended.\n",
    "#evaluator_d = MulticlassClassificationEvaluator(predictionCol='prediction_rounded_double_pre',\n",
    "                                            #    labelCol='ratings_double_pre',metricName='f1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy is  0.9685916919959473\n"
     ]
    }
   ],
   "source": [
    "#accuracy after amendment of rating class encoding is still 0.97!\n",
    "#print(\"Accuracy is \", evaluator_c_1.evaluate(df4_mod))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score for slightly-modified rating predictions by tuned ALS model-based collaborative filtering:  0.9820064562401125\n"
     ]
    }
   ],
   "source": [
    "#F1 score remained unchanged!\n",
    "#print(\"F1 score for slightly-modified rating predictions by tuned ALS model-based collaborative filtering: \",evaluator_d.evaluate(df4_mod))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[496,   0],\n",
       "        [  0,   1]],\n",
       "\n",
       "       [[495,   0],\n",
       "        [  0,   2]],\n",
       "\n",
       "       [[462,   0],\n",
       "        [  0,  35]],\n",
       "\n",
       "       [[244,   0],\n",
       "        [  0, 253]],\n",
       "\n",
       "       [[291,   0],\n",
       "        [  0, 206]]])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#having a rough look at the confusion matrix for model-based collaborative filtering via ALS...\n",
    "multilabel_confusion_matrix(mbcf_als_2.ratings,mbcf_als_2.prediction_rounded)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         1.0       1.00      1.00      1.00         1\n",
      "         2.0       1.00      1.00      1.00         2\n",
      "         3.0       1.00      1.00      1.00        35\n",
      "         4.0       1.00      1.00      1.00       253\n",
      "         5.0       1.00      1.00      1.00       206\n",
      "\n",
      "    accuracy                           1.00       497\n",
      "   macro avg       1.00      1.00      1.00       497\n",
      "weighted avg       1.00      1.00      1.00       497\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#apart from rating class 1.0, the ALS model seemed to be able to predict the rating classes well!\n",
    "print(classification_report(mbcf_als_2.ratings,mbcf_als_2.prediction_rounded))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining functions for evaluation of model (confusion matrix, micro-average precision, recall)\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "#defining function for obtaining tn, fp, fn, tp for each rating class for feeding into micro-avg precision and recall functions defined below\n",
    "def cm_spec(y_true,y_pred,rating,state):\n",
    "    if state=='tn':\n",
    "        return multilabel_confusion_matrix(y_true,y_pred)[rating-1][0][0]\n",
    "    elif state=='fp':\n",
    "        return multilabel_confusion_matrix(y_true,y_pred)[rating-1][0][1]\n",
    "    elif state=='fn':\n",
    "        return multilabel_confusion_matrix(y_true,y_pred)[rating-1][1][0]\n",
    "    else:\n",
    "        return multilabel_confusion_matrix(y_true,y_pred)[rating-1][1][1]\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "#defining function for obtaining micro-avg precision\n",
    "def micro_avg_precision(y_true,y_pred):\n",
    "    return ((cm_spec(y_true,y_pred,1,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,2,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,3,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,4,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,5,'tp'))/(\n",
    "                                                cm_spec(y_true,y_pred,1,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,2,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,3,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,4,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,5,'tp')+\n",
    "                                                cm_spec(y_true,y_pred,1,'fp')+\n",
    "                                                 cm_spec(y_true,y_pred,2,'fp')+\n",
    "                                                 cm_spec(y_true,y_pred,3,'fp')+\n",
    "                                                 cm_spec(y_true,y_pred,4,'fp')+\n",
    "                                                 cm_spec(y_true,y_pred,5,'fp')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "#defining function for obtaining micro-avg recall\n",
    "def micro_avg_recall(y_true,y_pred):\n",
    "    return ((cm_spec(y_true,y_pred,1,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,2,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,3,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,4,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,5,'tp'))/(\n",
    "                                                cm_spec(y_true,y_pred,1,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,2,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,3,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,4,'tp')+\n",
    "                                                 cm_spec(y_true,y_pred,5,'tp')+\n",
    "                                                cm_spec(y_true,y_pred,1,'fn')+\n",
    "                                                 cm_spec(y_true,y_pred,2,'fn')+\n",
    "                                                 cm_spec(y_true,y_pred,3,'fn')+\n",
    "                                                 cm_spec(y_true,y_pred,4,'fn')+\n",
    "                                                 cm_spec(y_true,y_pred,5,'fn')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "#defining function for obtaining micro_avg_f1\n",
    "def micro_avg_f1(y_true,y_pred):\n",
    "    return 2 * ((micro_avg_precision(y_true,y_pred) * micro_avg_recall(y_true,y_pred))/(micro_avg_precision(y_true,y_pred) + micro_avg_recall(y_true,y_pred)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "#function to print out confusion matrix breakdown for each rating class\n",
    "def confusion_breakdown(y_true,y_pred,rating):\n",
    "    print(\"True negatives for rating {}: {}\".format(\n",
    "        rating,multilabel_confusion_matrix(y_true,y_pred)[rating-1][0][0]))\n",
    "    print(\"False positives for rating {}: {}\".format(\n",
    "        rating,multilabel_confusion_matrix(y_true,y_pred)[rating-1][0][1]))\n",
    "    print(\"False negatives for rating {}: {}\".format(\n",
    "        rating,multilabel_confusion_matrix(y_true,y_pred)[rating-1][1][0]))\n",
    "    print(\"True positives for rating {}: {}\".format(\n",
    "        rating,multilabel_confusion_matrix(y_true,y_pred)[rating-1][1][1]))\n",
    "    return \"******************************************\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True negatives for rating 1: 496\n",
      "False positives for rating 1: 0\n",
      "False negatives for rating 1: 0\n",
      "True positives for rating 1: 1\n",
      "******************************************\n",
      "True negatives for rating 2: 495\n",
      "False positives for rating 2: 0\n",
      "False negatives for rating 2: 0\n",
      "True positives for rating 2: 2\n",
      "******************************************\n",
      "True negatives for rating 3: 462\n",
      "False positives for rating 3: 0\n",
      "False negatives for rating 3: 0\n",
      "True positives for rating 3: 35\n",
      "******************************************\n",
      "True negatives for rating 4: 244\n",
      "False positives for rating 4: 0\n",
      "False negatives for rating 4: 0\n",
      "True positives for rating 4: 253\n",
      "******************************************\n",
      "True negatives for rating 5: 291\n",
      "False positives for rating 5: 0\n",
      "False negatives for rating 5: 0\n",
      "True positives for rating 5: 206\n",
      "******************************************\n"
     ]
    }
   ],
   "source": [
    "print(confusion_breakdown(mbcf_als_2.ratings,mbcf_als_2.prediction_rounded,1))\n",
    "print(confusion_breakdown(mbcf_als_2.ratings,mbcf_als_2.prediction_rounded,2))\n",
    "print(confusion_breakdown(mbcf_als_2.ratings,mbcf_als_2.prediction_rounded,3))\n",
    "print(confusion_breakdown(mbcf_als_2.ratings,mbcf_als_2.prediction_rounded,4))\n",
    "print(confusion_breakdown(mbcf_als_2.ratings,mbcf_als_2.prediction_rounded,5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.0    253\n",
       "5.0    206\n",
       "3.0     35\n",
       "2.0      2\n",
       "1.0      1\n",
       "Name: ratings, dtype: int64"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#all correct!\n",
    "mbcf_als_2.ratings.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuned ALS yielded micro_avg_precision of  1.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Tuned ALS yielded micro_avg_precision of \", micro_avg_precision(mbcf_als_2.ratings,mbcf_als_2.prediction_rounded))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuned ALS yielded micro_avg_recall of  1.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Tuned ALS yielded micro_avg_recall of \", micro_avg_recall(mbcf_als_2.ratings,mbcf_als_2.prediction_rounded))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuned ALS yielded micro_avg_f1 of  1.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Tuned ALS yielded micro_avg_f1 of \", micro_avg_f1(mbcf_als_2.ratings,mbcf_als_2.prediction_rounded))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- Seems like the $F_1$ score computed outside of spark for ALS is also around 1.0! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shops</th>\n",
       "      <th>ratings</th>\n",
       "      <th>prediction_rounded</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>the-bao-makers-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.933102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>little-farms-cafe-singapore</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.895577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>lam-yeo-coffee-powder-fty-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.869991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>stateland-cafe-singapore</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.882576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>coffee-club-singapore-5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.850962</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  shops  ratings  prediction_rounded  \\\n",
       "0              the-bao-makers-singapore      4.0                 4.0   \n",
       "2           little-farms-cafe-singapore      4.0                 4.0   \n",
       "3   lam-yeo-coffee-powder-fty-singapore      5.0                 5.0   \n",
       "17             stateland-cafe-singapore      5.0                 5.0   \n",
       "19              coffee-club-singapore-5      5.0                 5.0   \n",
       "\n",
       "    prediction  \n",
       "0     3.933102  \n",
       "2     3.895577  \n",
       "3     4.869991  \n",
       "17    4.882576  \n",
       "19    4.850962  "
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#filtering out only userid2043 for comparison with content-based filtering later on\n",
    "#userid2043_mbcf_test_rev = mbcf_als_final_corrected[mbcf_als_final_corrected['userids']==2043][['shops','ratings','prediction_rounded','prediction']]\n",
    "#userid2043_mbcf_test_rev.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(185, 4)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#userid2043_mbcf_test_rev.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "#saving this df to csv for easier retrieval later on for comparison and fusion with content-based filtering's rating predictions\n",
    "mbcf_als_2.to_csv('yelp_data/mbcf_hundten_pred_actual_testeval.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "#old cells, kept for ref only\n",
    "#df2.show(5,False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#old cells, kept for ref only..\n",
    "#having a look at how some of the predictions compare with the actual ratings\n",
    "#df2.filter(\"userids = 2043\").select(\"shops\",\"ratings\",\"prediction_rounded\",\"prediction\").sort(\"prediction\", ascending=False).show(5, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "#old cells, kept for ref only...\n",
    "#filtering out only userid2043 for comparison with content-based filtering later on, converting to pandas df for easier manipulation and merging with content-based rating predictions\n",
    "#userid2043_mbcf_test = df2.filter(\"userids = 2043\").select(\"shops\",\"ratings\",\"prediction_rounded\",\"prediction\").toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "#old cells, kept for ref only...\n",
    "#taking a look at the first few rows of the prediction vs actual ratings (\"ratings\") for ALS tuned with CrossValidator\n",
    "#userid2043_mbcf_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "#old cells, kept for ref only...\n",
    "#userid2043_mbcf_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "#saving this df to csv for easier retrieval later on for comparison and fusion with content-based filtering's rating predictions\n",
    "#userid2043_mbcf_test.to_csv('yelp_data/userid2043_mbcf_pred_actual.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "#old method that would have worked had I saved the cvModel up above correctly...\n",
    "#extracting the index of the best params\n",
    "#best_model_params_idx = np.argmax(cvModel.avgMetrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "#old method that would have worked, see above cell\n",
    "#extracting the tuned hyperparameters of the best model\n",
    "#best_rank = list(cvModel.getEstimatorParamMaps()[best_model_params_idx].values())[0]\n",
    "#best_regParam = list(cvModel.getEstimatorParamMaps()[best_model_params_idx].values())[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- The best rank is 200 while best regParam is 0.1. Fed these into the ALS instantiation below and saved it for future use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "#instantiating ALS with the tuned hyperparameters of the best model\n",
    "#als_rec = ALS(rank=best_rank,regParam=best_regParam,\n",
    "#              maxIter=20,userCol='userids_index',\n",
    "#              itemCol='shops_index', ratingCol='ratings',\n",
    "#              coldStartStrategy='drop')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "#dumping the instantiated model above for later use in another jupyter notebook\n",
    "#als_rec.save(\"yelp_data/als_rec_prefitted.model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Summary and Result Interpretation\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "    \n",
    "- Except for rating 1 which contains a number of false positives, the tuned model-based ALS was able to predict the rating classes well!\n",
    "- Accuracy of 0.97, Micro-Averaged precision of 0.97, Micro-Averaged recall of 0.97, Micro-Averaged $F_1$ of 0.97"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Source(s)\n",
    "---\n",
    "\n",
    "- https://java.com/en/download/help/download_options.xml\n",
    "- https://www.oracle.com/java/technologies/javase-jdk8-downloads.html\n",
    "- https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz\n",
    "- https://github.com/sbt/sbt/releases/download/v0.13.17/sbt-0.13.17.tgz\n",
    "- https://spark.apache.org/downloads.html\n",
    "- https://medium.com/luckspark/installing-spark-2-3-0-on-macos-high-sierra-276a127b8b85"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}