recommendation_system.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Business context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Online streaming platforms like Netflix have plenty of movies in their repositories and if we can build a recommendation system to recommend relevant movies to the users based on their historical interactions, this would improve customer satisfaction and hence improve revenue. The techniques that we will learn here will not only be limited to movies, it can be any item for which you want to build a recommendation system. For this project, you can find the dataset [here](https://www.kaggle.com/somnambwl/bookcrossing-dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1 - Understanding collaborative filtering and dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](collaborative_filtering.PNG \"User Item Interactions\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this type of recommendation system, `we do not need any information` about the users or items. We only need user item interaction data to build a collaborative recommendation system. For example - \n",
    "<ol>\n",
    "    <li><b>Ratings</b> provided by users. For example - ratings of books on goodread, movie ratings on imdb etc</li>\n",
    "    <li><b>Likes</b> of users on different facebook posts, likes on youtube videos</li>\n",
    "    <li><b>Use/buying</b> of a product by users. For example - buying different items on e-commerce sites</li>\n",
    "    <li><b>Reading</b> of articles by readers on various blogs</li>\n",
    "</ol>    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import dataset\n",
    "\n",
    "movies = pd.read_csv('ratings_small.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1029</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1061</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1129</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1172</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759205</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating   timestamp\n",
       "0       1       31     2.5  1260759144\n",
       "1       1     1029     3.0  1260759179\n",
       "2       1     1061     3.0  1260759182\n",
       "3       1     1129     2.0  1260759185\n",
       "4       1     1172     4.0  1260759205"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# observe the head of the dataset\n",
    "\n",
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "in this dataset, the variable `timestamp` is not helpful to build this recommendation system, so we can delete this variable from our dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove the variable timestamp from the dataset\n",
    "\n",
    "movies = movies.drop(['timestamp'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2: Exploring the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100004, 3)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# see the shape of the dataset\n",
    "\n",
    "movies.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 100004 entries, 0 to 100003\n",
      "Data columns (total 3 columns):\n",
      " #   Column   Non-Null Count   Dtype  \n",
      "---  ------   --------------   -----  \n",
      " 0   userId   100004 non-null  int64  \n",
      " 1   movieId  100004 non-null  int64  \n",
      " 2   rating   100004 non-null  float64\n",
      "dtypes: float64(1), int64(2)\n",
      "memory usage: 2.3 MB\n"
     ]
    }
   ],
   "source": [
    "# check for missing values\n",
    "\n",
    "movies.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "671"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# total number of unique users\n",
    "\n",
    "movies['userId'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9066"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# total number of unique movies\n",
    "\n",
    "movies['movieId'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "547    2391\n",
       "564    1868\n",
       "624    1735\n",
       "15     1700\n",
       "73     1610\n",
       "       ... \n",
       "221      20\n",
       "444      20\n",
       "484      20\n",
       "35       20\n",
       "485      20\n",
       "Name: userId, Length: 671, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# which user has rated the most\n",
    "\n",
    "movies['userId'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([408., 111.,  55.,  34.,  14.,  12.,   7.,   5.,   8.,   1.,   2.,\n",
       "          4.,   2.,   1.,   0.,   0.,   2.,   0.,   0.,   0.,   1.,   2.,\n",
       "          0.,   1.,   0.,   0.,   0.,   0.,   0.,   1.]),\n",
       " array([  20.        ,   99.03333333,  178.06666667,  257.1       ,\n",
       "         336.13333333,  415.16666667,  494.2       ,  573.23333333,\n",
       "         652.26666667,  731.3       ,  810.33333333,  889.36666667,\n",
       "         968.4       , 1047.43333333, 1126.46666667, 1205.5       ,\n",
       "        1284.53333333, 1363.56666667, 1442.6       , 1521.63333333,\n",
       "        1600.66666667, 1679.7       , 1758.73333333, 1837.76666667,\n",
       "        1916.8       , 1995.83333333, 2074.86666667, 2153.9       ,\n",
       "        2232.93333333, 2311.96666667, 2391.        ]),\n",
       " <a list of 30 Patch objects>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAASJklEQVR4nO3dXYxcZ33H8e8PEwziRSTNJjK2JRtkqjpIGLpyqVIhSigxSVUHqamM1NQXqcyFI4FE1dpwQbiwlFYE2qpNJIdEuJTiWgIUi5cW44IQEorZpM6LY9wsjZsstuwFigg3bu38e7HHZXDW3tmdGdvr5/uRRuecZ55z5v8cnfXP58yZmVQVkqR2vexSFyBJurQMAklqnEEgSY0zCCSpcQaBJDXu5Ze6AIBrr722Vq1adanLkKRF5dFHH/1xVY0Nup3LIghWrVrFxMTEpS5DkhaVJP81jO14aUiSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhp3WXyyeFCrtn21r35H77l1xJVI0uLT9xlBkiVJ/j3JV7rla5LsS/JMN726p+/2JJNJjiS5eRSFS5KGYz6Xhj4EHO5Z3gbsr6o1wP5umSRrgU3ADcAG4L4kS4ZTriRp2PoKgiQrgFuBz/Q0bwR2dfO7gNt62ndX1amqehaYBNYPp1xJ0rD1e0bw18CfAy/2tF1fVccBuul1Xfty4PmeflNd269IsiXJRJKJ6enpeRcuSRqOOYMgye8DJ6vq0T63mVna6iUNVTuraryqxsfGBv46bUnSAvVz19CNwB8kuQV4JfC6JP8InEiyrKqOJ1kGnOz6TwEre9ZfARwbZtGSpOGZ84ygqrZX1YqqWsXMm8D/VlV/DOwFNnfdNgMPd/N7gU1JliZZDawBDgy9cknSUAzyOYJ7gD1J7gSeA24HqKpDSfYATwOnga1VdWbgSiVJIzGvIKiqbwPf7uZ/Atx0nn47gB0D1iZJugj8iglJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuP6+fH6VyY5kOTxJIeSfKJrvzvJj5Ic7B639KyzPclkkiNJbh7lACRJg+nnF8pOAe+uql8kuQr4bpKvd899uqo+2ds5yVpmftv4BuANwDeTvNmfq5Sky1M/P15fVfWLbvGq7lEXWGUjsLuqTlXVs8AksH7gSiVJI9HXewRJliQ5CJwE9lXVI91TdyV5IslDSa7u2pYDz/esPtW1nbvNLUkmkkxMT08PMARJ0iD6CoKqOlNV64AVwPokbwHuB94ErAOOA/d23TPbJmbZ5s6qGq+q8bGxsQUVL0ka3LzuGqqqnwHfBjZU1YkuIF4EHuCXl3+mgJU9q60Ajg2hVknSCPRz19BYktd3868C3gP8IMmynm7vB57q5vcCm5IsTbIaWAMcGG7ZkqRh6eeuoWXAriRLmAmOPVX1lSSfS7KOmcs+R4EPAlTVoSR7gKeB08BW7xiSpMvXnEFQVU8Ab5ul/Y4LrLMD2DFYaZKki8FPFktS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGtfPT1W+MsmBJI8nOZTkE137NUn2JXmmm17ds872JJNJjiS5eZQDkCQNpp8zglPAu6vqrcA6YEOSdwDbgP1VtQbY3y2TZC2wCbgB2ADc1/3MpSTpMjRnENSMX3SLV3WPAjYCu7r2XcBt3fxGYHdVnaqqZ4FJYP1Qq5YkDU1f7xEkWZLkIHAS2FdVjwDXV9VxgG56Xdd9OfB8z+pTXZsk6TLUVxBU1ZmqWgesANYnecsFume2TbykU7IlyUSSienp6f6qlSQN3bzuGqqqnwHfZuba/4kkywC66cmu2xSwsme1FcCxWba1s6rGq2p8bGxsAaVLkoahn7uGxpK8vpt/FfAe4AfAXmBz120z8HA3vxfYlGRpktXAGuDAsAuXJA3Hy/voswzY1d358zJgT1V9Jcn3gD1J7gSeA24HqKpDSfYATwOnga1VdWY05UuSBjVnEFTVE8DbZmn/CXDTedbZAewYuDpJ0sj5yWJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqXD+/WbwyybeSHE5yKMmHuva7k/woycHucUvPOtuTTCY5kuTmUQ5AkjSYfn6z+DTwkap6LMlrgUeT7Oue+3RVfbK3c5K1wCbgBuANwDeTvNnfLZaky9OcZwRVdbyqHuvmXwAOA8svsMpGYHdVnaqqZ4FJYP0wipUkDd+83iNIsoqZH7J/pGu6K8kTSR5KcnXXthx4vme1KWYJjiRbkkwkmZienp534ZKk4eg7CJK8Bvgi8OGq+jlwP/AmYB1wHLj3bNdZVq+XNFTtrKrxqhofGxubd+GSpOHoKwiSXMVMCHy+qr4EUFUnqupMVb0IPMAvL/9MASt7Vl8BHBteyZKkYernrqEADwKHq+pTPe3Lerq9H3iqm98LbEqyNMlqYA1wYHglS5KGqZ+7hm4E7gCeTHKwa/so8IEk65i57HMU+CBAVR1Ksgd4mpk7jrZ6x5AkXb7mDIKq+i6zX/f/2gXW2QHsGKAuSdJF4ieLJalxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmN6+enKlcm+VaSw0kOJflQ135Nkn1JnummV/essz3JZJIjSW4e5QAkSYPp54zgNPCRqvoN4B3A1iRrgW3A/qpaA+zvlume2wTcAGwA7kuyZBTFS5IGN2cQVNXxqnqsm38BOAwsBzYCu7puu4DbuvmNwO6qOlVVzwKTwPphFy5JGo55vUeQZBXwNuAR4PqqOg4zYQFc13VbDjzfs9pU13butrYkmUgyMT09Pf/KJUlD0XcQJHkN8EXgw1X18wt1naWtXtJQtbOqxqtqfGxsrN8yJElD1lcQJLmKmRD4fFV9qWs+kWRZ9/wy4GTXPgWs7Fl9BXBsOOVKkoatn7uGAjwIHK6qT/U8tRfY3M1vBh7uad+UZGmS1cAa4MDwSpYkDdPL++hzI3AH8GSSg13bR4F7gD1J7gSeA24HqKpDSfYATzNzx9HWqjoz9MolSUMxZxBU1XeZ/bo/wE3nWWcHsGOAuiRJF4mfLJakxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTG9fObxQ8lOZnkqZ62u5P8KMnB7nFLz3Pbk0wmOZLk5lEVLkkajn7OCD4LbJil/dNVta57fA0gyVpgE3BDt859SZYMq1hJ0vDNGQRV9R3gp31ubyOwu6pOVdWzwCSwfoD6JEkjNsh7BHcleaK7dHR117YceL6nz1TX9hJJtiSZSDIxPT09QBmSpEEsNAjuB94ErAOOA/d27Zmlb822garaWVXjVTU+Nja2wDIkSYNaUBBU1YmqOlNVLwIP8MvLP1PAyp6uK4Bjg5UoSRqlBQVBkmU9i+8Hzt5RtBfYlGRpktXAGuDAYCVKkkbp5XN1SPIF4F3AtUmmgI8D70qyjpnLPkeBDwJU1aEke4CngdPA1qo6M5rSJUnDMGcQVNUHZml+8AL9dwA7BilKknTx+MliSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxs357aNXklXbvtpXv6P33DriSiTp8uEZgSQ1ziCQpMbNGQRJHkpyMslTPW3XJNmX5JluenXPc9uTTCY5kuTmURUuSRqOfs4IPgtsOKdtG7C/qtYA+7tlkqwFNgE3dOvcl2TJ0KqVJA3dnEFQVd8BfnpO80ZgVze/C7itp313VZ2qqmeBSWD9kGqVJI3AQt8juL6qjgN00+u69uXA8z39prq2l0iyJclEkonp6ekFliFJGtSw3yzOLG01W8eq2llV41U1PjY2NuQyJEn9WmgQnEiyDKCbnuzap4CVPf1WAMcWXp4kadQWGgR7gc3d/Gbg4Z72TUmWJlkNrAEODFaiJGmU5vxkcZIvAO8Crk0yBXwcuAfYk+RO4DngdoCqOpRkD/A0cBrYWlVnRlS7JGkI5gyCqvrAeZ666Tz9dwA7BilKknTx+MliSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLj5vxhmgtJchR4ATgDnK6q8STXAP8MrAKOAn9UVf89WJmSpFEZKAg6v1tVP+5Z3gbsr6p7kmzrlv9iCK9z0aza9tW++h2959YRVyJJozeKS0MbgV3d/C7gthG8hiRpSAYNggK+keTRJFu6tuur6jhAN71uthWTbEkykWRienp6wDIkSQs16KWhG6vqWJLrgH1JftDvilW1E9gJMD4+XgPWIUlaoIHOCKrqWDc9CXwZWA+cSLIMoJueHLRISdLoLDgIkrw6yWvPzgPvBZ4C9gKbu26bgYcHLVKSNDqDXBq6HvhykrPb+aeq+pck3wf2JLkTeA64ffAyJUmjsuAgqKr/BN46S/tPgJsGKUqSdPH4yWJJapxBIEmNMwgkqXHD+IqJZvlVFJKuBJ4RSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4/xk8UXgJ5AlXc4MgsuIgSHpUvDSkCQ1zjOCRajfMwfo/+zBsxGpXSMLgiQbgL8BlgCfqap7RvVaOr/5hIakNo0kCJIsAf4e+D1gCvh+kr1V9fQoXk+Ll2ci0qU3qjOC9cBk97vGJNkNbAQMgkXuUp1hjOJ1hx0ul3uoXcp9eLnvm1FYTGNOVQ1/o8kfAhuq6k+75TuA36qqu3r6bAG2dIu/DhyZ58tcC/x4COUuVo6/7fGD+8Dxw6uramzQDY3qjCCztP1K4lTVTmDngl8gmaiq8YWuv9g5/rbHD+4Dx5+Jqlo1jG2N6vbRKWBlz/IK4NiIXkuSNIBRBcH3gTVJVid5BbAJ2Dui15IkDWAkl4aq6nSSu4B/Zeb20Yeq6tCQX2bBl5WuEI5fre8Dxz8kI3mzWJK0ePgVE5LUOINAkhq36IIgyYYkR5JMJtl2qesZlSRHkzyZ5GCSia7tmiT7kjzTTa/u6b+92ydHktx86SpfuCQPJTmZ5KmetnmPOclvdvtuMsnfJpntdubLznnGf3eSH3XHwcEkt/Q8d6WNf2WSbyU5nORQkg917U0cAxcY/+iPgapaNA9m3nj+IfBG4BXA48DaS13XiMZ6FLj2nLa/ArZ189uAv+zm13b7YimwuttHSy71GBYw5ncCbweeGmTMwAHgt5n5PMvXgfdd6rENMP67gT+bpe+VOP5lwNu7+dcC/9GNs4lj4ALjH/kxsNjOCP7/qyuq6n+As19d0YqNwK5ufhdwW0/77qo6VVXPApPM7KtFpaq+A/z0nOZ5jTnJMuB1VfW9mvmL+IeedS5r5xn/+VyJ4z9eVY918y8Ah4HlNHIMXGD85zO08S+2IFgOPN+zPMWFd9RiVsA3kjzafR0HwPVVdRxmDhrguq79St4v8x3z8m7+3PbF7K4kT3SXjs5eFrmix59kFfA24BEaPAbOGT+M+BhYbEEw51dXXEFurKq3A+8DtiZ55wX6trRfzjrfmK+0fXE/8CZgHXAcuLdrv2LHn+Q1wBeBD1fVzy/UdZa2Rb8PZhn/yI+BxRYEzXx1RVUd66YngS8zc6nnRHfaRzc92XW/kvfLfMc81c2f274oVdWJqjpTVS8CD/DLS35X5PiTXMXMP4Kfr6ovdc3NHAOzjf9iHAOLLQia+OqKJK9O8tqz88B7gaeYGevmrttm4OFufi+wKcnSJKuBNcy8WXQlmNeYu0sHLyR5R3enxJ/0rLPonP0HsPN+Zo4DuALH39X7IHC4qj7V81QTx8D5xn9RjoFL/U75At5Zv4WZd9N/CHzsUtczojG+kZm7AR4HDp0dJ/BrwH7gmW56Tc86H+v2yREWwR0S5xn3F5g59f1fZv5Xc+dCxgyMd38sPwT+ju4T9Jf74zzj/xzwJPBE94e/7Aoe/+8wcwnjCeBg97illWPgAuMf+THgV0xIUuMW26UhSdKQGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcf8HpNHdcud+sfEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# create a histogram of all the interactions by all the users present in the dataset\n",
    "\n",
    "interaction = movies.groupby('userId').count()['movieId']\n",
    "plt.hist(interaction, bins=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 3: Creating user item matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating user item interactions matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the user item matrix using the ratings dataset - Hint: try using pivot function \n",
    "\n",
    "interaction_matrix = movies.pivot(index='userId', columns='movieId', values='rating')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(671, 9066)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the shape of the matrix\n",
    "\n",
    "interaction_matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>movieId</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>...</th>\n",
       "      <th>161084</th>\n",
       "      <th>161155</th>\n",
       "      <th>161594</th>\n",
       "      <th>161830</th>\n",
       "      <th>161918</th>\n",
       "      <th>161944</th>\n",
       "      <th>162376</th>\n",
       "      <th>162542</th>\n",
       "      <th>162672</th>\n",
       "      <th>163949</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>userId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 9066 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "movieId  1       2       3       4       5       6       7       8       \\\n",
       "userId                                                                    \n",
       "1           NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN   \n",
       "2           NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN   \n",
       "3           NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN   \n",
       "4           NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN   \n",
       "5           NaN     NaN     4.0     NaN     NaN     NaN     NaN     NaN   \n",
       "\n",
       "movieId  9       10      ...  161084  161155  161594  161830  161918  161944  \\\n",
       "userId                   ...                                                   \n",
       "1           NaN     NaN  ...     NaN     NaN     NaN     NaN     NaN     NaN   \n",
       "2           NaN     4.0  ...     NaN     NaN     NaN     NaN     NaN     NaN   \n",
       "3           NaN     NaN  ...     NaN     NaN     NaN     NaN     NaN     NaN   \n",
       "4           NaN     4.0  ...     NaN     NaN     NaN     NaN     NaN     NaN   \n",
       "5           NaN     NaN  ...     NaN     NaN     NaN     NaN     NaN     NaN   \n",
       "\n",
       "movieId  162376  162542  162672  163949  \n",
       "userId                                   \n",
       "1           NaN     NaN     NaN     NaN  \n",
       "2           NaN     NaN     NaN     NaN  \n",
       "3           NaN     NaN     NaN     NaN  \n",
       "4           NaN     NaN     NaN     NaN  \n",
       "5           NaN     NaN     NaN     NaN  \n",
       "\n",
       "[5 rows x 9066 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check head of the matrix and see how it looks like\n",
    "\n",
    "interaction_matrix.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace all the missing values with zero\n",
    "\n",
    "interaction_matrix = interaction_matrix.fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 4: Finding similar users"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](cosine.PNG \"Cosine Similarity\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import cosine_similarity from sklearn\n",
    "\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "def similar_users(user_id, interaction_matrix):\n",
    "    \n",
    "    # compute similarity of each user to the provided user\n",
    "    similatity = []\n",
    "    for user in range(1, interaction_matrix.shape[0]+1):\n",
    "        sim = cosine_similarity([interaction_matrix.loc[user_id]], [interaction_matrix.loc[user]])\n",
    "        similatity.append((user, sim))\n",
    "    \n",
    "    # sort by similarity\n",
    "    similatity.sort(key=lambda x: x[1], reverse=True)\n",
    "    \n",
    "    # create list of just the user ids\n",
    "    most_similar_user = [tup[0] for tup in similatity]\n",
    "    \n",
    "    # create list of similarity score\n",
    "    similatity_score = [tup[1] for tup in similatity]\n",
    "    \n",
    "    # remove the user's own id\n",
    "    most_similar_user.remove(user_id)\n",
    "    \n",
    "    # remove the user's own similarity score\n",
    "    similatity_score.remove(similatity_score[0])\n",
    "       \n",
    "    return most_similar_user, similatity_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[306, 102, 518, 232, 119]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the implemented function above\n",
    "\n",
    "similar_users(564, interaction_matrix)[0][:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[0.4752071]]),\n",
       " array([[0.45945837]]),\n",
       " array([[0.45806577]]),\n",
       " array([[0.42648819]]),\n",
       " array([[0.42188474]])]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the implemented function above\n",
    "\n",
    "similar_users(564, interaction_matrix)[1][:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 5: Creating similarity based collaborative recommendation system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recommendations(user_id, num_of_movies, user_item_interactions):\n",
    "    \n",
    "    # find the most similar users to the user_id for which we want to recommend movies\n",
    "    most_similar_user = similar_users(user_id, user_item_interactions)[0]\n",
    "    \n",
    "    # find out those movies which this user has already interacted with\n",
    "    movie_id = set(list(interaction_matrix.columns[np.where(interaction_matrix.loc[user_id] > 0)]))\n",
    "    \n",
    "    # create an empty list to store the recommended movies\n",
    "    recommedation = []\n",
    "    \n",
    "    # copy those movies which are already interacted by user_id\n",
    "    already_interacted = movie_id.copy()\n",
    "    \n",
    "    # loop through each similar user from the list of most_similar_users\n",
    "    for similar_user in most_similar_user:    \n",
    "        \n",
    "        # implement the below code till the length of recommended movies does not become equal to num_of_movies\n",
    "        if len(recommedation) < num_of_movies:\n",
    "            \n",
    "            # store all the movies interacted by each similar user to user_id\n",
    "            similar_user_movie_id = set(list(interaction_matrix.columns[np.where(interaction_matrix.loc[similar_user] > 0)]))\n",
    "            \n",
    "            # add those movies in the recommended list which are present in similar_user_movie_ids but not present in already_interacted\n",
    "            recommedation.extend(list(similar_user_movie_id.difference(already_interacted)))\n",
    "            \n",
    "            # now add all those movies into already_interacted which we already added in recommendations\n",
    "            already_interacted = already_interacted.union(similar_user_movie_id)\n",
    "            \n",
    "        else:\n",
    "            break\n",
    "    \n",
    "    return recommedation[:num_of_movies]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1537, 515, 1027, 2565, 524]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the implemented function above\n",
    "\n",
    "recommendations(564, 5, interaction_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 6: Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](CF.png \"Similarity Based Collaborative Filtering\")\n",
    "image source - https://www.xpertup.com/downloads/movie-recommender-system/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you recommend three movies to a new user which do not have interactions history in the dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "356    341\n",
       "296    324\n",
       "318    311\n",
       "Name: movieId, dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# recommend top three movies to the new user\n",
    "\n",
    "movies['movieId'].value_counts()[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}