movies_recsys.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommendation Systems for movies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://labs.criteo.com/wp-content/uploads/2017/08/CustomersWhoBought3.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two MovieLens datasets are used here. (link : https://www.kaggle.com/rounakbanik/the-movies-dataset/data)\n",
    "\n",
    "* **The Full Dataset:** Consists of 26,000,000 ratings and 750,000 tag applications applied to 45,000 movies by 270,000 users. Includes tag genome data with 12 million relevance scores across 1,100 tags.\n",
    "* **The Small Dataset:** Comprises of 100,000 ratings and 1,300 tag applications applied to 9,000 movies by 700 users.\n",
    "\n",
    "We will build our Simple Recommender using movies from the *Full Dataset* whereas all personalized recommender systems will make use of the small dataset (due to the computing power I possess being very limited). As a first step, let us build our simple recommender system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#importing required libraries \n",
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "pd.options.display.max_columns = None\n",
    "\n",
    "from scipy import stats\n",
    "from ast import literal_eval\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
    "from sklearn.metrics.pairwise import linear_kernel, cosine_similarity\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "from nltk.corpus import wordnet\n",
    "from surprise import Reader, Dataset, SVD, evaluate\n",
    "\n",
    "\n",
    "import warnings; warnings.simplefilter('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Popularity recommendation model\n",
    "\n",
    "The model simply do implement generalize recommendation to all user based on popularity of movie. The logic behind the model is average audience probably pick or like movies which is critically acclaimed and most popular.This model does not give personalized recommendations based on the user. \n",
    "\n",
    "Procedure is as follow.\n",
    "\n",
    "- First sort movies based on ratings and popularity\n",
    "- Display the top movies of list. \n",
    "\n",
    "** ---- added step ------- ** <br> \n",
    "also used genre argument for the top movies of a particular genre. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>adult</th>\n",
       "      <th>belongs_to_collection</th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>poster_path</th>\n",
       "      <th>production_companies</th>\n",
       "      <th>production_countries</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>video</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>{'id': 10194, 'name': 'Toy Story Collection', ...</td>\n",
       "      <td>30000000</td>\n",
       "      <td>[{'id': 16, 'name': 'Animation'}, {'id': 35, '...</td>\n",
       "      <td>http://toystory.disney.com/toy-story</td>\n",
       "      <td>862</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>en</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "      <td>21.9469</td>\n",
       "      <td>/rhIRbceoE9lR4veEXuwCC2wARtG.jpg</td>\n",
       "      <td>[{'name': 'Pixar Animation Studios', 'id': 3}]</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-10-30</td>\n",
       "      <td>373554033.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>False</td>\n",
       "      <td>7.7</td>\n",
       "      <td>5415.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>65000000</td>\n",
       "      <td>[{'id': 12, 'name': 'Adventure'}, {'id': 14, '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8844</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>en</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>When siblings Judy and Peter discover an encha...</td>\n",
       "      <td>17.0155</td>\n",
       "      <td>/vzmL6fP7aPKNKPRTFnZmiUfciyV.jpg</td>\n",
       "      <td>[{'name': 'TriStar Pictures', 'id': 559}, {'na...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-12-15</td>\n",
       "      <td>262797249.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}, {'iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Roll the dice and unleash the excitement!</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>False</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2413.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>{'id': 119050, 'name': 'Grumpy Old Men Collect...</td>\n",
       "      <td>0</td>\n",
       "      <td>[{'id': 10749, 'name': 'Romance'}, {'id': 35, ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15602</td>\n",
       "      <td>tt0113228</td>\n",
       "      <td>en</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>A family wedding reignites the ancient feud be...</td>\n",
       "      <td>11.7129</td>\n",
       "      <td>/6ksm1sjKMFLbO7UY2i6G1ju9SML.jpg</td>\n",
       "      <td>[{'name': 'Warner Bros.', 'id': 6194}, {'name'...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-12-22</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Still Yelling. Still Fighting. Still Ready for...</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>False</td>\n",
       "      <td>6.5</td>\n",
       "      <td>92.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16000000</td>\n",
       "      <td>[{'id': 35, 'name': 'Comedy'}, {'id': 18, 'nam...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31357</td>\n",
       "      <td>tt0114885</td>\n",
       "      <td>en</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Cheated on, mistreated and stepped on, the wom...</td>\n",
       "      <td>3.85949</td>\n",
       "      <td>/16XOMpEaLWkrcPqSQqhTmeJuqQl.jpg</td>\n",
       "      <td>[{'name': 'Twentieth Century Fox Film Corporat...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-12-22</td>\n",
       "      <td>81452156.0</td>\n",
       "      <td>127.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Friends are the people who let you be yourself...</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>False</td>\n",
       "      <td>6.1</td>\n",
       "      <td>34.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>{'id': 96871, 'name': 'Father of the Bride Col...</td>\n",
       "      <td>0</td>\n",
       "      <td>[{'id': 35, 'name': 'Comedy'}]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11862</td>\n",
       "      <td>tt0113041</td>\n",
       "      <td>en</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Just when George Banks has recovered from his ...</td>\n",
       "      <td>8.38752</td>\n",
       "      <td>/e64sOI48hQXyru7naBFyssKFxVd.jpg</td>\n",
       "      <td>[{'name': 'Sandollar Productions', 'id': 5842}...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-02-10</td>\n",
       "      <td>76578911.0</td>\n",
       "      <td>106.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Just When His World Is Back To Normal... He's ...</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>False</td>\n",
       "      <td>5.7</td>\n",
       "      <td>173.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   adult                              belongs_to_collection    budget  \\\n",
       "0  False  {'id': 10194, 'name': 'Toy Story Collection', ...  30000000   \n",
       "1  False                                                NaN  65000000   \n",
       "2  False  {'id': 119050, 'name': 'Grumpy Old Men Collect...         0   \n",
       "3  False                                                NaN  16000000   \n",
       "4  False  {'id': 96871, 'name': 'Father of the Bride Col...         0   \n",
       "\n",
       "                                              genres  \\\n",
       "0  [{'id': 16, 'name': 'Animation'}, {'id': 35, '...   \n",
       "1  [{'id': 12, 'name': 'Adventure'}, {'id': 14, '...   \n",
       "2  [{'id': 10749, 'name': 'Romance'}, {'id': 35, ...   \n",
       "3  [{'id': 35, 'name': 'Comedy'}, {'id': 18, 'nam...   \n",
       "4                     [{'id': 35, 'name': 'Comedy'}]   \n",
       "\n",
       "                               homepage     id    imdb_id original_language  \\\n",
       "0  http://toystory.disney.com/toy-story    862  tt0114709                en   \n",
       "1                                   NaN   8844  tt0113497                en   \n",
       "2                                   NaN  15602  tt0113228                en   \n",
       "3                                   NaN  31357  tt0114885                en   \n",
       "4                                   NaN  11862  tt0113041                en   \n",
       "\n",
       "                original_title  \\\n",
       "0                    Toy Story   \n",
       "1                      Jumanji   \n",
       "2             Grumpier Old Men   \n",
       "3            Waiting to Exhale   \n",
       "4  Father of the Bride Part II   \n",
       "\n",
       "                                            overview popularity  \\\n",
       "0  Led by Woody, Andy's toys live happily in his ...    21.9469   \n",
       "1  When siblings Judy and Peter discover an encha...    17.0155   \n",
       "2  A family wedding reignites the ancient feud be...    11.7129   \n",
       "3  Cheated on, mistreated and stepped on, the wom...    3.85949   \n",
       "4  Just when George Banks has recovered from his ...    8.38752   \n",
       "\n",
       "                        poster_path  \\\n",
       "0  /rhIRbceoE9lR4veEXuwCC2wARtG.jpg   \n",
       "1  /vzmL6fP7aPKNKPRTFnZmiUfciyV.jpg   \n",
       "2  /6ksm1sjKMFLbO7UY2i6G1ju9SML.jpg   \n",
       "3  /16XOMpEaLWkrcPqSQqhTmeJuqQl.jpg   \n",
       "4  /e64sOI48hQXyru7naBFyssKFxVd.jpg   \n",
       "\n",
       "                                production_companies  \\\n",
       "0     [{'name': 'Pixar Animation Studios', 'id': 3}]   \n",
       "1  [{'name': 'TriStar Pictures', 'id': 559}, {'na...   \n",
       "2  [{'name': 'Warner Bros.', 'id': 6194}, {'name'...   \n",
       "3  [{'name': 'Twentieth Century Fox Film Corporat...   \n",
       "4  [{'name': 'Sandollar Productions', 'id': 5842}...   \n",
       "\n",
       "                                production_countries release_date  \\\n",
       "0  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-10-30   \n",
       "1  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-12-15   \n",
       "2  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-12-22   \n",
       "3  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-12-22   \n",
       "4  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-02-10   \n",
       "\n",
       "       revenue  runtime                                   spoken_languages  \\\n",
       "0  373554033.0     81.0           [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "1  262797249.0    104.0  [{'iso_639_1': 'en', 'name': 'English'}, {'iso...   \n",
       "2          0.0    101.0           [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "3   81452156.0    127.0           [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "4   76578911.0    106.0           [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "\n",
       "     status                                            tagline  \\\n",
       "0  Released                                                NaN   \n",
       "1  Released          Roll the dice and unleash the excitement!   \n",
       "2  Released  Still Yelling. Still Fighting. Still Ready for...   \n",
       "3  Released  Friends are the people who let you be yourself...   \n",
       "4  Released  Just When His World Is Back To Normal... He's ...   \n",
       "\n",
       "                         title  video  vote_average  vote_count  \n",
       "0                    Toy Story  False           7.7      5415.0  \n",
       "1                      Jumanji  False           6.9      2413.0  \n",
       "2             Grumpier Old Men  False           6.5        92.0  \n",
       "3            Waiting to Exhale  False           6.1        34.0  \n",
       "4  Father of the Bride Part II  False           5.7       173.0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m_df = pd.read_csv('data/movies_metadata.csv')\n",
    "m_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "m_df['genres'] = m_df['genres'].fillna('[]').apply(literal_eval).apply(lambda x: [i['name'] for i in x] \n",
    "                                                                       if isinstance(x, list) else [])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### IMDB's *weighted rating* formula \n",
    "\n",
    "Mathematically, it is represented as follows:\n",
    "\n",
    "Weighted Rating (WR) = $(\\frac{v}{v + m} . R) + (\\frac{m}{v + m} . C)$\n",
    "\n",
    "where,\n",
    "* *v* is the number of votes for the movie\n",
    "* *m* is the minimum votes required to be listed in the chart\n",
    "* *R* is the average rating of the movie\n",
    "* *C* is the mean vote across the whole report\n",
    "\n",
    "Now, determine an cutoff value for *m*, the minimum votes required to be listed in the chart. \n",
    "\n",
    "in this model used cutoff is **95th percentile** that means for a movie to feature in the top N must have more votes than at least 95% of the movies in the list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.244896612406511"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Claculation of c \n",
    "vote_counts = m_df[m_df['vote_count'].notnull()]['vote_count'].astype('int')\n",
    "vote_averages = m_df[m_df['vote_average'].notnull()]['vote_average'].astype('int')\n",
    "C = vote_averages.mean()\n",
    "C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate m "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "434.0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Claculation of m\n",
    "m = vote_counts.quantile(0.95)\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# extracting release year from release_date\n",
    "m_df['year'] = pd.to_datetime(m_df['release_date'], errors='coerce').apply(lambda x: str(x).split('-')[0] \n",
    "                                                                           if x != np.nan else np.nan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Getting qualified movies (cutoff: 95%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2274, 6)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_list = ['title', 'year', 'vote_count', 'vote_average', 'popularity', 'genres']\n",
    "qualified = m_df[(m_df['vote_count'] >= m) \n",
    "                 & (m_df['vote_count'].notnull()) \n",
    "                 & (m_df['vote_average'].notnull())][col_list]\n",
    "\n",
    "# converting vote_count and vote_average colums to integer\n",
    "qualified['vote_count'] = qualified['vote_count'].astype('int')\n",
    "qualified['vote_average'] = qualified['vote_average'].astype('int')\n",
    "qualified.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Findings as follow:\n",
    "1. for consideration in the chart, a movie has to have at least **434 votes**. \n",
    "2. the average rating for a movie is **5.244** on a scale of 10. \n",
    "3. **2274** Movies qualify to be on the chart."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating weighted ratings and making top 250 chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def weighted_rating(x):\n",
    "    v = x['vote_count']\n",
    "    R = x['vote_average']\n",
    "    return (v/(v+m) * R) + (m/(m+v) * C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "qualified['weighted_rating'] = qualified.apply(weighted_rating, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "qualified = qualified.sort_values('weighted_rating', ascending=False).head(250)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top 10 Movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>popularity</th>\n",
       "      <th>genres</th>\n",
       "      <th>weighted_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15480</th>\n",
       "      <td>Inception</td>\n",
       "      <td>2010</td>\n",
       "      <td>14075</td>\n",
       "      <td>8</td>\n",
       "      <td>29.1081</td>\n",
       "      <td>[Action, Thriller, Science Fiction, Mystery, A...</td>\n",
       "      <td>7.917588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12481</th>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>2008</td>\n",
       "      <td>12269</td>\n",
       "      <td>8</td>\n",
       "      <td>123.167</td>\n",
       "      <td>[Drama, Action, Crime, Thriller]</td>\n",
       "      <td>7.905871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22879</th>\n",
       "      <td>Interstellar</td>\n",
       "      <td>2014</td>\n",
       "      <td>11187</td>\n",
       "      <td>8</td>\n",
       "      <td>32.2135</td>\n",
       "      <td>[Adventure, Drama, Science Fiction]</td>\n",
       "      <td>7.897107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2843</th>\n",
       "      <td>Fight Club</td>\n",
       "      <td>1999</td>\n",
       "      <td>9678</td>\n",
       "      <td>8</td>\n",
       "      <td>63.8696</td>\n",
       "      <td>[Drama]</td>\n",
       "      <td>7.881753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4863</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>2001</td>\n",
       "      <td>8892</td>\n",
       "      <td>8</td>\n",
       "      <td>32.0707</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>7.871787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>292</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>1994</td>\n",
       "      <td>8670</td>\n",
       "      <td>8</td>\n",
       "      <td>140.95</td>\n",
       "      <td>[Thriller, Crime]</td>\n",
       "      <td>7.868660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>314</th>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>1994</td>\n",
       "      <td>8358</td>\n",
       "      <td>8</td>\n",
       "      <td>51.6454</td>\n",
       "      <td>[Drama, Crime]</td>\n",
       "      <td>7.864000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7000</th>\n",
       "      <td>The Lord of the Rings: The Return of the King</td>\n",
       "      <td>2003</td>\n",
       "      <td>8226</td>\n",
       "      <td>8</td>\n",
       "      <td>29.3244</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>7.861927</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351</th>\n",
       "      <td>Forrest Gump</td>\n",
       "      <td>1994</td>\n",
       "      <td>8147</td>\n",
       "      <td>8</td>\n",
       "      <td>48.3072</td>\n",
       "      <td>[Comedy, Drama, Romance]</td>\n",
       "      <td>7.860656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5814</th>\n",
       "      <td>The Lord of the Rings: The Two Towers</td>\n",
       "      <td>2002</td>\n",
       "      <td>7641</td>\n",
       "      <td>8</td>\n",
       "      <td>29.4235</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>7.851924</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   title  year  vote_count  \\\n",
       "15480                                          Inception  2010       14075   \n",
       "12481                                    The Dark Knight  2008       12269   \n",
       "22879                                       Interstellar  2014       11187   \n",
       "2843                                          Fight Club  1999        9678   \n",
       "4863   The Lord of the Rings: The Fellowship of the Ring  2001        8892   \n",
       "292                                         Pulp Fiction  1994        8670   \n",
       "314                             The Shawshank Redemption  1994        8358   \n",
       "7000       The Lord of the Rings: The Return of the King  2003        8226   \n",
       "351                                         Forrest Gump  1994        8147   \n",
       "5814               The Lord of the Rings: The Two Towers  2002        7641   \n",
       "\n",
       "       vote_average popularity  \\\n",
       "15480             8    29.1081   \n",
       "12481             8    123.167   \n",
       "22879             8    32.2135   \n",
       "2843              8    63.8696   \n",
       "4863              8    32.0707   \n",
       "292               8     140.95   \n",
       "314               8    51.6454   \n",
       "7000              8    29.3244   \n",
       "351               8    48.3072   \n",
       "5814              8    29.4235   \n",
       "\n",
       "                                                  genres  weighted_rating  \n",
       "15480  [Action, Thriller, Science Fiction, Mystery, A...         7.917588  \n",
       "12481                   [Drama, Action, Crime, Thriller]         7.905871  \n",
       "22879                [Adventure, Drama, Science Fiction]         7.897107  \n",
       "2843                                             [Drama]         7.881753  \n",
       "4863                        [Adventure, Fantasy, Action]         7.871787  \n",
       "292                                    [Thriller, Crime]         7.868660  \n",
       "314                                       [Drama, Crime]         7.864000  \n",
       "7000                        [Adventure, Fantasy, Action]         7.861927  \n",
       "351                             [Comedy, Drama, Romance]         7.860656  \n",
       "5814                        [Adventure, Fantasy, Action]         7.851924  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qualified.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5 out 10 movies of genres drama. Also three Christopher Nolan Films, **Inception**, **The Dark Knight** and **Interstellar** are part of top 10 list. The list coins a strong bias towards meta data like genres, directors etc. \n",
    "\n",
    "Now, building list for particular genres. For that, cutoff is relaxed to **85%** instead of 95. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "temp = m_df.apply(lambda x: pd.Series(x['genres']),axis=1).stack().reset_index(level=1, drop=True)\n",
    "temp.name = 'genre'\n",
    "mgen_df = m_df.drop('genres', axis=1).join(temp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_toplist(genre, percentile=0.85):\n",
    "    df = mgen_df[mgen_df['genre'] == genre]\n",
    "    vote_counts = df[df['vote_count'].notnull()]['vote_count'].astype('int')\n",
    "    vote_averages = df[df['vote_average'].notnull()]['vote_average'].astype('int')\n",
    "    C = vote_averages.mean()\n",
    "    m = vote_counts.quantile(percentile)\n",
    "    \n",
    "    col_list = ['title', 'year', 'vote_count', 'vote_average', 'popularity', 'genre']\n",
    "    qualified = df[(df['vote_count'] >= m) \n",
    "                   & (df['vote_count'].notnull()) \n",
    "                   & (df['vote_average'].notnull())][col_list]\n",
    "    qualified['vote_count'] = qualified['vote_count'].astype('int')\n",
    "    qualified['vote_average'] = qualified['vote_average'].astype('int')\n",
    "    \n",
    "    qualified['weighted_rating'] = qualified.apply(lambda x: \n",
    "                                                   (x['vote_count']/(x['vote_count']+m) * x['vote_average']) + (m/(m+x['vote_count']) * C), \n",
    "                                                   axis=1)\n",
    "    qualified = qualified.sort_values('weighted_rating', ascending=False).head(250)\n",
    "    \n",
    "    return qualified"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top 10 Romance Movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>popularity</th>\n",
       "      <th>genre</th>\n",
       "      <th>weighted_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10309</th>\n",
       "      <td>Dilwale Dulhania Le Jayenge</td>\n",
       "      <td>1995</td>\n",
       "      <td>661</td>\n",
       "      <td>9</td>\n",
       "      <td>34.457</td>\n",
       "      <td>Romance</td>\n",
       "      <td>8.565285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351</th>\n",
       "      <td>Forrest Gump</td>\n",
       "      <td>1994</td>\n",
       "      <td>8147</td>\n",
       "      <td>8</td>\n",
       "      <td>48.3072</td>\n",
       "      <td>Romance</td>\n",
       "      <td>7.971357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>876</th>\n",
       "      <td>Vertigo</td>\n",
       "      <td>1958</td>\n",
       "      <td>1162</td>\n",
       "      <td>8</td>\n",
       "      <td>18.2082</td>\n",
       "      <td>Romance</td>\n",
       "      <td>7.811667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40251</th>\n",
       "      <td>Your Name.</td>\n",
       "      <td>2016</td>\n",
       "      <td>1030</td>\n",
       "      <td>8</td>\n",
       "      <td>34.461252</td>\n",
       "      <td>Romance</td>\n",
       "      <td>7.789489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>883</th>\n",
       "      <td>Some Like It Hot</td>\n",
       "      <td>1959</td>\n",
       "      <td>835</td>\n",
       "      <td>8</td>\n",
       "      <td>11.8451</td>\n",
       "      <td>Romance</td>\n",
       "      <td>7.745154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1132</th>\n",
       "      <td>Cinema Paradiso</td>\n",
       "      <td>1988</td>\n",
       "      <td>834</td>\n",
       "      <td>8</td>\n",
       "      <td>14.177</td>\n",
       "      <td>Romance</td>\n",
       "      <td>7.744878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19901</th>\n",
       "      <td>Paperman</td>\n",
       "      <td>2012</td>\n",
       "      <td>734</td>\n",
       "      <td>8</td>\n",
       "      <td>7.19863</td>\n",
       "      <td>Romance</td>\n",
       "      <td>7.713951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37863</th>\n",
       "      <td>Sing Street</td>\n",
       "      <td>2016</td>\n",
       "      <td>669</td>\n",
       "      <td>8</td>\n",
       "      <td>10.672862</td>\n",
       "      <td>Romance</td>\n",
       "      <td>7.689483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>882</th>\n",
       "      <td>The Apartment</td>\n",
       "      <td>1960</td>\n",
       "      <td>498</td>\n",
       "      <td>8</td>\n",
       "      <td>11.9943</td>\n",
       "      <td>Romance</td>\n",
       "      <td>7.599317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38718</th>\n",
       "      <td>The Handmaiden</td>\n",
       "      <td>2016</td>\n",
       "      <td>453</td>\n",
       "      <td>8</td>\n",
       "      <td>16.727405</td>\n",
       "      <td>Romance</td>\n",
       "      <td>7.566166</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             title  year  vote_count  vote_average popularity  \\\n",
       "10309  Dilwale Dulhania Le Jayenge  1995         661             9     34.457   \n",
       "351                   Forrest Gump  1994        8147             8    48.3072   \n",
       "876                        Vertigo  1958        1162             8    18.2082   \n",
       "40251                   Your Name.  2016        1030             8  34.461252   \n",
       "883               Some Like It Hot  1959         835             8    11.8451   \n",
       "1132               Cinema Paradiso  1988         834             8     14.177   \n",
       "19901                     Paperman  2012         734             8    7.19863   \n",
       "37863                  Sing Street  2016         669             8  10.672862   \n",
       "882                  The Apartment  1960         498             8    11.9943   \n",
       "38718               The Handmaiden  2016         453             8  16.727405   \n",
       "\n",
       "         genre  weighted_rating  \n",
       "10309  Romance         8.565285  \n",
       "351    Romance         7.971357  \n",
       "876    Romance         7.811667  \n",
       "40251  Romance         7.789489  \n",
       "883    Romance         7.745154  \n",
       "1132   Romance         7.744878  \n",
       "19901  Romance         7.713951  \n",
       "37863  Romance         7.689483  \n",
       "882    Romance         7.599317  \n",
       "38718  Romance         7.566166  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "make_toplist('Romance').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LOL!!!\n",
    "#### A bollywood movie Dilwale Dulhania Le Jayenge is getting 1st place"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Content Based Recommendation model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://johnolamendy.files.wordpress.com/2015/10/01.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Limitation of Popularity model ** <br>\n",
    "It gives the same recommendation to everyone, regardless of the user's personal interest. <br>\n",
    "\n",
    "_For example:_ If a person who loves romantic movies were to look at Top 15 romantic movies, and he wouldn't probably like most of the listed movies. So, he were to go one step further and look at movie lists by genre, he wouldn't still be getting the interesting recommendations. Let say, he loves *Dilwale Dulhania Le Jayenge*, *My Name is Khan* and *Kabhi Khushi Kabhi Gham*. We can probable assume that he loves the actor Shahrukh Khan and the director Karan Johar. Even if he were to access the top romance list, he wouldn't find these in the list.\n",
    "\n",
    "Therefore, let's build a model that computes similarity between movies based on certain metrics and suggests movies that are most similar to a particular movie that a user liked. For that we have to consider metadata (or content), hence, it also known as **Content Based Filtering.**\n",
    "\n",
    "Two Content Based Recommendation is implemented based on different contents:\n",
    "1. Description Based (content: Movie Overviews and Taglines)\n",
    "2. Meta Data Based (content : Movie Cast, Crew, Keywords and Genre)\n",
    "\n",
    "** Note **: A small movie data set is used due to limiting computing power available to me. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "small_mdf = pd.read_csv('data/links_small.csv')\n",
    "small_mdf = small_mdf[small_mdf['tmdbId'].notnull()]['tmdbId'].astype('int')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before extracting small data set, we need to make sure that the ID column of our main dataframe is clean and of type integer. To do this, let us try to perform an integer conversion of our IDs and if an exception is raised,we will replace the ID with NaN. We will then proceed to drop these rows from our dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convert_int(x):\n",
    "    try:\n",
    "        return int(x)\n",
    "    except:\n",
    "        return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "m_df['id'] = m_df['id'].apply(convert_int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>adult</th>\n",
       "      <th>belongs_to_collection</th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>poster_path</th>\n",
       "      <th>production_companies</th>\n",
       "      <th>production_countries</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>video</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19730</th>\n",
       "      <td>- Written by Ørnås</td>\n",
       "      <td>0.065736</td>\n",
       "      <td>/ff9qCepilowshEtG2GYWwzt2bs4.jpg</td>\n",
       "      <td>[Carousel Productions, Vision View Entertainme...</td>\n",
       "      <td>[{'iso_3166_1': 'CA', 'name': 'Canada'}, {'iso...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Midnight Man</td>\n",
       "      <td>False</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29503</th>\n",
       "      <td>Rune Balot goes to a casino connected to the ...</td>\n",
       "      <td>1.931659</td>\n",
       "      <td>/zV8bHuSL6WXoD6FWogP9j4x80bL.jpg</td>\n",
       "      <td>[Aniplex, GoHands, BROSTA TV, Mardock Scramble...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>[{'iso_639_1': 'ja', 'name': '日本語'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mardock Scramble: The Third Exhaust</td>\n",
       "      <td>False</td>\n",
       "      <td>7.0</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35587</th>\n",
       "      <td>Avalanche Sharks tells the story of a bikini ...</td>\n",
       "      <td>2.185485</td>\n",
       "      <td>/zaSf5OG7V8X8gqFvly88zDdRm46.jpg</td>\n",
       "      <td>[Odyssey Media, Pulser Productions, Rogue Stat...</td>\n",
       "      <td>[{'iso_3166_1': 'CA', 'name': 'Canada'}]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Beware Of Frost Bites</td>\n",
       "      <td>Avalanche Sharks</td>\n",
       "      <td>False</td>\n",
       "      <td>4.3</td>\n",
       "      <td>22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   adult  \\\n",
       "19730                                 - Written by Ørnås   \n",
       "29503   Rune Balot goes to a casino connected to the ...   \n",
       "35587   Avalanche Sharks tells the story of a bikini ...   \n",
       "\n",
       "      belongs_to_collection                            budget  \\\n",
       "19730              0.065736  /ff9qCepilowshEtG2GYWwzt2bs4.jpg   \n",
       "29503              1.931659  /zV8bHuSL6WXoD6FWogP9j4x80bL.jpg   \n",
       "35587              2.185485  /zaSf5OG7V8X8gqFvly88zDdRm46.jpg   \n",
       "\n",
       "                                                  genres  \\\n",
       "19730  [Carousel Productions, Vision View Entertainme...   \n",
       "29503  [Aniplex, GoHands, BROSTA TV, Mardock Scramble...   \n",
       "35587  [Odyssey Media, Pulser Productions, Rogue Stat...   \n",
       "\n",
       "                                                homepage  id imdb_id  \\\n",
       "19730  [{'iso_3166_1': 'CA', 'name': 'Canada'}, {'iso... NaN       0   \n",
       "29503  [{'iso_3166_1': 'US', 'name': 'United States o... NaN       0   \n",
       "35587           [{'iso_3166_1': 'CA', 'name': 'Canada'}] NaN       0   \n",
       "\n",
       "      original_language                            original_title  overview  \\\n",
       "19730             104.0  [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "29503              68.0      [{'iso_639_1': 'ja', 'name': '日本語'}]  Released   \n",
       "35587              82.0  [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "\n",
       "                  popularity                          poster_path  \\\n",
       "19730                    NaN                         Midnight Man   \n",
       "29503                    NaN  Mardock Scramble: The Third Exhaust   \n",
       "35587  Beware Of Frost Bites                     Avalanche Sharks   \n",
       "\n",
       "      production_companies production_countries release_date  revenue  \\\n",
       "19730                False                  6.0            1      NaN   \n",
       "29503                False                  7.0           12      NaN   \n",
       "35587                False                  4.3           22      NaN   \n",
       "\n",
       "       runtime spoken_languages status tagline title video  vote_average  \\\n",
       "19730      NaN              NaN    NaN     NaN   NaN   NaN           NaN   \n",
       "29503      NaN              NaN    NaN     NaN   NaN   NaN           NaN   \n",
       "35587      NaN              NaN    NaN     NaN   NaN   NaN           NaN   \n",
       "\n",
       "       vote_count year  \n",
       "19730         NaN  NaT  \n",
       "29503         NaN  NaT  \n",
       "35587         NaN  NaT  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m_df[m_df['id'].isnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "m_df = m_df.drop([19730, 29503, 35587])\n",
    "m_df['id'] = m_df['id'].astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9099, 25)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sm_df = m_df[m_df['id'].isin(small_mdf)]\n",
    "sm_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**9099** movies avaiable in our small movies metadata dataset "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Description Based Recommendation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sm_df['tagline'] = sm_df['tagline'].fillna('')\n",
    "sm_df['description'] = sm_df['overview'] + sm_df['tagline']\n",
    "sm_df['description'] = sm_df['description'].fillna('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compute TF-IDF matrix "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9099, 268124)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf = TfidfVectorizer(analyzer='word',ngram_range=(1, 2),min_df=0, stop_words='english')\n",
    "tfidf_matrix = tf.fit_transform(sm_df['description'])\n",
    "tfidf_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cosine Similarity\n",
    "\n",
    "The Cosine Similarity is used to calculate a numeric quantity that denotes the similarity between two movies. Mathematically, it is defined as follows:\n",
    "\n",
    "$cosine(x,y) = \\frac{x. y^\\intercal}{||x||.||y||} $\n",
    "\n",
    "Since the TF-IDF Vectorizer is used, calculating the Dot Product will directly give us the Cosine Similarity Score. Therefore, sklearn's **linear_kernel** is used instead of cosine_similarities as it's much faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.        ,  0.00680476,  0.        , ...,  0.        ,\n",
       "        0.00344913,  0.        ])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cosine_sim[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a pairwise cosine similarity matrix for all the movies in our dataset. The next step is to write a function that returns the 50 most similar movies based on the cosine similarity score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sm_df = sm_df.reset_index()\n",
    "titles = sm_df['title']\n",
    "indices = pd.Series(sm_df.index, index=sm_df['title'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_recommendations(title):\n",
    "    idx = indices[title]\n",
    "    sim_scores = list(enumerate(cosine_sim[idx]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    sim_scores = sim_scores[1:51]\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    return titles.iloc[movie_indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now try and get the top 10 recommendations for a few movies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "973      The Godfather: Part II\n",
       "8387                 The Family\n",
       "3509                       Made\n",
       "4196         Johnny Dangerously\n",
       "29               Shanghai Triad\n",
       "5667                       Fury\n",
       "2412             American Movie\n",
       "1582    The Godfather: Part III\n",
       "4221                    8 Women\n",
       "2159              Summer of Sam\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Godfather').head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7931                      The Dark Knight Rises\n",
       "132                              Batman Forever\n",
       "1113                             Batman Returns\n",
       "8227    Batman: The Dark Knight Returns, Part 2\n",
       "7565                 Batman: Under the Red Hood\n",
       "524                                      Batman\n",
       "7901                           Batman: Year One\n",
       "2579               Batman: Mask of the Phantasm\n",
       "2696                                        JFK\n",
       "8165    Batman: The Dark Knight Returns, Part 1\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Dark Knight').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that for **The Dark Knight**, our system is able to identify it as a Batman film and subsequently recommend other Batman films as its top recommendations. But unfortunately, that is all this system can do at the moment. This is not of much use to most people as it doesn't take into considerations very important features such as cast, crew, director and genre, which determine the rating and the popularity of a movie. Someone who liked **The Dark Knight** probably likes it more because of Nolan and would hate **Batman Forever** and every other substandard movie in the Batman Franchise.\n",
    "\n",
    "Therefore, we are going to use much more suggestive metadata than **Overview** and **Tagline**. In the next subsection, Let's build a more sophisticated recommender that takes **genre**, **keywords**, **cast** and **crew** into consideration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.  Metadata Based Recommendation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "credits = pd.read_csv('data/credits.csv')\n",
    "keywords = pd.read_csv('data/keywords.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(45463, 25)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keywords['id'] = keywords['id'].astype('int')\n",
    "credits['id'] = credits['id'].astype('int')\n",
    "m_df['id'] = m_df['id'].astype('int')\n",
    "m_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# merging require details with our main dataframe \n",
    "m_df = m_df.merge(credits, on='id')\n",
    "m_df = m_df.merge(keywords, on='id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9219, 28)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sm_df = m_df[m_df['id'].isin(small_mdf)]\n",
    "sm_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have our cast, crew, genres and credits, all in one dataframe. Let us wrangle this a little more using the following intuitions:\n",
    "\n",
    "1. **Crew:** From the crew, we will only pick the director as our feature since the others don't contribute that much to the *feel* of the movie.\n",
    "2. **Cast:** Choosing Cast is a little more tricky. Lesser known actors and minor roles do not really affect people's opinion of a movie. Therefore, we must only select the major characters and their respective actors. Arbitrarily we will choose the top 3 actors that appear in the credits list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sm_df['cast'] = sm_df['cast'].apply(literal_eval)\n",
    "sm_df['crew'] = sm_df['crew'].apply(literal_eval)\n",
    "sm_df['keywords'] = sm_df['keywords'].apply(literal_eval)\n",
    "sm_df['cast_size'] = sm_df['cast'].apply(lambda x: len(x))\n",
    "sm_df['crew_size'] = sm_df['crew'].apply(lambda x: len(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_director(x):\n",
    "    for i in x:\n",
    "        if i['job'] == 'Director':\n",
    "            return i['name']\n",
    "    return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sm_df['director'] = sm_df['crew'].apply(get_director)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sm_df['cast'] = sm_df['cast'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])\n",
    "sm_df['cast'] = sm_df['cast'].apply(lambda x: x[:3] if len(x) >=3 else x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sm_df['keywords'] = sm_df['keywords'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The approach to building the recommendation is going to be extremely *hacky*. The plan on doing is creating a metadata dump for every movie which consists of **genres, director, main actors and keywords.** I then use a **Count Vectorizer** to create our count matrix as we did in the Description Recommender. The remaining steps are similar to what we did earlier: we calculate the cosine similarities and return movies that are most similar.\n",
    "\n",
    "These are steps I follow in the preparation of genres and credits data:\n",
    "1. **Strip Spaces and Convert to Lowercase** from all our features. This way, engine will not confuse between **Johnny Depp** and **Johnny Galecki.** \n",
    "2. **Mention Director 2 times** to give it more weight relative to the entire cast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sm_df['cast'] = sm_df['cast'].apply(lambda x: [str.lower(i.replace(\" \", \"\")) for i in x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sm_df['director'] = sm_df['director'].astype('str').apply(lambda x: str.lower(x.replace(\" \", \"\")))\n",
    "sm_df['director'] = sm_df['director'].apply(lambda x: [x,x])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Keywords\n",
    "\n",
    "We will do a small amount of pre-processing of our keywords before putting them to any use. As a first step, we calculate the frequenct counts of every keyword that appears in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s = sm_df.apply(lambda x: pd.Series(x['keywords']),axis=1).stack().reset_index(level=1, drop=True)\n",
    "s.name = 'keyword'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "independent film        610\n",
       "woman director          550\n",
       "murder                  399\n",
       "duringcreditsstinger    327\n",
       "based on novel          318\n",
       "Name: keyword, dtype: int64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = s.value_counts()\n",
    "s[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keywords occur in frequencies ranging from 1 to 610. We do not have any use for keywords that occur only once. Therefore, these can be safely removed. Finally, we will convert every word to its stem so that words such as *Dogs* and *Dog* are considered the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s = s[s > 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'dog'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmer = SnowballStemmer('english')\n",
    "stemmer.stem('dogs')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def filter_keywords(x):\n",
    "    words = []\n",
    "    for i in x:\n",
    "        if i in s:\n",
    "            words.append(i)\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sm_df['keywords'] = sm_df['keywords'].apply(filter_keywords)\n",
    "sm_df['keywords'] = sm_df['keywords'].apply(lambda x: [stemmer.stem(i) for i in x])\n",
    "sm_df['keywords'] = sm_df['keywords'].apply(lambda x: [str.lower(i.replace(\" \", \"\")) for i in x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sm_df['soup'] = sm_df['keywords'] + sm_df['cast'] + sm_df['director'] + sm_df['genres']\n",
    "sm_df['soup'] = sm_df['soup'].apply(lambda x: ' '.join(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "count = CountVectorizer(analyzer='word',ngram_range=(1, 2),min_df=0, stop_words='english')\n",
    "count_matrix = count.fit_transform(sm_df['soup'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cosine_sim = cosine_similarity(count_matrix, count_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sm_df = sm_df.reset_index()\n",
    "titles = sm_df['title']\n",
    "indices = pd.Series(sm_df.index, index=sm_df['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will reuse the get_recommendations function that we had written earlier. Since our cosine similarity scores have changed, we expect it to give us different (and probably better) results. Let us check for **The Dark Knight** again and see what recommendations I get this time around."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8031                 The Dark Knight Rises\n",
       "6218                         Batman Begins\n",
       "7659            Batman: Under the Red Hood\n",
       "6623                          The Prestige\n",
       "1134                        Batman Returns\n",
       "8927               Kidnapping Mr. Heineken\n",
       "5943                              Thursday\n",
       "1260                        Batman & Robin\n",
       "2085                             Following\n",
       "9024    Batman v Superman: Dawn of Justice\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Dark Knight').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The recommendations seem to have recognized other Christopher Nolan movies (due to the high weightage given to director) and put them as top recommendations. Person enjoyed watching **The Dark Knight** as well as some of the other ones in the list including **Batman Begins**, **The Prestige** and **The Dark Knight Rises**. \n",
    "\n",
    "We can of course experiment on this engine by trying out different weights for our features (directors, actors, genres), limiting the number of keywords that can be used in the soup, weighing genres based on their frequency, only showing movies with the same languages, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1381         Jackie Brown\n",
       "8905    The Hateful Eight\n",
       "5200    Kill Bill: Vol. 2\n",
       "4595                Basic\n",
       "4764             S.W.A.T.\n",
       "898        Reservoir Dogs\n",
       "6939              Cleaner\n",
       "4903    Kill Bill: Vol. 1\n",
       "231         Kiss of Death\n",
       "4306       The 51st State\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('Pulp Fiction').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Popularity and Ratings\n",
    "\n",
    "One thing that we notice about our recommendation system is that it recommends movies regardless of ratings and popularity. It is true that **Batman and Robin** has a lot of similar characters as compared to **The Dark Knight** but it was a terrible movie that shouldn't be recommended to anyone.\n",
    "\n",
    "Therefore, we will add a mechanism to remove bad movies and return movies which are popular and have had a good critical response.\n",
    "\n",
    "I will take the top 50 movies based on similarity scores and calculate the vote of the 60th percentile movie. Then, using this as the value of $m$, we will calculate the weighted rating of each movie using IMDB's formula like we did in the Simple Recommender section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def improved_recommendations(title):\n",
    "    idx = indices[title]\n",
    "    sim_scores = list(enumerate(cosine_sim[idx]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    sim_scores = sim_scores[1:51]\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    \n",
    "    movies = sm_df.iloc[movie_indices][['title', 'vote_count', 'vote_average', 'year']]\n",
    "    vote_counts = movies[movies['vote_count'].notnull()]['vote_count'].astype('int')\n",
    "    vote_averages = movies[movies['vote_average'].notnull()]['vote_average'].astype('int')\n",
    "    C = vote_averages.mean()\n",
    "    m = vote_counts.quantile(0.60)\n",
    "    qualified = movies[(movies['vote_count'] >= m) & (movies['vote_count'].notnull()) & (movies['vote_average'].notnull())]\n",
    "    qualified['vote_count'] = qualified['vote_count'].astype('int')\n",
    "    qualified['vote_average'] = qualified['vote_average'].astype('int')\n",
    "    qualified['wr'] = qualified.apply(weighted_rating, axis=1)\n",
    "    qualified = qualified.sort_values('wr', ascending=False).head(10)\n",
    "    return qualified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>year</th>\n",
       "      <th>wr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7648</th>\n",
       "      <td>Inception</td>\n",
       "      <td>14075</td>\n",
       "      <td>8</td>\n",
       "      <td>2010</td>\n",
       "      <td>7.917588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6623</th>\n",
       "      <td>The Prestige</td>\n",
       "      <td>4510</td>\n",
       "      <td>8</td>\n",
       "      <td>2006</td>\n",
       "      <td>7.758148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8031</th>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>9263</td>\n",
       "      <td>7</td>\n",
       "      <td>2012</td>\n",
       "      <td>6.921448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6218</th>\n",
       "      <td>Batman Begins</td>\n",
       "      <td>7511</td>\n",
       "      <td>7</td>\n",
       "      <td>2005</td>\n",
       "      <td>6.904127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>524</th>\n",
       "      <td>Batman</td>\n",
       "      <td>2145</td>\n",
       "      <td>7</td>\n",
       "      <td>1989</td>\n",
       "      <td>6.704647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1031</th>\n",
       "      <td>M</td>\n",
       "      <td>465</td>\n",
       "      <td>8</td>\n",
       "      <td>1931</td>\n",
       "      <td>6.669950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7659</th>\n",
       "      <td>Batman: Under the Red Hood</td>\n",
       "      <td>459</td>\n",
       "      <td>7</td>\n",
       "      <td>2010</td>\n",
       "      <td>6.147016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2085</th>\n",
       "      <td>Following</td>\n",
       "      <td>363</td>\n",
       "      <td>7</td>\n",
       "      <td>1998</td>\n",
       "      <td>6.044272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1134</th>\n",
       "      <td>Batman Returns</td>\n",
       "      <td>1706</td>\n",
       "      <td>6</td>\n",
       "      <td>1992</td>\n",
       "      <td>5.846862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4145</th>\n",
       "      <td>Insomnia</td>\n",
       "      <td>1181</td>\n",
       "      <td>6</td>\n",
       "      <td>2002</td>\n",
       "      <td>5.797081</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           title  vote_count  vote_average  year        wr\n",
       "7648                   Inception       14075             8  2010  7.917588\n",
       "6623                The Prestige        4510             8  2006  7.758148\n",
       "8031       The Dark Knight Rises        9263             7  2012  6.921448\n",
       "6218               Batman Begins        7511             7  2005  6.904127\n",
       "524                       Batman        2145             7  1989  6.704647\n",
       "1031                           M         465             8  1931  6.669950\n",
       "7659  Batman: Under the Red Hood         459             7  2010  6.147016\n",
       "2085                   Following         363             7  1998  6.044272\n",
       "1134              Batman Returns        1706             6  1992  5.846862\n",
       "4145                    Insomnia        1181             6  2002  5.797081"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "improved_recommendations('The Dark Knight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>year</th>\n",
       "      <th>wr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>898</th>\n",
       "      <td>Reservoir Dogs</td>\n",
       "      <td>3821</td>\n",
       "      <td>8</td>\n",
       "      <td>1992</td>\n",
       "      <td>7.718986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7280</th>\n",
       "      <td>Inglourious Basterds</td>\n",
       "      <td>6598</td>\n",
       "      <td>7</td>\n",
       "      <td>2009</td>\n",
       "      <td>6.891679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4903</th>\n",
       "      <td>Kill Bill: Vol. 1</td>\n",
       "      <td>5091</td>\n",
       "      <td>7</td>\n",
       "      <td>2003</td>\n",
       "      <td>6.862133</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8905</th>\n",
       "      <td>The Hateful Eight</td>\n",
       "      <td>4405</td>\n",
       "      <td>7</td>\n",
       "      <td>2015</td>\n",
       "      <td>6.842588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5200</th>\n",
       "      <td>Kill Bill: Vol. 2</td>\n",
       "      <td>4061</td>\n",
       "      <td>7</td>\n",
       "      <td>2004</td>\n",
       "      <td>6.830542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1381</th>\n",
       "      <td>Jackie Brown</td>\n",
       "      <td>1580</td>\n",
       "      <td>7</td>\n",
       "      <td>1997</td>\n",
       "      <td>6.621790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4185</th>\n",
       "      <td>Road to Perdition</td>\n",
       "      <td>1102</td>\n",
       "      <td>7</td>\n",
       "      <td>2002</td>\n",
       "      <td>6.504092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8110</th>\n",
       "      <td>The Raid</td>\n",
       "      <td>1076</td>\n",
       "      <td>7</td>\n",
       "      <td>2011</td>\n",
       "      <td>6.495553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8408</th>\n",
       "      <td>Fast &amp; Furious 6</td>\n",
       "      <td>5282</td>\n",
       "      <td>6</td>\n",
       "      <td>2013</td>\n",
       "      <td>5.942667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8390</th>\n",
       "      <td>Pain &amp; Gain</td>\n",
       "      <td>1595</td>\n",
       "      <td>6</td>\n",
       "      <td>2013</td>\n",
       "      <td>5.838485</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     title  vote_count  vote_average  year        wr\n",
       "898         Reservoir Dogs        3821             8  1992  7.718986\n",
       "7280  Inglourious Basterds        6598             7  2009  6.891679\n",
       "4903     Kill Bill: Vol. 1        5091             7  2003  6.862133\n",
       "8905     The Hateful Eight        4405             7  2015  6.842588\n",
       "5200     Kill Bill: Vol. 2        4061             7  2004  6.830542\n",
       "1381          Jackie Brown        1580             7  1997  6.621790\n",
       "4185     Road to Perdition        1102             7  2002  6.504092\n",
       "8110              The Raid        1076             7  2011  6.495553\n",
       "8408      Fast & Furious 6        5282             6  2013  5.942667\n",
       "8390           Pain & Gain        1595             6  2013  5.838485"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "improved_recommendations('Pulp Fiction')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collaborative Filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cdn-images-1.medium.com/max/1600/1*6_NlX6CJYhtxzRM-t6ywkQ.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "** Limitation of content based recommendation**: \n",
    "- It is only capable of suggesting movies which are *close* to a certain movie. That is, it is not capable of capturing interest and providing recommendations across genres.\n",
    "\n",
    "- It doesn't capture the personal intrest and biases of a user. Anyone querying on model for recommendations based on a movie will receive the same recommendations for that movie, regardless of who he is.\n",
    "\n",
    "Therefore, in this section, we will use a technique called **Collaborative Filtering** to make recommendations to Movie Watchers. Collaborative Filtering is based on the idea that users similar to a me can be used to predict how much I will like a particular product or service those users have used/experienced but I have not.\n",
    "\n",
    "We will use the **Surprise** library that used extremely powerful algorithms like **Singular Value Decomposition (SVD)** to minimise RMSE (Root Mean Square Error) and give great recommendations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reader = Reader()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1029</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1061</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1129</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1172</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759205</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating   timestamp\n",
       "0       1       31     2.5  1260759144\n",
       "1       1     1029     3.0  1260759179\n",
       "2       1     1061     3.0  1260759182\n",
       "3       1     1129     2.0  1260759185\n",
       "4       1     1172     4.0  1260759205"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings = pd.read_csv('data/ratings_small.csv')\n",
    "ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = Dataset.load_from_df(ratings[['userId', 'movieId', 'rating']], reader)\n",
    "data.split(n_folds=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating RMSE, MAE of algorithm SVD.\n",
      "\n",
      "------------\n",
      "Fold 1\n",
      "RMSE: 0.9008\n",
      "MAE:  0.6944\n",
      "------------\n",
      "Fold 2\n",
      "RMSE: 0.8919\n",
      "MAE:  0.6840\n",
      "------------\n",
      "Fold 3\n",
      "RMSE: 0.8964\n",
      "MAE:  0.6881\n",
      "------------\n",
      "Fold 4\n",
      "RMSE: 0.9026\n",
      "MAE:  0.6968\n",
      "------------\n",
      "Fold 5\n",
      "RMSE: 0.8906\n",
      "MAE:  0.6870\n",
      "------------\n",
      "------------\n",
      "Mean RMSE: 0.8965\n",
      "Mean MAE : 0.6901\n",
      "------------\n",
      "------------\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "CaseInsensitiveDefaultDict(list,\n",
       "                           {'mae': [0.69443477456023961,\n",
       "                             0.68397211592587936,\n",
       "                             0.68813511558111051,\n",
       "                             0.69675387746923578,\n",
       "                             0.68701148071308493],\n",
       "                            'rmse': [0.9008020040112541,\n",
       "                             0.89189730970017422,\n",
       "                             0.89637981271588829,\n",
       "                             0.9025859116774404,\n",
       "                             0.8906018126837052]})"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd = SVD()\n",
    "evaluate(svd, data, measures=['RMSE', 'MAE'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get a mean **Root Mean Sqaure Error** of 0.8965 which is more than good enough right now. Let us now train on dataset and arrive at predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<surprise.prediction_algorithms.matrix_factorization.SVD at 0x1745e850550>"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainset = data.build_full_trainset()\n",
    "svd.train(trainset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us pick user 5000 and check the ratings he has given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1029</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1061</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1129</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1172</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>1263</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>1287</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759187</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>1293</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>1339</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1260759125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>1343</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>1371</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1</td>\n",
       "      <td>1405</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1260759203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1</td>\n",
       "      <td>1953</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>2105</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1</td>\n",
       "      <td>2150</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1</td>\n",
       "      <td>2193</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1</td>\n",
       "      <td>2294</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1</td>\n",
       "      <td>2455</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1</td>\n",
       "      <td>2968</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1260759200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1</td>\n",
       "      <td>3671</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759117</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    userId  movieId  rating   timestamp\n",
       "0        1       31     2.5  1260759144\n",
       "1        1     1029     3.0  1260759179\n",
       "2        1     1061     3.0  1260759182\n",
       "3        1     1129     2.0  1260759185\n",
       "4        1     1172     4.0  1260759205\n",
       "5        1     1263     2.0  1260759151\n",
       "6        1     1287     2.0  1260759187\n",
       "7        1     1293     2.0  1260759148\n",
       "8        1     1339     3.5  1260759125\n",
       "9        1     1343     2.0  1260759131\n",
       "10       1     1371     2.5  1260759135\n",
       "11       1     1405     1.0  1260759203\n",
       "12       1     1953     4.0  1260759191\n",
       "13       1     2105     4.0  1260759139\n",
       "14       1     2150     3.0  1260759194\n",
       "15       1     2193     2.0  1260759198\n",
       "16       1     2294     2.0  1260759108\n",
       "17       1     2455     2.5  1260759113\n",
       "18       1     2968     1.0  1260759200\n",
       "19       1     3671     3.0  1260759117"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings[ratings['userId'] == 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prediction(uid=1, iid=302, r_ui=3, est=2.8735075863145378, details={'was_impossible': False})"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd.predict(1, 302, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For movie with ID 302, we get an estimated prediction of **2.873**. One startling feature of this recommender system is that it doesn't care what the movie is (or what it contains). It works purely on the basis of an assigned movie ID and tries to predict ratings based on how the other users have predicted the movie."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hybrid Recommender\n",
    "\n",
    "![](https://www.toonpool.com/user/250/files/hybrid_20095.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we build a simple hybrid recommender that brings together techniques we have implemented in the content based and collaborative filter based engines. This is how it will work:\n",
    "\n",
    "* **Input:** User ID and the Title of a Movie\n",
    "* **Output:** Similar movies sorted on the basis of expected ratings by that particular user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convert_int(x):\n",
    "    try:\n",
    "        return int(x)\n",
    "    except:\n",
    "        return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "id_map = pd.read_csv('data/links_small.csv')[['movieId', 'tmdbId']]\n",
    "id_map['tmdbId'] = id_map['tmdbId'].apply(convert_int)\n",
    "id_map.columns = ['movieId', 'id']\n",
    "id_map = id_map.merge(sm_df[['title', 'id']], on='id').set_index('title')\n",
    "#id_map = id_map.set_index('tmdbId')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "indices_map = id_map.set_index('id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def hybrid(userId, title):\n",
    "    idx = indices[title]\n",
    "    tmdbId = id_map.loc[title]['id']\n",
    "    #print(idx)\n",
    "    movie_id = id_map.loc[title]['movieId']\n",
    "    \n",
    "    sim_scores = list(enumerate(cosine_sim[int(idx)]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    sim_scores = sim_scores[1:26]\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    \n",
    "    movies = sm_df.iloc[movie_indices][['title', 'vote_count', 'vote_average', 'year', 'id']]\n",
    "    movies['est'] = movies['id'].apply(lambda x: svd.predict(userId, indices_map.loc[x]['movieId']).est)\n",
    "    movies = movies.sort_values('est', ascending=False)\n",
    "    return movies.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>year</th>\n",
       "      <th>id</th>\n",
       "      <th>est</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>522</th>\n",
       "      <td>Terminator 2: Judgment Day</td>\n",
       "      <td>4274.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1991</td>\n",
       "      <td>280</td>\n",
       "      <td>3.177291</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8658</th>\n",
       "      <td>X-Men: Days of Future Past</td>\n",
       "      <td>6155.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>2014</td>\n",
       "      <td>127585</td>\n",
       "      <td>3.057280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>The Terminator</td>\n",
       "      <td>4208.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>1984</td>\n",
       "      <td>218</td>\n",
       "      <td>3.054917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>974</th>\n",
       "      <td>Aliens</td>\n",
       "      <td>3282.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1986</td>\n",
       "      <td>679</td>\n",
       "      <td>2.958559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7705</th>\n",
       "      <td>Alice in Wonderland</td>\n",
       "      <td>8.0</td>\n",
       "      <td>5.4</td>\n",
       "      <td>1933</td>\n",
       "      <td>25694</td>\n",
       "      <td>2.883120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4017</th>\n",
       "      <td>Hawk the Slayer</td>\n",
       "      <td>13.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1980</td>\n",
       "      <td>25628</td>\n",
       "      <td>2.861426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1668</th>\n",
       "      <td>Return from Witch Mountain</td>\n",
       "      <td>38.0</td>\n",
       "      <td>5.6</td>\n",
       "      <td>1978</td>\n",
       "      <td>14822</td>\n",
       "      <td>2.775316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8401</th>\n",
       "      <td>Star Trek Into Darkness</td>\n",
       "      <td>4479.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>2013</td>\n",
       "      <td>54138</td>\n",
       "      <td>2.772912</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>922</th>\n",
       "      <td>The Abyss</td>\n",
       "      <td>822.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>1989</td>\n",
       "      <td>2756</td>\n",
       "      <td>2.770524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8865</th>\n",
       "      <td>Star Wars: The Force Awakens</td>\n",
       "      <td>7993.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>2015</td>\n",
       "      <td>140607</td>\n",
       "      <td>2.769325</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             title  vote_count  vote_average  year      id  \\\n",
       "522     Terminator 2: Judgment Day      4274.0           7.7  1991     280   \n",
       "8658    X-Men: Days of Future Past      6155.0           7.5  2014  127585   \n",
       "1011                The Terminator      4208.0           7.4  1984     218   \n",
       "974                         Aliens      3282.0           7.7  1986     679   \n",
       "7705           Alice in Wonderland         8.0           5.4  1933   25694   \n",
       "4017               Hawk the Slayer        13.0           4.5  1980   25628   \n",
       "1668    Return from Witch Mountain        38.0           5.6  1978   14822   \n",
       "8401       Star Trek Into Darkness      4479.0           7.4  2013   54138   \n",
       "922                      The Abyss       822.0           7.1  1989    2756   \n",
       "8865  Star Wars: The Force Awakens      7993.0           7.5  2015  140607   \n",
       "\n",
       "           est  \n",
       "522   3.177291  \n",
       "8658  3.057280  \n",
       "1011  3.054917  \n",
       "974   2.958559  \n",
       "7705  2.883120  \n",
       "4017  2.861426  \n",
       "1668  2.775316  \n",
       "8401  2.772912  \n",
       "922   2.770524  \n",
       "8865  2.769325  "
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hybrid(1, 'Avatar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>year</th>\n",
       "      <th>id</th>\n",
       "      <th>est</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7705</th>\n",
       "      <td>Alice in Wonderland</td>\n",
       "      <td>8.0</td>\n",
       "      <td>5.4</td>\n",
       "      <td>1933</td>\n",
       "      <td>25694</td>\n",
       "      <td>3.328704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8401</th>\n",
       "      <td>Star Trek Into Darkness</td>\n",
       "      <td>4479.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>2013</td>\n",
       "      <td>54138</td>\n",
       "      <td>3.308255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>922</th>\n",
       "      <td>The Abyss</td>\n",
       "      <td>822.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>1989</td>\n",
       "      <td>2756</td>\n",
       "      <td>3.240657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8724</th>\n",
       "      <td>Jupiter Ascending</td>\n",
       "      <td>2816.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2015</td>\n",
       "      <td>76757</td>\n",
       "      <td>3.210003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2132</th>\n",
       "      <td>Superman II</td>\n",
       "      <td>642.0</td>\n",
       "      <td>6.5</td>\n",
       "      <td>1980</td>\n",
       "      <td>8536</td>\n",
       "      <td>3.122031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014</th>\n",
       "      <td>Fantastic Planet</td>\n",
       "      <td>140.0</td>\n",
       "      <td>7.6</td>\n",
       "      <td>1973</td>\n",
       "      <td>16306</td>\n",
       "      <td>2.988982</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>522</th>\n",
       "      <td>Terminator 2: Judgment Day</td>\n",
       "      <td>4274.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1991</td>\n",
       "      <td>280</td>\n",
       "      <td>2.982938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8658</th>\n",
       "      <td>X-Men: Days of Future Past</td>\n",
       "      <td>6155.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>2014</td>\n",
       "      <td>127585</td>\n",
       "      <td>2.976679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>974</th>\n",
       "      <td>Aliens</td>\n",
       "      <td>3282.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1986</td>\n",
       "      <td>679</td>\n",
       "      <td>2.934363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7265</th>\n",
       "      <td>Dragonball Evolution</td>\n",
       "      <td>475.0</td>\n",
       "      <td>2.9</td>\n",
       "      <td>2009</td>\n",
       "      <td>14164</td>\n",
       "      <td>2.922109</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           title  vote_count  vote_average  year      id  \\\n",
       "7705         Alice in Wonderland         8.0           5.4  1933   25694   \n",
       "8401     Star Trek Into Darkness      4479.0           7.4  2013   54138   \n",
       "922                    The Abyss       822.0           7.1  1989    2756   \n",
       "8724           Jupiter Ascending      2816.0           5.2  2015   76757   \n",
       "2132                 Superman II       642.0           6.5  1980    8536   \n",
       "2014            Fantastic Planet       140.0           7.6  1973   16306   \n",
       "522   Terminator 2: Judgment Day      4274.0           7.7  1991     280   \n",
       "8658  X-Men: Days of Future Past      6155.0           7.5  2014  127585   \n",
       "974                       Aliens      3282.0           7.7  1986     679   \n",
       "7265        Dragonball Evolution       475.0           2.9  2009   14164   \n",
       "\n",
       "           est  \n",
       "7705  3.328704  \n",
       "8401  3.308255  \n",
       "922   3.240657  \n",
       "8724  3.210003  \n",
       "2132  3.122031  \n",
       "2014  2.988982  \n",
       "522   2.982938  \n",
       "8658  2.976679  \n",
       "974   2.934363  \n",
       "7265  2.922109  "
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hybrid(500, 'Avatar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that for our hybrid recommender, we get different recommendations for different users although the movie is the same. Hence, our recommendations are more personalized and tailored towards particular users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "In this notebook, we have built 4 different recommendation engines based on different ideas and algorithms. They are as follows:\n",
    "\n",
    "1. **Simple Recommender:** This system used overall TMDB Vote Count and Vote Averages to build Top Movies Charts, in general and for a specific genre. The IMDB Weighted Rating System was used to calculate ratings on which the sorting was finally performed.\n",
    "2. **Content Based Recommender:** We built two content based engines; one that took movie overview and taglines as input and the other which took metadata such as cast, crew, genre and keywords to come up with predictions. We also deviced a simple filter to give greater preference to movies with more votes and higher ratings.\n",
    "3. **Collaborative Filtering:** We used the powerful Surprise Library to build a collaborative filter based on single value decomposition. The RMSE obtained was less than 1 and the engine gave estimated ratings for a given user and movie.\n",
    "4. **Hybrid Engine:** We brought together ideas from content and collaborative filterting to build an engine that gave movie suggestions to a particular user based on the estimated ratings that it had internally calculated for that user."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}