Movie_Recommendation_System_Complete_Guide.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "premium"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Movie Recommender System - Model Training & Inference\n",
        "\n",
        "*Table of Content*\n",
        "\n",
        "| Sr. No | Section Title | Description | Reference Link |\n",
        "| ------ | ------------- | ----------- | -------------- |\n",
        "| 00 | Drive & Files | Linking your google drive and uploading your Kaggle API token | To be Updated |\n",
        "| 01 | Kaggle Setup | Information & details on downloading dataset via Kaggle API | To be Updated |\n",
        "| 02 | Modules & Library | Information on setting up requirements for training and inference | To be Updated |\n",
        "| 03 | Data Cleaning & Engineering | Information on preparing data for trainable features | To be Updated |\n",
        "| 04 | Recommendation Matrix | Building the matrix which contains similarity scores between movies based on the features | To be Updated |\n",
        "| 05 | Model & Data Export | Exporting the trained model & dataset efficiently | To be Updated |\n",
        "| 06 | Inference | Loading the trained model to execute Inference | To be Updated |"
      ],
      "metadata": {
        "id": "bLlWM-puE5mD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Section 0: Drive & Files\n",
        "\n",
        ">     Linking your google drive and uploading your Kaggle API token"
      ],
      "metadata": {
        "id": "dG4l46IEIChc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 1: Mounting your Google Drive"
      ],
      "metadata": {
        "id": "p1HPlxYPINoh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive, files\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5A667famIGgW",
        "outputId": "97e5caf2-9c2f-4e4f-f9ce-37d08c268837"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 2: Uploading your Kaggle API Token File"
      ],
      "metadata": {
        "id": "rfN2T3MzIQ0W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "files.upload()"
      ],
      "metadata": {
        "id": "InKGIBDMIUgy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Section 1: Kaggle Setup\n",
        "\n",
        "> \tInformation & details on downloading dataset via Kaggle API"
      ],
      "metadata": {
        "id": "6VD74_AsHR7b"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 1: Enable Kaggle API for User Mode Aceess"
      ],
      "metadata": {
        "id": "7u2yJBokHa6d"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q kaggle\n",
        "!mkdir ~/.kaggle\n",
        "!cp kaggle.json ~/.kaggle/\n",
        "!chmod 600 ~/.kaggle/kaggle.json"
      ],
      "metadata": {
        "id": "uhxd7OHky0fO"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 2: Downloading the Dataset from Kaggle to Google Colab"
      ],
      "metadata": {
        "id": "IlxdiNQ8I59g"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!kaggle datasets download -d rounakbanik/the-movies-dataset"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LF1GwGzdyzRB",
        "outputId": "b852a828-bc99-4b8e-ae97-ba7b742efd56"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading the-movies-dataset.zip to /content\n",
            "100% 228M/228M [00:11<00:00, 23.3MB/s]\n",
            "100% 228M/228M [00:11<00:00, 20.8MB/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 3: Extracting Dataset\n",
        "\n",
        "Whenever we download dataset from Kaggle, it is usually in zip or tar compression technique. To be able to access dataset, we need to extract it using suitable decompression technique."
      ],
      "metadata": {
        "id": "4UWjm4GLI_vb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## Unzip the dataset into /content/data directory \n",
        "!unzip the-movies-dataset.zip -d /content/data\n",
        "\n",
        "## Once extracted, we then remove it to save disk space.\n",
        "!rm the-movies-dataset.zip"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GPRRbnETzmwX",
        "outputId": "790dea7c-31e2-49de-ca35-85134c2f3063"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Archive:  the-movies-dataset.zip\n",
            "  inflating: /content/data/credits.csv  \n",
            "  inflating: /content/data/keywords.csv  \n",
            "  inflating: /content/data/links.csv  \n",
            "  inflating: /content/data/links_small.csv  \n",
            "  inflating: /content/data/movies_metadata.csv  \n",
            "  inflating: /content/data/ratings.csv  \n",
            "  inflating: /content/data/ratings_small.csv  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Section 2: Modules & Library\n",
        "\n",
        ">     Information on setting up requirements for training and inference"
      ],
      "metadata": {
        "id": "jTLFhb4dJsSL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 1: Installing required packages"
      ],
      "metadata": {
        "id": "ewM4nXZ3Knxr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install --quiet fastparquet\n",
        "!pip install --quiet pyarrow"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M5VnX1WH1qjS",
        "outputId": "8462469d-2853-4e6c-8121-065edd45ac56"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m47.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.2/12.2 MB\u001b[0m \u001b[31m75.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m63.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "For your Information: This notebook is complied on Google Colab that provides most of the modules pre-installed in working environment. If you happen to run it locally on your system, you may need to install additional dependencies. \n",
        "\n",
        "---"
      ],
      "metadata": {
        "id": "YT1LOx8PKr2c"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 2: Importing Rerquired Packages"
      ],
      "metadata": {
        "id": "5tymq4LKXkBv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%matplotlib inline\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "from ast import literal_eval\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "from nltk.stem.snowball import SnowballStemmer\n",
        "\n",
        "import pyarrow as pa\n",
        "import pyarrow.parquet as pq\n",
        "\n",
        "import warnings\n",
        "warnings.simplefilter('ignore')"
      ],
      "metadata": {
        "id": "GZp5ZWzWyUXE"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Section 3: Data Cleaning & Engineering\n",
        "\n",
        ">     Information on preparing data for trainable features"
      ],
      "metadata": {
        "id": "HHYRCCwCXsIP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 1: Utility Functions For Data Cleaning & Engineering"
      ],
      "metadata": {
        "id": "XJqzQzFKfaJ0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_director(x):\n",
        "    \"\"\"\n",
        "    Extract the Name of the Director for a movie if it is present inside the job\n",
        "    \"\"\"\n",
        "    for i in x:\n",
        "        if i['job'] == 'Director':\n",
        "            return i['name']\n",
        "    return np.nan"
      ],
      "metadata": {
        "id": "z_SmfY9WfBXk"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 2: Reading dataset and merging them to form master dataset"
      ],
      "metadata": {
        "id": "imLY6sRy_OBs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_dataset  = pd.read_csv('/content/data/movies_metadata.csv')\n",
        "credits         = pd.read_csv('/content/data/credits.csv')\n",
        "keywords        = pd.read_csv('/content/data/keywords.csv')\n",
        "links           = pd.read_csv('/content/data/links.csv')"
      ],
      "metadata": {
        "id": "6ZDrGO9oyUUs"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Dropping these 3 rows because Date Column value for them is string date instead of Int with ID.\n",
        "movies_dataset = movies_dataset.drop([19730, 29503, 35587])"
      ],
      "metadata": {
        "id": "2ruYT_8N_Cw6"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Extracting Genres of movies from the genres dictionary. If not present, append empty list\n",
        "movies_dataset['genres'] = movies_dataset['genres'].fillna('[]').apply(literal_eval).apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])"
      ],
      "metadata": {
        "id": "g8DNnxq4yUS_"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Convert to common data type for primary key in our dataset\n",
        "keywords['id'] = keywords['id'].astype('int')\n",
        "credits['id'] = credits['id'].astype('int')\n",
        "movies_dataset['id'] = movies_dataset['id'].astype('int')"
      ],
      "metadata": {
        "id": "rrLCuGdT2xo3"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Merging movies dataset with credits & keywords to form master dataset\n",
        "movies_dataset = movies_dataset.merge(credits, on='id')\n",
        "master_dataset = movies_dataset.merge(keywords, on='id')"
      ],
      "metadata": {
        "id": "kCZEqojC2xm9"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "master_dataset.head(2)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 396
        },
        "id": "Vb-6uHBCV0Cp",
        "outputId": "f254cdae-1e02-4907-ee0f-7cf61b04599c"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   adult                              belongs_to_collection    budget  \\\n",
              "0  False  {'id': 10194, 'name': 'Toy Story Collection', ...  30000000   \n",
              "1  False                                                NaN  65000000   \n",
              "\n",
              "                         genres                              homepage    id  \\\n",
              "0   [Animation, Comedy, Family]  http://toystory.disney.com/toy-story   862   \n",
              "1  [Adventure, Fantasy, Family]                                   NaN  8844   \n",
              "\n",
              "     imdb_id original_language original_title  \\\n",
              "0  tt0114709                en      Toy Story   \n",
              "1  tt0113497                en        Jumanji   \n",
              "\n",
              "                                            overview  ...  \\\n",
              "0  Led by Woody, Andy's toys live happily in his ...  ...   \n",
              "1  When siblings Judy and Peter discover an encha...  ...   \n",
              "\n",
              "                                    spoken_languages    status  \\\n",
              "0           [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
              "1  [{'iso_639_1': 'en', 'name': 'English'}, {'iso...  Released   \n",
              "\n",
              "                                     tagline      title  video  vote_average  \\\n",
              "0                                        NaN  Toy Story  False           7.7   \n",
              "1  Roll the dice and unleash the excitement!    Jumanji  False           6.9   \n",
              "\n",
              "   vote_count                                               cast  \\\n",
              "0      5415.0  [{'cast_id': 14, 'character': 'Woody (voice)',...   \n",
              "1      2413.0  [{'cast_id': 1, 'character': 'Alan Parrish', '...   \n",
              "\n",
              "                                                crew  \\\n",
              "0  [{'credit_id': '52fe4284c3a36847f8024f49', 'de...   \n",
              "1  [{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...   \n",
              "\n",
              "                                            keywords  \n",
              "0  [{'id': 931, 'name': 'jealousy'}, {'id': 4290,...  \n",
              "1  [{'id': 10090, 'name': 'board game'}, {'id': 1...  \n",
              "\n",
              "[2 rows x 27 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-263d26bb-d5fa-4e3c-9a14-6bdc52355986\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>adult</th>\n",
              "      <th>belongs_to_collection</th>\n",
              "      <th>budget</th>\n",
              "      <th>genres</th>\n",
              "      <th>homepage</th>\n",
              "      <th>id</th>\n",
              "      <th>imdb_id</th>\n",
              "      <th>original_language</th>\n",
              "      <th>original_title</th>\n",
              "      <th>overview</th>\n",
              "      <th>...</th>\n",
              "      <th>spoken_languages</th>\n",
              "      <th>status</th>\n",
              "      <th>tagline</th>\n",
              "      <th>title</th>\n",
              "      <th>video</th>\n",
              "      <th>vote_average</th>\n",
              "      <th>vote_count</th>\n",
              "      <th>cast</th>\n",
              "      <th>crew</th>\n",
              "      <th>keywords</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>False</td>\n",
              "      <td>{'id': 10194, 'name': 'Toy Story Collection', ...</td>\n",
              "      <td>30000000</td>\n",
              "      <td>[Animation, Comedy, Family]</td>\n",
              "      <td>http://toystory.disney.com/toy-story</td>\n",
              "      <td>862</td>\n",
              "      <td>tt0114709</td>\n",
              "      <td>en</td>\n",
              "      <td>Toy Story</td>\n",
              "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
              "      <td>...</td>\n",
              "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
              "      <td>Released</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Toy Story</td>\n",
              "      <td>False</td>\n",
              "      <td>7.7</td>\n",
              "      <td>5415.0</td>\n",
              "      <td>[{'cast_id': 14, 'character': 'Woody (voice)',...</td>\n",
              "      <td>[{'credit_id': '52fe4284c3a36847f8024f49', 'de...</td>\n",
              "      <td>[{'id': 931, 'name': 'jealousy'}, {'id': 4290,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>65000000</td>\n",
              "      <td>[Adventure, Fantasy, Family]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8844</td>\n",
              "      <td>tt0113497</td>\n",
              "      <td>en</td>\n",
              "      <td>Jumanji</td>\n",
              "      <td>When siblings Judy and Peter discover an encha...</td>\n",
              "      <td>...</td>\n",
              "      <td>[{'iso_639_1': 'en', 'name': 'English'}, {'iso...</td>\n",
              "      <td>Released</td>\n",
              "      <td>Roll the dice and unleash the excitement!</td>\n",
              "      <td>Jumanji</td>\n",
              "      <td>False</td>\n",
              "      <td>6.9</td>\n",
              "      <td>2413.0</td>\n",
              "      <td>[{'cast_id': 1, 'character': 'Alan Parrish', '...</td>\n",
              "      <td>[{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...</td>\n",
              "      <td>[{'id': 10090, 'name': 'board game'}, {'id': 1...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2 rows × 27 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-263d26bb-d5fa-4e3c-9a14-6bdc52355986')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-263d26bb-d5fa-4e3c-9a14-6bdc52355986 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-263d26bb-d5fa-4e3c-9a14-6bdc52355986');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(master_dataset.columns)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pbSAFU9X-vGh",
        "outputId": "d193275d-a2cb-4644-8c84-c19cd7dfa599"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Index(['adult', 'belongs_to_collection', 'budget', 'genres', 'homepage', 'id',\n",
            "       'imdb_id', 'original_language', 'original_title', 'overview',\n",
            "       'popularity', 'poster_path', 'production_companies',\n",
            "       'production_countries', 'release_date', 'revenue', 'runtime',\n",
            "       'spoken_languages', 'status', 'tagline', 'title', 'video',\n",
            "       'vote_average', 'vote_count', 'cast', 'crew', 'keywords'],\n",
            "      dtype='object')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "links = links[links['tmdbId'].notnull()]['tmdbId'].astype('int')\n",
        "master_dataset = master_dataset[master_dataset['id'].isin(links)]\n",
        "print(master_dataset.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0Yw0oWEY2xk3",
        "outputId": "dd5ce026-ba2a-4acc-a6cd-ef59917d2073"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(46628, 27)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 3: Data cleaning and Engineering"
      ],
      "metadata": {
        "id": "oVnHryY-_Tgi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## Updating cast, crew and keyword columns by parsing them as their loaded data type is string but need to be converted to list\n",
        "master_dataset['cast']      = master_dataset['cast'].apply(literal_eval)\n",
        "master_dataset['crew']      = master_dataset['crew'].apply(literal_eval)\n",
        "master_dataset['keywords']  = master_dataset['keywords'].apply(literal_eval)"
      ],
      "metadata": {
        "id": "gJYGvLA_2xis"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Updating cast to maintain proportion between different lengths (keeping top 3 cast members)\n",
        "master_dataset['cast']      = master_dataset['cast'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])\n",
        "master_dataset['cast']      = master_dataset['cast'].apply(lambda x: x[:3] if len(x) >=3 else x)\n",
        "\n",
        "## Setting keywords to empty list if does not exists, otherwise taking into account for each word as keyword\n",
        "master_dataset['keywords']  = master_dataset['keywords'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])\n",
        "\n",
        "## Extracting directory names from the crew\n",
        "master_dataset['director']  = master_dataset['crew'].apply(get_director)"
      ],
      "metadata": {
        "id": "MTXcRtg72xgX"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## for uniqueness, removing all the spaces in between the names\n",
        "master_dataset['cast']          = master_dataset['cast'].apply(lambda x: [str.lower(i.replace(\" \", \"\")) for i in x])\n",
        "\n",
        "## Maintaining the original director name as main director\n",
        "master_dataset['main_director'] = master_dataset['director']\n",
        "\n",
        "## Maintaining the number of director to maintain proportion (similar to cast column above)\n",
        "master_dataset['director']      = master_dataset['director'].astype('str').apply(lambda x: str.lower(x.replace(\" \", \"\")))\n",
        "master_dataset['director']      = master_dataset['director'].apply(lambda x: [x,x,x])"
      ],
      "metadata": {
        "id": "Jaqq7Wwk-E2-"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Stacking the keywords and keeping the movies which containers X number of keywords as minimum\n",
        "s = master_dataset.apply(lambda x: pd.Series(x['keywords']),axis=1).stack().reset_index(level=1, drop=True)\n",
        "s.name = 'keyword'\n",
        "s = s.value_counts()\n",
        "print(s[:5])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PKR9dFAL-G63",
        "outputId": "df7658b3-6519-42df-f8b9-a9f58bc2219a"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "woman director      3128\n",
            "independent film    1942\n",
            "murder              1314\n",
            "based on novel       841\n",
            "musical              734\n",
            "Name: keyword, dtype: int64\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "## Will try to map where more than 1 keyword is present for the movie\n",
        "s = s[s > 1]"
      ],
      "metadata": {
        "id": "jN2OJj8H-I-V"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## creating an object for ENGLISH Stemmer - Snowball to trim down keywords to their stem words\n",
        "stemmer                     = SnowballStemmer('english')\n",
        "\n",
        "## Trim down keywords to their stem words and then remove the space between keywords which are having more than 1 length for uniqueness\n",
        "master_dataset['keywords']  = master_dataset['keywords'].apply(lambda x: [stemmer.stem(i) for i in x])\n",
        "master_dataset['keywords']  = master_dataset['keywords'].apply(lambda x: [str.lower(i.replace(\" \", \"\")) for i in x])"
      ],
      "metadata": {
        "id": "j3lSxkmC-N-h"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "master_dataset['keywords'].head(3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3n-PuVspO2fI",
        "outputId": "80b12c89-967f-45cd-ff2f-4a3b7cd9b736"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    [jealousi, toy, boy, friendship, friend, rival...\n",
              "1    [boardgam, disappear, basedonchildren'sbook, n...\n",
              "2       [fish, bestfriend, duringcreditssting, oldmen]\n",
              "Name: keywords, dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "## Creating a soup feature - combination of (keywords, cast, director, genres)\n",
        "master_dataset['soup'] = master_dataset['keywords'] + master_dataset['cast'] + master_dataset['director'] + master_dataset['genres']\n",
        "\n",
        "## Modifying by placing single space between all the soup words\n",
        "master_dataset['soup'] = master_dataset['soup'].apply(lambda x: ' '.join(x))"
      ],
      "metadata": {
        "id": "uSncyfonOyvC"
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "master_dataset['soup'].head(3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_kIbzUnHPBBD",
        "outputId": "8020f807-2c6e-4118-c85b-c290afc484d4"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    jealousi toy boy friendship friend rivalri boy...\n",
              "1    boardgam disappear basedonchildren'sbook newho...\n",
              "2    fish bestfriend duringcreditssting oldmen walt...\n",
              "Name: soup, dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(master_dataset.columns)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6LaAIqSo-chN",
        "outputId": "eb933be4-fc0c-4c25-d9f1-d6592de4f0de"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Index(['adult', 'belongs_to_collection', 'budget', 'genres', 'homepage', 'id',\n",
            "       'imdb_id', 'original_language', 'original_title', 'overview',\n",
            "       'popularity', 'poster_path', 'production_companies',\n",
            "       'production_countries', 'release_date', 'revenue', 'runtime',\n",
            "       'spoken_languages', 'status', 'tagline', 'title', 'video',\n",
            "       'vote_average', 'vote_count', 'cast', 'crew', 'keywords', 'director',\n",
            "       'main_director', 'soup'],\n",
            "      dtype='object')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "## Removing unwanted columns from the dataset - these features can be used if you wish to add more features to your recommender system.\n",
        "## We are not going to use them, so we are removing them.\n",
        "master_dataset.drop(['adult', 'belongs_to_collection', 'budget','homepage','original_language', 'production_companies','production_countries', 'revenue', 'runtime','spoken_languages','status','video'],axis=1,inplace=True)\n",
        "master_dataset.drop(['overview', 'tagline','vote_average', 'vote_count', 'cast', 'crew','keywords', 'director'],axis=1,inplace=True)\n",
        "master_dataset.drop(['id','imdb_id','original_title','poster_path','genres'],axis=1,inplace=True)"
      ],
      "metadata": {
        "id": "GfmKSQ3f-d0e"
      },
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Checking popularity column for being non-float data type and removing them\n",
        "master_dataset['popularity']    = master_dataset.apply(lambda r: r['popularity'] if type(r['popularity'])==float else np.nan, axis=1)\n",
        "master_dataset.dropna(inplace=True)\n",
        "\n",
        "## Checking director column for being non-string data type and removing them\n",
        "master_dataset['main_director'] = master_dataset.apply(lambda r: r['main_director'] if len(r['main_director'])>1 else np.nan, axis=1)\n",
        "master_dataset.dropna(inplace=True)"
      ],
      "metadata": {
        "id": "tTN76P-3QF6Q"
      },
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Sorting the whole dataset based on popularity. This will help us to take top X number of movies based on popularity.\n",
        "master_dataset.sort_values(by=['popularity'],ascending=False,inplace=True)\n",
        "\n",
        "## Dropping popularity column after sorting based on popularity\n",
        "master_dataset.drop(['popularity'],axis=1,inplace=True)\n",
        "master_dataset.dropna(inplace=True)"
      ],
      "metadata": {
        "id": "ovSdqYXFPw0x"
      },
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Reset index because after sorting, the index values have changed.\n",
        "master_dataset.reset_index(inplace=True,drop=True)"
      ],
      "metadata": {
        "id": "5t60gsB6Qjh3"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Checking release date column for being non-string data type and removing them\n",
        "master_dataset['release_date'] = master_dataset.apply(lambda r: r['release_date'] if len(r['release_date'])>1 else np.nan, axis=1)\n",
        "master_dataset.dropna(inplace=True)"
      ],
      "metadata": {
        "id": "PaqeTa8DZrEz"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "IMPORTANT NOTE:\n",
        "\n",
        "The following cell contains comments based on different sizes of models which can be created. If you happen to have Google Colab Pro account or your local system has atleast 32Gb RAM, you may run the full dataset. Otherwise, it is adviced to run smaller dataset which fits your memory.\n",
        "\n",
        "---"
      ],
      "metadata": {
        "id": "9Zwz3zm9HOlU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## For Demo, we will take top 2500 movies, which is hosted online already.\n",
        "master_dataset = master_dataset[:2500]\n",
        "\n",
        "## For Tiny-Model, we will take top 1000 movies\n",
        "# master_dataset = master_dataset[:1000]\n",
        "\n",
        "## For Extra-Small-Model, we will take top 5000 movies\n",
        "# master_dataset = master_dataset[:5000]\n",
        "\n",
        "## For Small-Model, we will take top 10000 movies\n",
        "# master_dataset = master_dataset[:10000]\n",
        "\n",
        "## For Medium-Model, we will take top 20000 movies\n",
        "# master_dataset = master_dataset[:20000]\n",
        "\n",
        "## For Large-Model, we will take top 30000 movies\n",
        "# master_dataset = master_dataset[:30000]\n",
        "\n",
        "## LEAVE ALL THE LINES COMMENTED IF YOU WISH TO TRAIN FULL MOVIES DATASET."
      ],
      "metadata": {
        "id": "6Vx_CcUnQqLR"
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "hpL1QTwOIgH7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## This is our final dataset which we will be using for training our word and cosine similarity matrix\n",
        "master_dataset.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "XRz17iHZZnRc",
        "outputId": "46d95aef-0fed-425a-94c9-3652495ca57b"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  release_date                           title   main_director  \\\n",
              "0   2015-06-17                         Minions      Kyle Balda   \n",
              "1   2014-10-24                      Big Hero 6  Chris Williams   \n",
              "2   2016-02-09                        Deadpool      Tim Miller   \n",
              "3   2017-04-19  Guardians of the Galaxy Vol. 2      James Gunn   \n",
              "4   2009-12-10                          Avatar   James Cameron   \n",
              "\n",
              "                                                soup  \n",
              "0  assist aftercreditssting duringcreditssting ev...  \n",
              "1  brotherbrotherrelationship hero talent reveng ...  \n",
              "2  antihero mercenari marvelcom superhero basedon...  \n",
              "3  sequel superhero basedoncom misfit space outer...  \n",
              "4  cultureclash futur spacewar spacecoloni societ...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-bd4b63f7-29cc-46a4-86d6-a2c93ea7b3e2\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>release_date</th>\n",
              "      <th>title</th>\n",
              "      <th>main_director</th>\n",
              "      <th>soup</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2015-06-17</td>\n",
              "      <td>Minions</td>\n",
              "      <td>Kyle Balda</td>\n",
              "      <td>assist aftercreditssting duringcreditssting ev...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2014-10-24</td>\n",
              "      <td>Big Hero 6</td>\n",
              "      <td>Chris Williams</td>\n",
              "      <td>brotherbrotherrelationship hero talent reveng ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2016-02-09</td>\n",
              "      <td>Deadpool</td>\n",
              "      <td>Tim Miller</td>\n",
              "      <td>antihero mercenari marvelcom superhero basedon...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2017-04-19</td>\n",
              "      <td>Guardians of the Galaxy Vol. 2</td>\n",
              "      <td>James Gunn</td>\n",
              "      <td>sequel superhero basedoncom misfit space outer...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2009-12-10</td>\n",
              "      <td>Avatar</td>\n",
              "      <td>James Cameron</td>\n",
              "      <td>cultureclash futur spacewar spacecoloni societ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bd4b63f7-29cc-46a4-86d6-a2c93ea7b3e2')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-bd4b63f7-29cc-46a4-86d6-a2c93ea7b3e2 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-bd4b63f7-29cc-46a4-86d6-a2c93ea7b3e2');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(master_dataset.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vB09EkGsHK7c",
        "outputId": "3963b746-ca89-4761-a6b8-8b575ec47039"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(2500, 4)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Section 4: Recommendation Matrix\n",
        "\n",
        ">     Building the matrix which contains similarity scores between movies based on the features"
      ],
      "metadata": {
        "id": "brYr66w7GO_M"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 1: Training Word based count vectorizer model"
      ],
      "metadata": {
        "id": "tu_PsFWFJH_l"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## Creating a Count Vectorizer object which will be based on word analyzer, with ngram 1-2 and minimum number of occurances of words as 2\n",
        "count = CountVectorizer(analyzer='word',ngram_range=(1, 2),min_df=2, stop_words='english')\n",
        "\n",
        "## Adjusting the count vectorizer object with respect to our dataset\n",
        "count_matrix = count.fit_transform(master_dataset['soup'])"
      ],
      "metadata": {
        "id": "73hkdAZWAJ76"
      },
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(count_matrix.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V98dZQmkAdhq",
        "outputId": "ad473708-4662-49ee-ad9e-a35c0304498d"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(2500, 7277)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 2: Building Cosine Similarity Matrix\n",
        "\n",
        "**NOTE: THE FOLLOWING CODE CELL CAN CONSUME LARGE MEMORY**"
      ],
      "metadata": {
        "id": "bpjCz58UI62D"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## We build it as an pyarrow dataframe because it is the most efficient \n",
        "table = pa.Table.from_pandas(pd.DataFrame(cosine_similarity(count_matrix, count_matrix)))"
      ],
      "metadata": {
        "id": "M3rn6GT_RUIh"
      },
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model & Data Export\n",
        "\n",
        ">     Exporting the trained model & dataset efficiently"
      ],
      "metadata": {
        "id": "-BLBAThCiQ7u"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We export the model into parquet format. We have 3 awesome reasons (even recommend for you in your new project)\n",
        "\n",
        "1. Uses Less Storage\n",
        "2. Best Compression Ratio\n",
        "3. Fast & Optimized for efficient Read/Write"
      ],
      "metadata": {
        "id": "CfR_tWcMihg4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## save the Master Dataset\n",
        "master_dataset.to_parquet('/content/movie_database.parquet',engine='fastparquet',index=False)"
      ],
      "metadata": {
        "id": "0lSgay2QBLfR"
      },
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Writing the Matrix table\n",
        "pq.write_table(table, '/content/model.parquet')"
      ],
      "metadata": {
        "id": "KoXEYlfldl1F"
      },
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Uncomment the following 2 lines of code if you want to save the model to your mounted Google Drive\n",
        "# !mv large_model.parquet ./drive/MyDrive/\n",
        "# !mv large_model.parquet ./drive/MyDrive/"
      ],
      "metadata": {
        "id": "wMlHzAkORabA"
      },
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Inference\n",
        "\n",
        ">     Loading the trained model to execute Inference"
      ],
      "metadata": {
        "id": "e416IXE3RCYJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import pyarrow as pa"
      ],
      "metadata": {
        "id": "QR7FWs_eF8mv"
      },
      "execution_count": 43,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "master_dataset = pd.read_parquet('/content/movie_database.parquet')"
      ],
      "metadata": {
        "id": "Xo0PYdBwFxRc"
      },
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "master_dataset.head(3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "P1CdagSLNpC8",
        "outputId": "5d19e7ac-1104-4b3d-b8f0-42ac341e50b9"
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  release_date       title   main_director  \\\n",
              "0   2015-06-17     Minions      Kyle Balda   \n",
              "1   2014-10-24  Big Hero 6  Chris Williams   \n",
              "2   2016-02-09    Deadpool      Tim Miller   \n",
              "\n",
              "                                                soup  \n",
              "0  assist aftercreditssting duringcreditssting ev...  \n",
              "1  brotherbrotherrelationship hero talent reveng ...  \n",
              "2  antihero mercenari marvelcom superhero basedon...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b116155d-fa0d-4294-a1a7-c3a9e30a1a34\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>release_date</th>\n",
              "      <th>title</th>\n",
              "      <th>main_director</th>\n",
              "      <th>soup</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2015-06-17</td>\n",
              "      <td>Minions</td>\n",
              "      <td>Kyle Balda</td>\n",
              "      <td>assist aftercreditssting duringcreditssting ev...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2014-10-24</td>\n",
              "      <td>Big Hero 6</td>\n",
              "      <td>Chris Williams</td>\n",
              "      <td>brotherbrotherrelationship hero talent reveng ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2016-02-09</td>\n",
              "      <td>Deadpool</td>\n",
              "      <td>Tim Miller</td>\n",
              "      <td>antihero mercenari marvelcom superhero basedon...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b116155d-fa0d-4294-a1a7-c3a9e30a1a34')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-b116155d-fa0d-4294-a1a7-c3a9e30a1a34 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-b116155d-fa0d-4294-a1a7-c3a9e30a1a34');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "table = pa.parquet.read_table('/content/model.parquet').to_pandas()"
      ],
      "metadata": {
        "id": "5sQlU1ujNBQt"
      },
      "execution_count": 48,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "master_dataset = master_dataset.reset_index()\n",
        "titles = master_dataset['title']\n",
        "indices = pd.Series(master_dataset.index, index=master_dataset['title'])"
      ],
      "metadata": {
        "id": "Z8YlkFPwEitp"
      },
      "execution_count": 49,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def get_recommendations(movie_id_from_db,movie_db):\n",
        "    try:\n",
        "        sim_scores = list(enumerate(movie_db[movie_id_from_db]))\n",
        "        sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
        "        sim_scores = sim_scores[1:15] ## get top 15 Recommendations\n",
        "        \n",
        "        movie_indices = [i[0] for i in sim_scores]\n",
        "        output = master_dataset.iloc[movie_indices]\n",
        "        output.reset_index(inplace=True, drop=True)\n",
        "\n",
        "        response = []\n",
        "        for i in range(len(output)):\n",
        "            response.append({\n",
        "                'movie_title':output['title'].iloc[i],\n",
        "                'movie_release_date':output['release_date'].iloc[i],\n",
        "                'movie_director':output['main_director'].iloc[i],\n",
        "                'google_link':\"https://www.google.com/search?q=\" + '+'.join(output['title'].iloc[i].strip().split())\n",
        "            })\n",
        "        return response\n",
        "    except Exception as e:\n",
        "        print(\"error: \",e)\n",
        "        return []"
      ],
      "metadata": {
        "id": "on_-itFvj-6x"
      },
      "execution_count": 52,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "movie_name = input('Enter a movie Name: ')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "obiDKWO3kPQO",
        "outputId": "c5505e5f-1d1d-445d-e873-d3be67c90e6f"
      },
      "execution_count": 50,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter a movie Name: Toy Story\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "movie_index = titles.to_list().index(movie_name)\n",
        "recommendations = get_recommendations(movie_index,table)"
      ],
      "metadata": {
        "id": "lmp4yv5CFgw6"
      },
      "execution_count": 54,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(f\"{'Movie Title':<40} | {'Director':<20} | {'Release Date':<15}\")\n",
        "print(f\"-\"*80)\n",
        "for recommendation in recommendations:\n",
        "    print(f\"{recommendation['movie_title']:<40} | {recommendation['movie_director']:<20} | {recommendation['movie_release_date']:<15}\")"
      ],
      "metadata": {
        "id": "lIr4jsTWGap5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "70cb630a-a61d-4ad5-86ca-efbf0db5e0c8"
      },
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Movie Title                              | Director             | Release Date   \n",
            "--------------------------------------------------------------------------------\n",
            "Toy Story 2                              | John Lasseter        | 1999-10-30     \n",
            "A Bug's Life                             | John Lasseter        | 1998-11-25     \n",
            "Cars 2                                   | John Lasseter        | 2011-06-11     \n",
            "Cars                                     | John Lasseter        | 2006-06-08     \n",
            "Oliver & Company                         | George Scribner      | 1988-11-18     \n",
            "Toy Story 3                              | Lee Unkrich          | 2010-06-16     \n",
            "Dug's Special Mission                    | Ronnie del Carmen    | 2009-11-09     \n",
            "Leroy & Stitch                           | Roberts Gannaway     | 2006-06-23     \n",
            "Blank Check                              | Rupert Wainwright    | 1994-02-11     \n",
            "Jungle 2 Jungle                          | John Pasquin         | 1997-03-07     \n",
            "The Boxtrolls                            | Anthony Stacchi      | 2014-09-10     \n",
            "Monsters, Inc.                           | Pete Docter          | 2001-11-01     \n",
            "Spirit: Stallion of the Cimarron         | Kelly Asbury         | 2002-05-24     \n",
            "Surf's Up                                | Ash Brannon          | 2007-06-08     \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "g48aHqAimTaX"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}