03-Spark/002.01-Analyze-MovieLens-using-PySpark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9af8f54a-2495-403f-9d66-8c79a44f1aa4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Data Analysis using __PySpark__  \n",
    "*Fun with the __MovieLens__ dataset*  \n",
    "\n",
    "**Part 1: Overview, Starting Spark and Loading the data**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5622fda-b8fa-475a-b09e-b9f7c010bb3b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<font color='green'>__Support for Google Colab__  </font>\n",
    "\n",
    "open this notebook in Colab using the following button:  \n",
    "  \n",
    "<a href=\"https://colab.research.google.com/github/shauryashaurya/learn-data-munging/blob/main/03-Spark/002.01-Analyze-MovieLens-using-PySpark.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>  \n",
    "\n",
    "<font color='green'>uncomment and execute the cell below to setup and run this Spark notebook on Google Colab.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7aeeb078-e887-48f9-986c-1bd229bdbd40",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # SETUP FOR COLAB: select all the lines below and uncomment (CTRL+/ on windows)\n",
    "\n",
    "# # grab spark\n",
    "# # as of 2023-06-23, the latest version is 3.4.1, get the link from Apache Spark's website\n",
    "# ! wget -q https://dlcdn.apache.org/spark/spark-3.4.1/spark-3.4.1-bin-hadoop3.tgz\n",
    "# # unzip spark\n",
    "# !tar xf spark-3.4.1-bin-hadoop3.tgz\n",
    "# # install findspark package\n",
    "# !pip install -q findspark\n",
    "# # Let's download and unzip the MovieLens 25M Dataset as well.\n",
    "# ! mkdir ./../data\n",
    "# ! wget -q https://files.grouplens.org/datasets/movielens/ml-25m.zip\n",
    "# ! unzip ./ml-25m.zip -d ./../data/\n",
    "\n",
    "# # got to provide JAVA_HOME and SPARK_HOME vairables\n",
    "# import os\n",
    "# os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-11-openjdk-amd64\"\n",
    "# # IMPORTANT - UPDATE THE SPARK_HOME PATH BASED ON THE PACKAGE YOU DOWNLOAD\n",
    "# os.environ[\"SPARK_HOME\"] = \"/content/spark-3.4.1-bin-hadoop3\"\n",
    "# ! echo \"DONE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3c19b82-bdab-482f-ac90-7d4ccea755d9",
   "metadata": {},
   "source": [
    "## Overview\n",
    "Features of the PySpark DataFrames most commonly used in data analysis - select, filter, join, groupby, pivot, and windows.  \n",
    "Instead of toy examples and '10 minutes to xx' we load an actual dataset and ask meaningful questions about it.\n",
    "  \n",
    "We'll use the [MovieLens](https://grouplens.org/datasets/movielens/#:~:text=MovieLens%2025M%20Dataset) dataset for these exercises.  \n",
    "This dataset is non trivial and should expand to about __1GB__ on you local disk.  \n",
    "\n",
    "Download and unzip [MovieLens 25M Dataset](https://grouplens.org/datasets/movielens/25m/) for this analysis.\n",
    "\n",
    "Either ensure the data is in ```\"./data/ml-25m\"``` folder or update the path to the data below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cfb4213-4460-4f9a-abe9-f9ce16fb8ee4",
   "metadata": {},
   "source": [
    "**Citation**:  \n",
    "*F. Maxwell Harper and Joseph A. Konstan.* 2015.  \n",
    "The MovieLens Datasets: History and Context.  \n",
    "ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4: 19:1–19:19. <https://doi.org/10.1145/2827872>  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "966dd9f2-ecfa-4a2e-9a96-c776fc287123",
   "metadata": {},
   "source": [
    "You got this.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb27535b-37c8-4e60-926b-d3b8ad681ce5",
   "metadata": {},
   "source": [
    "## Approach\n",
    "\n",
    "The idea is to tackle simple Spark use-cases first and move on to more complex ones.  \n",
    "\n",
    "Starting with simply loading the data into a dataframe, we then perform a data evaluation, some cleanup and finally analysis. We first ask questions based on individual data files, then move on to combining data from multiple files.\n",
    "\n",
    "We are going to try and avoid the more mathematically involved parts of exploratory data analysis - for e.g. statistical analysis on various features etc. - the core focus in the ability to grok pyspark functions and have fun while doing it.  \n",
    "\n",
    "By the end you'd not only have an idea of PySpark, but also how we ask questions and analyze a chunk of data.  \n",
    "\n",
    "_You may also end up with a watch-list to binge on your next weekend._ :)   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fdd1d06-da4c-4b1d-9810-098a40a60dff",
   "metadata": {},
   "source": [
    "## Setup the Spark Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e16d0b93-7d0e-4072-a127-faa6a76eef58",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 1: initialize findspark\n",
    "import findspark\n",
    "\n",
    "findspark.init()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "64c106f7-2578-45c7-b15c-86450312ba0a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.5.0'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 2: import pyspark\n",
    "import pyspark\n",
    "from pyspark.sql import SparkSession\n",
    "\n",
    "pyspark.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "17968984-c899-4148-8d1c-4a2d3bcda06f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 3: Create a spark session\n",
    "\n",
    "# using local[*] to use as many logical cores as available, use 1 when in doubt\n",
    "# 'local[1]' indicates spark on 1 core on the local machine or specify the number of cores needed\n",
    "# use .config(\"spark.some.config.option\", \"some-value\") for additional configuration\n",
    "\n",
    "spark = (\n",
    "    SparkSession.builder.master(\"local[1]\")\n",
    "    .appName(\"Analyzing Movielens Data\")\n",
    "    .getOrCreate()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cacdf372-f5b3-4558-99f8-0cee451f4a76",
   "metadata": {},
   "outputs": [],
   "source": [
    "# spark"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00a3646a-3986-4362-86bf-43716c1327dd",
   "metadata": {},
   "source": [
    "# ...to read and load the data *correctly*\n",
    "\n",
    "This is typically the first problem you need to work out. You'll see.  \n",
    "  \n",
    "If you've downloaded and unzipped the data, you'll see that some of the files are quite large (genome-scores.csv is 400+ Mb, ratings.csv is 600+ Mb).  \n",
    "\n",
    "So before we start loading the data to explore further, let's go through the [readme](https://files.grouplens.org/datasets/movielens/ml-25m-README.html) file to build a strategy for loading and analyzing data without clogging up the system.  \n",
    "\n",
    "In real life, either you'll have to load files in small chunks to work out a strategy or you'll have to rely on defined schema for data.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "540b8fc8-9a6d-4c41-9568-ab2b1e8c349b",
   "metadata": {},
   "source": [
    "## Schema Spec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54f8948a-c6a4-4b69-a608-fe039c55c25b",
   "metadata": {},
   "source": [
    "Here's the list of files (as of Aug 2022) that you get when you unzip the dataset:\n",
    "1. **movies**.csv - list of movies with at least one rating.  \n",
    "    Header: ```movieId,title,genres```  \n",
    "1. **links**.csv - IDs to generate links to the movie listing on imdb.com and themoviedb.org  \n",
    "    Header: ```movieId,imdbId,tmdbId```  \n",
    "1. **ratings**.csv - Each line of this file after the header row represents one rating of one movie by one user.  \n",
    "    Header: ```userId,movieId,rating,timestamp```  \n",
    "1. **tags**.csv - Each line of this file after the header row represents one tag applied to one movie by one user.  \n",
    "    Header: ```userId,movieId,tag,timestamp```  \n",
    "1. Tag Genome: The tag genome contains tag relevance scores for movies. See [this](http://files.grouplens.org/papers/tag_genome.pdf)  \n",
    "\t1. **genome-tags**.csv - A list of tags  \n",
    "    Header: ```tagId,tag```  \n",
    "\t1. **genome-scores**.csv - Each movie in the genome has a relevance score value for every tag in the genome  \n",
    "    Header: ```movieId,tagId,relevance```  \n",
    "1. README.txt - Check out the README.txt for more details about the files.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dfdeecc-5aab-4a18-b401-b45fa2fb7c5b",
   "metadata": {},
   "source": [
    "## Data encoding details"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3666d767-1bc0-4360-bb26-2660d06baabf",
   "metadata": {},
   "source": [
    "From the Readme file, we have the following observations about the data:\n",
    "1. Each file is a CSV with a single header row\n",
    "1. Separator char is ```,```\n",
    "1. Escape char is ```\"```\n",
    "1. Encoding is UTF-8\n",
    "\n",
    "Let's set these options when reading the CSV files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fefa7de8-7e2f-40da-b208-9494a1e2a93e",
   "metadata": {},
   "source": [
    "## Specify the schema for Spark  \n",
    "  \n",
    "Avoid ```inferSchema``` as much as possible, just cleaner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7b290706-59df-45fe-996d-244cfdf78159",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.types import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "be8e7975-a27d-4b5d-a5be-c59a1b1958fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "schema_movies = StructType(\n",
    "    [\n",
    "        StructField(\"movieId\", StringType(), False),\n",
    "        StructField(\"title\", StringType(), False),\n",
    "        StructField(\"genres\", StringType(), True),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0549adb0-1b68-4d45-abef-e1aab9dcc24f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "schema_links = StructType(\n",
    "    [\n",
    "        StructField(\"movieId\", StringType(), False),\n",
    "        StructField(\"imdbId\", StringType(), True),\n",
    "        StructField(\"tmdbId\", StringType(), True),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9f941e2c-8ad6-41f1-9ec1-bf236d2a87df",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "schema_ratings = StructType(\n",
    "    [\n",
    "        StructField(\"userId\", StringType(), False),\n",
    "        StructField(\"movieId\", StringType(), False),\n",
    "        StructField(\"rating\", FloatType(), True),\n",
    "        StructField(\"timestamp\", StringType(), True),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "497ff195-e214-4930-a4ec-94f27705a491",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "schema_tags = StructType(\n",
    "    [\n",
    "        StructField(\"userId\", StringType(), False),\n",
    "        StructField(\"movieId\", StringType(), False),\n",
    "        StructField(\"tag\", StringType(), True),\n",
    "        StructField(\"timestamp\", StringType(), True),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "26066321-48db-46dc-be64-4c86a4ccd9b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "schema_genome_tags = StructType(\n",
    "    [\n",
    "\t\tStructField(\"tagId\", StringType(), False), \n",
    "\t\tStructField(\"tag\", StringType(), False)\n",
    "\t]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d6515e80-2824-41fc-bc4f-bb74fcf7a604",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# using arbitrary precision signed decimals (java.math.BigDecimal) for relevance scores\n",
    "schema_genome_scores = StructType(\n",
    "    [\n",
    "        StructField(\"movieId\", StringType(), False),\n",
    "        StructField(\"tagId\", StringType(), False),\n",
    "        StructField(\"relevance\", DecimalType(), False),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb5dcd2f-42c2-4914-855c-5d2cb5f719ae",
   "metadata": {},
   "source": [
    "## Specify the location of your data  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "942bdb26-547c-49a7-a020-db1167a59079",
   "metadata": {},
   "source": [
    "Change this folder if you are saving the data at the different place"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "105147c9-8ad0-49e9-8b3a-f2c4f651fc4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "datalocation = \"../data/ml-25m/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "71f0b203-23bd-4b1b-955c-66a4dcec4e3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify file names\n",
    "file_path_movies = datalocation + \"movies.csv\"\n",
    "file_path_links = datalocation + \"links.csv\"\n",
    "file_path_ratings = datalocation + \"ratings.csv\"\n",
    "file_path_tags = datalocation + \"tags.csv\"\n",
    "file_path_genome_tags = datalocation + \"genome-tags.csv\"\n",
    "file_path_genome_scores = datalocation + \"genome-scores.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d829fd0-cf6d-49e5-9de1-3dc4b58156bc",
   "metadata": {},
   "source": [
    "## Load the data and review"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de22fc4f-d698-4007-b307-d87b8937341b",
   "metadata": {},
   "source": [
    "Let's load each file in turn and observe, just to get a sense of familiarity with the data.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5c24251-90e4-40b5-bfec-3aba1fa14a7a",
   "metadata": {},
   "source": [
    "### A note on comparing the *method-chaining* syntax between pandas and pyspark  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9450d7a-db73-4e00-986e-ba2a0085d184",
   "metadata": {},
   "source": [
    "Pandas supports that nice \"method chaining\" syntax where you can club everything in parens  \n",
    "and write one operation per line  \n",
    "to do that in spark,  \n",
    "we use the multi-line format - end each line with a space-backslash  \n",
    "and python will continue to add the next line to your single link of code  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b7d809e-c263-4245-af11-861db715d607",
   "metadata": {},
   "source": [
    "The good thing about the pandas syntax is   \n",
    "you can comment a line and the next one is picked up just fine  \n",
    "also you can pipe() things to another variable for debugging or capturing state  \n",
    "commenting in the middle definetely breaks in pyspark.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4999b4d6-7b37-409f-b912-1eaa80ec364a",
   "metadata": {},
   "source": [
    "### Movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7cf70a0d-aecc-4df1-97b5-d2dc387854d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "movies_raw = (\n",
    "    spark.read.format(\"csv\")\n",
    "    .option(\"encoding\", \"UTF-8\")\n",
    "    .option(\"header\", True)\n",
    "    .option(\"sep\", \",\")\n",
    "    .option(\"escape\", '\"')\n",
    "    .schema(schema_movies)\n",
    "    .load(file_path_movies)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1765f792-e655-44be-90bb-1a9858645e5b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+----------------------------------+-------------------------------------------+\n",
      "|movieId|title                             |genres                                     |\n",
      "+-------+----------------------------------+-------------------------------------------+\n",
      "|1      |Toy Story (1995)                  |Adventure|Animation|Children|Comedy|Fantasy|\n",
      "|2      |Jumanji (1995)                    |Adventure|Children|Fantasy                 |\n",
      "|3      |Grumpier Old Men (1995)           |Comedy|Romance                             |\n",
      "|4      |Waiting to Exhale (1995)          |Comedy|Drama|Romance                       |\n",
      "|5      |Father of the Bride Part II (1995)|Comedy                                     |\n",
      "|6      |Heat (1995)                       |Action|Crime|Thriller                      |\n",
      "|7      |Sabrina (1995)                    |Comedy|Romance                             |\n",
      "|8      |Tom and Huck (1995)               |Adventure|Children                         |\n",
      "|9      |Sudden Death (1995)               |Action                                     |\n",
      "|10     |GoldenEye (1995)                  |Action|Adventure|Thriller                  |\n",
      "+-------+----------------------------------+-------------------------------------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Spark collects all transformations needed\n",
    "# and execution doesn't begin until an \"action\" is triggered\n",
    "# \n",
    "# 'show' triggers a partial execution \n",
    "#  'show' - limiting computation (where relevant) to the number of rows you want to display\n",
    "movies_raw.show(10, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50e32f59-a514-4f29-831a-ffd8ccfc49b7",
   "metadata": {},
   "source": [
    "### RDDs and DataFrames  \n",
    "\n",
    "RDDs are the fundamental data structures.\n",
    "DataFrames are high level entities that operate on RDDs.\n",
    "DataFrames have lots of underlying optimization built in, so when DataFrame code gets translated to RDDs, it's optimal.\n",
    "Prefer DataFrames unless RDDs are absolutely needed - cleaner API, better performance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cae9ba67-aa5a-467d-b436-8ef26bd17c8d",
   "metadata": {},
   "source": [
    "References:\n",
    "* [RDD Actions](https://spark.apache.org/docs/latest/rdd-programming-guide.html#actions) - this stuff triggers execution\n",
    "* [RDD Transformations](https://spark.apache.org/docs/latest/rdd-programming-guide.html#transformations) - these are accumulated in the DAG and executed in order when an action is triggered\n",
    "* [DataFrames: Quickstart](https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_df.html#Viewing-Data) - This is what we'll leverage in the workshop"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ec8b41e-223a-456c-866d-c6cfc698de8d",
   "metadata": {},
   "source": [
    "Onwards with loading and viewing the rest of the data files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f308da29-da5f-418a-a87d-c788a8e379f3",
   "metadata": {},
   "source": [
    "### Links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c19d22ac-3d6a-4b17-965a-581651021eae",
   "metadata": {},
   "outputs": [],
   "source": [
    "links_raw = (\n",
    "    spark.read.format(\"csv\")\n",
    "    .option(\"encoding\", \"UTF-8\")\n",
    "    .option(\"header\", True)\n",
    "    .option(\"sep\", \",\")\n",
    "    .option(\"escape\", '\"')\n",
    "    .schema(schema_links)\n",
    "    .load(file_path_links)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "007f7c67-1c72-4b47-81c7-2983a1a530f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------+------+\n",
      "|movieId|imdbId |tmdbId|\n",
      "+-------+-------+------+\n",
      "|1      |0114709|862   |\n",
      "|2      |0113497|8844  |\n",
      "|3      |0113228|15602 |\n",
      "|4      |0114885|31357 |\n",
      "|5      |0113041|11862 |\n",
      "|6      |0113277|949   |\n",
      "|7      |0114319|11860 |\n",
      "|8      |0112302|45325 |\n",
      "|9      |0114576|9091  |\n",
      "|10     |0113189|710   |\n",
      "+-------+-------+------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "links_raw.show(10, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5279e591-5857-461b-b803-dd2caf04c672",
   "metadata": {},
   "source": [
    "### Ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "89ef74dd-dae2-4bdb-8414-0577c4352694",
   "metadata": {},
   "outputs": [],
   "source": [
    "ratings_raw = (\n",
    "    spark.read.format(\"csv\")\n",
    "    .option(\"encoding\", \"UTF-8\")\n",
    "    .option(\"header\", True)\n",
    "    .option(\"sep\", \",\")\n",
    "    .option(\"escape\", '\"')\n",
    "    .schema(schema_ratings)\n",
    "    .load(file_path_ratings)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e3c39e6d-14e9-4bd4-9914-27a701d5194d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-------+------+----------+\n",
      "|userId|movieId|rating|timestamp |\n",
      "+------+-------+------+----------+\n",
      "|1     |296    |5.0   |1147880044|\n",
      "|1     |306    |3.5   |1147868817|\n",
      "|1     |307    |5.0   |1147868828|\n",
      "|1     |665    |5.0   |1147878820|\n",
      "|1     |899    |3.5   |1147868510|\n",
      "|1     |1088   |4.0   |1147868495|\n",
      "|1     |1175   |3.5   |1147868826|\n",
      "|1     |1217   |3.5   |1147878326|\n",
      "|1     |1237   |5.0   |1147868839|\n",
      "|1     |1250   |4.0   |1147868414|\n",
      "+------+-------+------+----------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ratings_raw.show(10, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "927eb31c-f4ba-417f-b0a8-d8f9e6136def",
   "metadata": {},
   "source": [
    "### Tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "215a080f-600f-4c7e-abd9-1dc1afdc5206",
   "metadata": {},
   "outputs": [],
   "source": [
    "tags_raw = (\n",
    "    spark.read.format(\"csv\")\n",
    "    .option(\"encoding\", \"UTF-8\")\n",
    "    .option(\"header\", True)\n",
    "    .option(\"sep\", \",\")\n",
    "    .option(\"escape\", '\"')\n",
    "    .schema(schema_tags)\n",
    "    .load(file_path_tags)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7af4bd78-f784-4dd9-bbb2-851f506585b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-------+-----------------------+----------+\n",
      "|userId|movieId|tag                    |timestamp |\n",
      "+------+-------+-----------------------+----------+\n",
      "|3     |260    |classic                |1439472355|\n",
      "|3     |260    |sci-fi                 |1439472256|\n",
      "|4     |1732   |dark comedy            |1573943598|\n",
      "|4     |1732   |great dialogue         |1573943604|\n",
      "|4     |7569   |so bad it's good       |1573943455|\n",
      "|4     |44665  |unreliable narrators   |1573943619|\n",
      "|4     |115569 |tense                  |1573943077|\n",
      "|4     |115713 |artificial intelligence|1573942979|\n",
      "|4     |115713 |philosophical          |1573943033|\n",
      "|4     |115713 |tense                  |1573943042|\n",
      "+------+-------+-----------------------+----------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tags_raw.show(10, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c08fa7a-4945-440d-ba3f-16c8467c3034",
   "metadata": {},
   "source": [
    "### Tag Genome"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a138fde7-0acf-4486-93ae-18c91a476e20",
   "metadata": {},
   "outputs": [],
   "source": [
    "genome_tags_raw = (\n",
    "    spark.read.format(\"csv\")\n",
    "    .option(\"encoding\", \"UTF-8\")\n",
    "    .option(\"header\", True)\n",
    "    .option(\"sep\", \",\")\n",
    "    .option(\"escape\", '\"')\n",
    "    .schema(schema_genome_tags)\n",
    "    .load(file_path_genome_tags)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3c5cb365-82b7-4c81-a58d-05887645bec6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+------------+\n",
      "|tagId|tag         |\n",
      "+-----+------------+\n",
      "|1    |007         |\n",
      "|2    |007 (series)|\n",
      "|3    |18th century|\n",
      "|4    |1920s       |\n",
      "|5    |1930s       |\n",
      "|6    |1950s       |\n",
      "|7    |1960s       |\n",
      "|8    |1970s       |\n",
      "|9    |1980s       |\n",
      "|10   |19th century|\n",
      "+-----+------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "genome_tags_raw.show(10, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b6ec100-7314-43d7-8222-f8ea85619bfc",
   "metadata": {},
   "source": [
    "### Tag Genome Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "70c13d92-ed65-47f8-aec5-fb1e7af29d99",
   "metadata": {},
   "outputs": [],
   "source": [
    "genome_scores_raw = (\n",
    "    spark.read.format(\"csv\")\n",
    "    .option(\"encoding\", \"UTF-8\")\n",
    "    .option(\"header\", True)\n",
    "    .option(\"sep\", \",\")\n",
    "    .option(\"escape\", '\"')\n",
    "    .schema(schema_genome_scores)\n",
    "    .load(file_path_genome_scores)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d9bd8562-5cbc-4a28-a09b-c76ed6d34605",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-----+---------+\n",
      "|movieId|tagId|relevance|\n",
      "+-------+-----+---------+\n",
      "|1      |1    |0        |\n",
      "|1      |2    |0        |\n",
      "|1      |3    |0        |\n",
      "|1      |4    |0        |\n",
      "|1      |5    |0        |\n",
      "|1      |6    |0        |\n",
      "|1      |7    |0        |\n",
      "|1      |8    |0        |\n",
      "|1      |9    |0        |\n",
      "|1      |10   |0        |\n",
      "+-------+-----+---------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "genome_scores_raw.show(10, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0fbf647-a271-4be4-9f48-4d0bd2c81efb",
   "metadata": {},
   "source": [
    "# Clear cache and stop the spark cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "fade3b28-51c0-4744-9907-72909ae2ccdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# clear cache\n",
    "spark.catalog.clearCache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "a054984c-dda3-48c6-b1a4-f26922060ca1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# stop spark\n",
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b40e00a8-51be-480f-a5e7-4b6febc5f467",
   "metadata": {},
   "source": [
    "# Insights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70cbdaa4-5a99-408d-8805-54d7cd022a33",
   "metadata": {},
   "source": [
    "In most cases, prefer loading files in a just-in-time manner to conserve memory and computing resources.  \n",
    "\n",
    "IRL you'd load a file only when needed - big data means big memory, big processing, big everything but it doesn't mean big bull in a china shop. Brute force is rarely going to be the answer - you've got to learn to be lean in your approach. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dc204d8-cb3d-4ad0-a3e1-990629ad949e",
   "metadata": {},
   "source": [
    "# Next"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddbd9e40-5fc8-4d92-8641-713080ef7a02",
   "metadata": {},
   "source": [
    "We will next start analysing the data through a series of data analysis exercises.  \n",
    "First set of exercises work around the tags.csv data in the MovieLens Dataset."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}