02-Pandas/02.01-Data-Wrangling-with-MovieLens-and-Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7e9f4107-83a1-4cf6-8340-9f9454b57eae",
   "metadata": {},
   "source": [
    "# Data Wrangling in Python  \n",
    "*__[Pandas](https://pandas.pydata.org/)__ with the __MovieLens__ dataset*  \n",
    "\n",
    "**Part 1: Getting Started, Load the MovieLens dataset**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f21bef00-086e-44fd-b516-01016a08432d",
   "metadata": {},
   "source": [
    "### <font color='green'>__Support for Google Colab__  </font>  \n",
    "    \n",
    "open this notebook in Colab using the following button:  \n",
    "  \n",
    "<a href=\"https://colab.research.google.com/github/shauryashaurya/learn-data-munging/blob/main/02-Pandas/02.01-Data-Wrangling-with-MovieLens-and-Pandas.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>  \n",
    "\n",
    "  \n",
    "<font color='green'>uncomment and execute the cell below to setup and run this notebook on Google Colab.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cbba6f93-2b10-45bf-9975-a41d30ffd219",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # SETUP FOR COLAB: select all the lines below and uncomment (CTRL+/ on windows)\n",
    "# # Let's download and unzip the Small MovieLens Dataset\n",
    "# ! mkdir ./../data\n",
    "# ! wget -q https://files.grouplens.org/datasets/movielens/ml-latest-small.zip\n",
    "# ! unzip ./ml-latest-small.zip -d ./../data/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "868129c9-cedd-4513-997b-53f9427b6f1a",
   "metadata": {},
   "source": [
    "### Get the _Small_ MovieLens Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67afc9a9-31b8-483d-89c1-699867c8700b",
   "metadata": {},
   "source": [
    "We'll use the [small MovieLens dataset](https://grouplens.org/datasets/movielens/#:~:text=Small%3A%20100%2C000%20ratings%20and%203%2C600%20tag%20applications) here.\n",
    "\n",
    "Download it and unzip to the data folder under the name `ml-latest-small`.\n",
    "\n",
    "This dataset expands to about 3.2 MB on your local disk. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb164ece-2d5e-4451-b455-24fe2add5bee",
   "metadata": {},
   "source": [
    "# Locate the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4a045b14-f578-47ad-9972-a3f5aaba0911",
   "metadata": {},
   "outputs": [],
   "source": [
    "datalocation = \"./../data/ml-latest-small/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0428c276-5ef7-4bde-addf-3bb79337783d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify file names\n",
    "file_path_movies = datalocation + \"movies.csv\"\n",
    "file_path_links = datalocation + \"links.csv\"\n",
    "file_path_ratings = datalocation + \"ratings.csv\"\n",
    "file_path_tags = datalocation + \"tags.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b518bef-b08d-40ff-b63b-7395f06de060",
   "metadata": {},
   "source": [
    "# Setup Pandas and Numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "53c0617c-fe63-4ac9-8131-dc20a9220e2b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numpy version:  1.26.2\n",
      "pandas version:  2.1.4\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "print(\"numpy version: \", np.__version__)\n",
    "print(\"pandas version: \", pd.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96574358-90c0-4eed-8f4e-1dd5692bd60c",
   "metadata": {},
   "source": [
    "# Load the dataset(s)\n",
    "\n",
    "From the ```README.txt``` file in the small MovieLens dataset:\n",
    "The dataset files are written as [**comma-separated values**](http://en.wikipedia.org/wiki/Comma-separated_values) files with a **single header row**. Columns that contain commas (`,`) are **escaped using double-quotes (`\"`)**. These files are encoded as **UTF-8**. If accented characters in movie titles or tag values (e.g. Misérables, Les (1995)) display incorrectly, make sure that any program reading the data, such as a text editor, terminal, or script, is configured for UTF-8.\n",
    "\n",
    "So, we specify:\n",
    "* Separator - ```,```\n",
    "* Escape Character - ```\"```\n",
    "* Encoding - ```UTF-8```\n",
    "\n",
    "Often this is called the **dialect** of the CSV file.\n",
    "These dialects vary often, so need our attention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "51505e23-1133-4302-beaf-1690963ec807",
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_separator = \",\"\n",
    "csv_escapechar = '\"'\n",
    "csv_encoding = \"utf-8\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "503f86f0-c81b-4f2c-80ab-090afbe049d2",
   "metadata": {},
   "source": [
    "## Movies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2583f6a3-72c2-4414-9c12-b56ac69fb00a",
   "metadata": {},
   "source": [
    "Let's specify the [-  ```dtypes```  ](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes) of each of the columns in the movies file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c9a1e502-96a1-4ae1-91e9-cba5e695e7a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# schema, inferred from the README.txt file\n",
    "movies_schema = {\"movieId\": \"Int32\", \"title\": \"string\", \"genres\": \"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39569d7e-8c3c-44f2-a40b-776adc33d13d",
   "metadata": {},
   "source": [
    "Two of the columns are [strings of text](https://pandas.pydata.org/docs/user_guide/text.html#working-with-text-data). Pandas may treat those as ```object```, but we wanted to use the [```pandas.StringDType```](https://pandas.pydata.org/docs/reference/api/pandas.StringDtype.html#pandas-stringdtype) here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "dd298b5c-5105-45fb-84a6-60ab177eed50",
   "metadata": {},
   "outputs": [],
   "source": [
    "# gives an error.\n",
    "# movies = pd.read_csv(file_path_movies, \n",
    "# dtype=movies_schema, \n",
    "# sep=csv_separator, \n",
    "# escapechar=csv_escapechar, \n",
    "# encoding=csv_encoding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9aa4c109-2eee-46ac-8177-8dd9caa35eff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we need the quote character, not the escape character\n",
    "# just to keep things readable, let's create another variable\n",
    "csv_quotechar = csv_escapechar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5c90ad76-3592-4ed7-8c9f-fc67b94648d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "movies = pd.read_csv(\n",
    "    file_path_movies,\n",
    "    dtype=movies_schema,\n",
    "    sep=csv_separator,\n",
    "    quotechar=csv_quotechar,\n",
    "    encoding=csv_encoding,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f1b032d0-cd24-4aa7-8d98-422ba8d4900e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>Adventure|Children|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale (1995)</td>\n",
       "      <td>Comedy|Drama|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II (1995)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>Heat (1995)</td>\n",
       "      <td>Action|Crime|Thriller</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>Sabrina (1995)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>Tom and Huck (1995)</td>\n",
       "      <td>Adventure|Children</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>Sudden Death (1995)</td>\n",
       "      <td>Action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>GoldenEye (1995)</td>\n",
       "      <td>Action|Adventure|Thriller</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>American President, The (1995)</td>\n",
       "      <td>Comedy|Drama|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>Dracula: Dead and Loving It (1995)</td>\n",
       "      <td>Comedy|Horror</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>Balto (1995)</td>\n",
       "      <td>Adventure|Animation|Children</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>Nixon (1995)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>Cutthroat Island (1995)</td>\n",
       "      <td>Action|Adventure|Romance</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    movieId                               title  \\\n",
       "0         1                    Toy Story (1995)   \n",
       "1         2                      Jumanji (1995)   \n",
       "2         3             Grumpier Old Men (1995)   \n",
       "3         4            Waiting to Exhale (1995)   \n",
       "4         5  Father of the Bride Part II (1995)   \n",
       "5         6                         Heat (1995)   \n",
       "6         7                      Sabrina (1995)   \n",
       "7         8                 Tom and Huck (1995)   \n",
       "8         9                 Sudden Death (1995)   \n",
       "9        10                    GoldenEye (1995)   \n",
       "10       11      American President, The (1995)   \n",
       "11       12  Dracula: Dead and Loving It (1995)   \n",
       "12       13                        Balto (1995)   \n",
       "13       14                        Nixon (1995)   \n",
       "14       15             Cutthroat Island (1995)   \n",
       "\n",
       "                                         genres  \n",
       "0   Adventure|Animation|Children|Comedy|Fantasy  \n",
       "1                    Adventure|Children|Fantasy  \n",
       "2                                Comedy|Romance  \n",
       "3                          Comedy|Drama|Romance  \n",
       "4                                        Comedy  \n",
       "5                         Action|Crime|Thriller  \n",
       "6                                Comedy|Romance  \n",
       "7                            Adventure|Children  \n",
       "8                                        Action  \n",
       "9                     Action|Adventure|Thriller  \n",
       "10                         Comedy|Drama|Romance  \n",
       "11                                Comedy|Horror  \n",
       "12                 Adventure|Animation|Children  \n",
       "13                                        Drama  \n",
       "14                     Action|Adventure|Romance  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show the first 15 lines\n",
    "movies.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82c4106b-64cb-4512-94f7-94bc37bf1abb",
   "metadata": {},
   "source": [
    "Look at Row #10 - \"American President, The (1995)\" - so pandas seems to have correctly interpreted the quotation marks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76f26c28-7c33-4615-9258-b066884735fa",
   "metadata": {},
   "source": [
    "For now we'll keep things simple and let pandas give us an index.  \n",
    "In some cases it would be interesting to use the ```movieId``` column [as the index](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html#pandas.read_csv:~:text=index_colHashable%2C%20Sequence%20of%20Hashable%20or%20False%2C%20optional).  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "521d5f6a-1be8-496b-9d75-ebd120a3475f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "movieId             Int32\n",
       "title      string[python]\n",
       "genres     string[python]\n",
       "dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data types of each column\n",
    "movies.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21309cbb-2489-4b4d-96cc-6532de8cc6bb",
   "metadata": {},
   "source": [
    "Just for practice, let's load the other datasets too..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4ca7df0-19e3-4535-99ec-efe5dd1c6dfd",
   "metadata": {},
   "source": [
    "## Links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c2e6aa7b-4f71-4e33-95fb-a74b9014368d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# schema, inferred from the README.txt file\n",
    "# load imdbId,tmdbId as strings because the are a part of a URL.\n",
    "#   IMDB: http://www.imdb.com/title/imdbId/\n",
    "#   TMDB: https://www.themoviedb.org/movie/tmdbId\n",
    "links_schema = {\"movieId\": \"Int32\", \"imdbId\": \"string\", \"tmdbId\": \"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8e0390f3-e09e-4a66-a335-23e71a15ba25",
   "metadata": {},
   "outputs": [],
   "source": [
    "links = pd.read_csv(\n",
    "    file_path_links,\n",
    "    dtype=links_schema,\n",
    "    sep=csv_separator,\n",
    "    quotechar=csv_quotechar,\n",
    "    encoding=csv_encoding,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ebaa2447-2217-46e3-968f-6da807d7fb1a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>imdbId</th>\n",
       "      <th>tmdbId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0114709</td>\n",
       "      <td>862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0113497</td>\n",
       "      <td>8844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0113228</td>\n",
       "      <td>15602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0114885</td>\n",
       "      <td>31357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0113041</td>\n",
       "      <td>11862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>0113277</td>\n",
       "      <td>949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>0114319</td>\n",
       "      <td>11860</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>0112302</td>\n",
       "      <td>45325</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>0114576</td>\n",
       "      <td>9091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>0113189</td>\n",
       "      <td>710</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>0112346</td>\n",
       "      <td>9087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>0112896</td>\n",
       "      <td>12110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>0112453</td>\n",
       "      <td>21032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>0113987</td>\n",
       "      <td>10858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>0112760</td>\n",
       "      <td>1408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    movieId   imdbId tmdbId\n",
       "0         1  0114709    862\n",
       "1         2  0113497   8844\n",
       "2         3  0113228  15602\n",
       "3         4  0114885  31357\n",
       "4         5  0113041  11862\n",
       "5         6  0113277    949\n",
       "6         7  0114319  11860\n",
       "7         8  0112302  45325\n",
       "8         9  0114576   9091\n",
       "9        10  0113189    710\n",
       "10       11  0112346   9087\n",
       "11       12  0112896  12110\n",
       "12       13  0112453  21032\n",
       "13       14  0113987  10858\n",
       "14       15  0112760   1408"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links.head(15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "035a2d76-69ab-4f49-a6d2-262c84e14257",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "movieId             Int32\n",
       "imdbId     string[python]\n",
       "tmdbId     string[python]\n",
       "dtype: object"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25e7b872-1db9-49c9-9288-d327cae2fdf1",
   "metadata": {},
   "source": [
    "## Ratings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "342bbe4a-c904-4313-8b31-252dc87cda76",
   "metadata": {},
   "source": [
    "Reading through the ```README``` file:  \n",
    "Ratings are made on a 5-star scale, with half-star increments (0.5 stars - 5.0 stars).  \n",
    "Timestamps represent seconds since midnight Coordinated Universal Time (UTC) of January 1, 1970.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dee402a-94df-4afb-b691-2c1397ea3f36",
   "metadata": {},
   "source": [
    "Ooh! Ooh! We got our first [DateTime](https://pandas.pydata.org/docs/reference/api/pandas.DatetimeTZDtype.html#pandas.DatetimeTZDtype)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "672561a7-1b7f-4168-aaec-85245faec9b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# schema, inferred from the README.txt file\n",
    "# read timestamps as integers then convert to dates later.\n",
    "\n",
    "ratings_schema = {\n",
    "    \"userId\": \"Int32\",\n",
    "    \"movieId\": \"Int32\",\n",
    "    \"rating\": \"Float32\",\n",
    "    \"timestamp\": \"Int64\",\n",
    "}\n",
    "#"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5d102b11-3aca-43ef-977a-21c67ca7f610",
   "metadata": {},
   "outputs": [],
   "source": [
    "ratings = pd.read_csv(\n",
    "    file_path_ratings,\n",
    "    dtype=ratings_schema,\n",
    "    sep=csv_separator,\n",
    "    quotechar=csv_quotechar,\n",
    "    encoding=csv_encoding,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5f36be99-fb8b-48d4-897a-d1076092022c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964981247</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964983815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964982931</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating  timestamp\n",
       "0       1        1     4.0  964982703\n",
       "1       1        3     4.0  964981247\n",
       "2       1        6     4.0  964982224\n",
       "3       1       47     5.0  964983815\n",
       "4       1       50     5.0  964982931"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5fda8643-8179-4e79-9c56-663a66830e9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now let's add a datetime column that we derive from the raw timestamp\n",
    "ratings[\"datetime\"] = pd.to_datetime(ratings[\"timestamp\"], unit=\"s\", utc=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3c1a2a42-ed8b-4a45-a90c-a8e97d5b7cbd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId                    Int32\n",
       "movieId                   Int32\n",
       "rating                  Float32\n",
       "timestamp                 Int64\n",
       "datetime     datetime64[s, UTC]\n",
       "dtype: object"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7b57798-6cab-47b1-8cad-da71d7735d41",
   "metadata": {},
   "source": [
    "#### ```pandas.Series.dt.date```  \n",
    "Let's [extract the dates](https://pandas.pydata.org/docs/reference/api/pandas.Series.dt.date.html#pandas-series-dt-date) into a new column"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "039e6b08-604e-4095-aad4-c227a27a5362",
   "metadata": {},
   "source": [
    "...just in case you are wondering what's that [```dt``` part](https://github.com/pandas-dev/pandas/blob/9e1096e8373bc99675fd1b3490cfb7cf26041395/pandas/core/series.py#L5777C1-L5777C2), and want to dive into the code where it's defined as a [CachedAccessor](https://github.com/pandas-dev/pandas/blob/9e1096e8373bc99675fd1b3490cfb7cf26041395/pandas/core/accessor.py#L196) for ```datetimelike``` values in ```pandas/core/accessor.py```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "aa55cc44-f2f2-435f-ab47-2553bbbdcc2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "ratings[\"date\"] = ratings[\"datetime\"].dt.date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b634931a-1b8f-48d2-a399-b932dcba0cd7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982703</td>\n",
       "      <td>2000-07-30 18:45:03+00:00</td>\n",
       "      <td>2000-07-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964981247</td>\n",
       "      <td>2000-07-30 18:20:47+00:00</td>\n",
       "      <td>2000-07-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982224</td>\n",
       "      <td>2000-07-30 18:37:04+00:00</td>\n",
       "      <td>2000-07-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964983815</td>\n",
       "      <td>2000-07-30 19:03:35+00:00</td>\n",
       "      <td>2000-07-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964982931</td>\n",
       "      <td>2000-07-30 18:48:51+00:00</td>\n",
       "      <td>2000-07-30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating  timestamp                  datetime        date\n",
       "0       1        1     4.0  964982703 2000-07-30 18:45:03+00:00  2000-07-30\n",
       "1       1        3     4.0  964981247 2000-07-30 18:20:47+00:00  2000-07-30\n",
       "2       1        6     4.0  964982224 2000-07-30 18:37:04+00:00  2000-07-30\n",
       "3       1       47     5.0  964983815 2000-07-30 19:03:35+00:00  2000-07-30\n",
       "4       1       50     5.0  964982931 2000-07-30 18:48:51+00:00  2000-07-30"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b8a743c-d528-4e02-8b3f-4658951b7dc1",
   "metadata": {},
   "source": [
    "Niiice!  \n",
    "Wait, let's check the data types once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ed38a86b-2820-43e3-9cd5-e1752e481f54",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId                    Int32\n",
       "movieId                   Int32\n",
       "rating                  Float32\n",
       "timestamp                 Int64\n",
       "datetime     datetime64[s, UTC]\n",
       "date                     object\n",
       "dtype: object"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c17bfb7b-4963-4520-bb35-47b859392b42",
   "metadata": {},
   "source": [
    "We'd prefer if date was a datetime type as well.  \n",
    "Let's prepare the date column again, wrapping it in ```pd.to_datetime()```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2b2d2175-123f-4866-83e9-678ea4655e74",
   "metadata": {},
   "outputs": [],
   "source": [
    "ratings[\"date\"] = pd.to_datetime(ratings[\"datetime\"].dt.date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3eeec631-6ec1-45a4-a1a9-8f25d691c2b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId                    Int32\n",
       "movieId                   Int32\n",
       "rating                  Float32\n",
       "timestamp                 Int64\n",
       "datetime     datetime64[s, UTC]\n",
       "date             datetime64[ns]\n",
       "dtype: object"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the data types again\n",
    "ratings.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e558931-f805-49c1-acff-3bc99c80418d",
   "metadata": {},
   "source": [
    "Ah! much better.  \n",
    "Why you say?  \n",
    "We could easily extract and manipulate dates this way.  \n",
    "for e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e9207f2d-cf0d-41b9-bc43-8eefce4ab1f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract the day, month and year of each rating\n",
    "ratings[\"day\"] = ratings[\"date\"].dt.day\n",
    "ratings[\"month\"] = ratings[\"date\"].dt.month\n",
    "ratings[\"year\"] = ratings[\"date\"].dt.year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "89f4fab2-5691-4bf2-89cd-aef1c90dcd18",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId                    Int32\n",
       "movieId                   Int32\n",
       "rating                  Float32\n",
       "timestamp                 Int64\n",
       "datetime     datetime64[s, UTC]\n",
       "date             datetime64[ns]\n",
       "day                       int32\n",
       "month                     int32\n",
       "year                      int32\n",
       "dtype: object"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c962401-b2e3-4678-911d-8a9cd9ac9064",
   "metadata": {},
   "source": [
    "very clean!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1fa35971-7b52-44ef-86d8-d413e9924420",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982703</td>\n",
       "      <td>2000-07-30 18:45:03+00:00</td>\n",
       "      <td>2000-07-30</td>\n",
       "      <td>30</td>\n",
       "      <td>7</td>\n",
       "      <td>2000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964981247</td>\n",
       "      <td>2000-07-30 18:20:47+00:00</td>\n",
       "      <td>2000-07-30</td>\n",
       "      <td>30</td>\n",
       "      <td>7</td>\n",
       "      <td>2000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982224</td>\n",
       "      <td>2000-07-30 18:37:04+00:00</td>\n",
       "      <td>2000-07-30</td>\n",
       "      <td>30</td>\n",
       "      <td>7</td>\n",
       "      <td>2000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964983815</td>\n",
       "      <td>2000-07-30 19:03:35+00:00</td>\n",
       "      <td>2000-07-30</td>\n",
       "      <td>30</td>\n",
       "      <td>7</td>\n",
       "      <td>2000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964982931</td>\n",
       "      <td>2000-07-30 18:48:51+00:00</td>\n",
       "      <td>2000-07-30</td>\n",
       "      <td>30</td>\n",
       "      <td>7</td>\n",
       "      <td>2000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating  timestamp                  datetime       date  \\\n",
       "0       1        1     4.0  964982703 2000-07-30 18:45:03+00:00 2000-07-30   \n",
       "1       1        3     4.0  964981247 2000-07-30 18:20:47+00:00 2000-07-30   \n",
       "2       1        6     4.0  964982224 2000-07-30 18:37:04+00:00 2000-07-30   \n",
       "3       1       47     5.0  964983815 2000-07-30 19:03:35+00:00 2000-07-30   \n",
       "4       1       50     5.0  964982931 2000-07-30 18:48:51+00:00 2000-07-30   \n",
       "\n",
       "   day  month  year  \n",
       "0   30      7  2000  \n",
       "1   30      7  2000  \n",
       "2   30      7  2000  \n",
       "3   30      7  2000  \n",
       "4   30      7  2000  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae72f8b9-6d60-47f6-a9c3-33a21c985bc1",
   "metadata": {},
   "source": [
    "Let's do this for the Tags data set too - just for practice."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96f67371-838c-48fd-967d-c2d9c9c210f8",
   "metadata": {},
   "source": [
    "\n",
    "## Tags"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a9e1cbe-bd07-4926-b770-b7e476ed09e3",
   "metadata": {},
   "source": [
    "From ```README```:  \n",
    "Tags are user-generated metadata about movies. Each tag is typically a single word or short phrase. The meaning, value, and purpose of a particular tag is determined by each user.  \n",
    "  \n",
    "Timestamps represent seconds since midnight Coordinated Universal Time (UTC) of January 1, 1970."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "29e98718-59dd-4682-87bd-38019d1a792c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# schema, inferred from the README.txt file\n",
    "# read timestamps as integers then convert to dates later.\n",
    "# userId,movieId,tag,timestamp\n",
    "tags_schema = {\n",
    "    \"userId\": \"Int32\",\n",
    "    \"movieId\": \"Int32\",\n",
    "    \"tag\": \"string\",\n",
    "    \"timestamp\": \"Int64\",\n",
    "}\n",
    "#"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "823b2316-bdc9-4081-b903-c67b2cac472d",
   "metadata": {},
   "outputs": [],
   "source": [
    "tags = pd.read_csv(\n",
    "    file_path_tags,\n",
    "    dtype=tags_schema,\n",
    "    sep=csv_separator,\n",
    "    quotechar=csv_quotechar,\n",
    "    encoding=csv_encoding,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "8c5bb70a-e903-4962-b4e9-abc7521937ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>tag</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>60756</td>\n",
       "      <td>funny</td>\n",
       "      <td>1445714994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>60756</td>\n",
       "      <td>Highly quotable</td>\n",
       "      <td>1445714996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>60756</td>\n",
       "      <td>will ferrell</td>\n",
       "      <td>1445714992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>89774</td>\n",
       "      <td>Boxing story</td>\n",
       "      <td>1445715207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>89774</td>\n",
       "      <td>MMA</td>\n",
       "      <td>1445715200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId              tag   timestamp\n",
       "0       2    60756            funny  1445714994\n",
       "1       2    60756  Highly quotable  1445714996\n",
       "2       2    60756     will ferrell  1445714992\n",
       "3       2    89774     Boxing story  1445715207\n",
       "4       2    89774              MMA  1445715200"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb95a477-114e-4372-9119-1ffc0c18de23",
   "metadata": {},
   "source": [
    "just like before let's add a more readable ```datetime``` column here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "91bbfb1e-95b3-4448-a639-5a06f625b799",
   "metadata": {},
   "outputs": [],
   "source": [
    "tags[\"datetime\"] = pd.to_datetime(tags[\"timestamp\"], unit=\"s\", utc=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "52c21c7a-bf91-417f-bcb4-6f86e86930ce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId                    Int32\n",
       "movieId                   Int32\n",
       "tag              string[python]\n",
       "timestamp                 Int64\n",
       "datetime     datetime64[s, UTC]\n",
       "dtype: object"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "2de8efdd-9f30-46b5-84b5-195ab949c4f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract date into a new column\n",
    "tags[\"date\"] = pd.to_datetime(tags[\"datetime\"].dt.date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "702e7d50-0b07-4062-929c-e292aa57399f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId                    Int32\n",
       "movieId                   Int32\n",
       "tag              string[python]\n",
       "timestamp                 Int64\n",
       "datetime     datetime64[s, UTC]\n",
       "date             datetime64[ns]\n",
       "dtype: object"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "1723c691-f631-4550-b4f5-cb2f787b56ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>tag</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>60756</td>\n",
       "      <td>funny</td>\n",
       "      <td>1445714994</td>\n",
       "      <td>2015-10-24 19:29:54+00:00</td>\n",
       "      <td>2015-10-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>60756</td>\n",
       "      <td>Highly quotable</td>\n",
       "      <td>1445714996</td>\n",
       "      <td>2015-10-24 19:29:56+00:00</td>\n",
       "      <td>2015-10-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>60756</td>\n",
       "      <td>will ferrell</td>\n",
       "      <td>1445714992</td>\n",
       "      <td>2015-10-24 19:29:52+00:00</td>\n",
       "      <td>2015-10-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>89774</td>\n",
       "      <td>Boxing story</td>\n",
       "      <td>1445715207</td>\n",
       "      <td>2015-10-24 19:33:27+00:00</td>\n",
       "      <td>2015-10-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>89774</td>\n",
       "      <td>MMA</td>\n",
       "      <td>1445715200</td>\n",
       "      <td>2015-10-24 19:33:20+00:00</td>\n",
       "      <td>2015-10-24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId              tag   timestamp                  datetime  \\\n",
       "0       2    60756            funny  1445714994 2015-10-24 19:29:54+00:00   \n",
       "1       2    60756  Highly quotable  1445714996 2015-10-24 19:29:56+00:00   \n",
       "2       2    60756     will ferrell  1445714992 2015-10-24 19:29:52+00:00   \n",
       "3       2    89774     Boxing story  1445715207 2015-10-24 19:33:27+00:00   \n",
       "4       2    89774              MMA  1445715200 2015-10-24 19:33:20+00:00   \n",
       "\n",
       "        date  \n",
       "0 2015-10-24  \n",
       "1 2015-10-24  \n",
       "2 2015-10-24  \n",
       "3 2015-10-24  \n",
       "4 2015-10-24  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6eb6c9f-d156-48fa-801c-754e0c899aac",
   "metadata": {},
   "source": [
    "umm... go nuts.  \n",
    "Extract the day, month and year from date, because why not?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "64f6aeb1-3a1b-46da-be87-292161d9d999",
   "metadata": {},
   "outputs": [],
   "source": [
    "tags[\"day\"] = tags[\"date\"].dt.day\n",
    "tags[\"month\"] = tags[\"date\"].dt.month\n",
    "tags[\"year\"] = tags[\"date\"].dt.year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "1618bd04-750f-499a-a635-8415854b17c8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>tag</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>60756</td>\n",
       "      <td>funny</td>\n",
       "      <td>1445714994</td>\n",
       "      <td>2015-10-24 19:29:54+00:00</td>\n",
       "      <td>2015-10-24</td>\n",
       "      <td>24</td>\n",
       "      <td>10</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>60756</td>\n",
       "      <td>Highly quotable</td>\n",
       "      <td>1445714996</td>\n",
       "      <td>2015-10-24 19:29:56+00:00</td>\n",
       "      <td>2015-10-24</td>\n",
       "      <td>24</td>\n",
       "      <td>10</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>60756</td>\n",
       "      <td>will ferrell</td>\n",
       "      <td>1445714992</td>\n",
       "      <td>2015-10-24 19:29:52+00:00</td>\n",
       "      <td>2015-10-24</td>\n",
       "      <td>24</td>\n",
       "      <td>10</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>89774</td>\n",
       "      <td>Boxing story</td>\n",
       "      <td>1445715207</td>\n",
       "      <td>2015-10-24 19:33:27+00:00</td>\n",
       "      <td>2015-10-24</td>\n",
       "      <td>24</td>\n",
       "      <td>10</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>89774</td>\n",
       "      <td>MMA</td>\n",
       "      <td>1445715200</td>\n",
       "      <td>2015-10-24 19:33:20+00:00</td>\n",
       "      <td>2015-10-24</td>\n",
       "      <td>24</td>\n",
       "      <td>10</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId              tag   timestamp                  datetime  \\\n",
       "0       2    60756            funny  1445714994 2015-10-24 19:29:54+00:00   \n",
       "1       2    60756  Highly quotable  1445714996 2015-10-24 19:29:56+00:00   \n",
       "2       2    60756     will ferrell  1445714992 2015-10-24 19:29:52+00:00   \n",
       "3       2    89774     Boxing story  1445715207 2015-10-24 19:33:27+00:00   \n",
       "4       2    89774              MMA  1445715200 2015-10-24 19:33:20+00:00   \n",
       "\n",
       "        date  day  month  year  \n",
       "0 2015-10-24   24     10  2015  \n",
       "1 2015-10-24   24     10  2015  \n",
       "2 2015-10-24   24     10  2015  \n",
       "3 2015-10-24   24     10  2015  \n",
       "4 2015-10-24   24     10  2015  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00e87a98-9330-49cd-8bbb-78bf7f38ac1d",
   "metadata": {},
   "source": [
    "# Insights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c32a1f8-01d0-4685-91a4-48abb08dc38a",
   "metadata": {},
   "source": [
    "1. For CSV data pay attention to the dialect\n",
    "2. EscapeChar vs QuoteChar in Pandas\n",
    "3. Opinion: Safe approach for timestamps - import as Integers/Numeric and convert using ```pd.to_datetime```\n",
    "4. ```pandas.Series.dt.date```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "b8966b7b-2eeb-48c3-839d-840e78acd254",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Zen of Python, by Tim Peters\n",
      "\n",
      "Beautiful is better than ugly.\n",
      "Explicit is better than implicit.\n",
      "Simple is better than complex.\n",
      "Complex is better than complicated.\n",
      "Flat is better than nested.\n",
      "Sparse is better than dense.\n",
      "Readability counts.\n",
      "Special cases aren't special enough to break the rules.\n",
      "Although practicality beats purity.\n",
      "Errors should never pass silently.\n",
      "Unless explicitly silenced.\n",
      "In the face of ambiguity, refuse the temptation to guess.\n",
      "There should be one-- and preferably only one --obvious way to do it.\n",
      "Although that way may not be obvious at first unless you're Dutch.\n",
      "Now is better than never.\n",
      "Although never is often better than *right* now.\n",
      "If the implementation is hard to explain, it's a bad idea.\n",
      "If the implementation is easy to explain, it may be a good idea.\n",
      "Namespaces are one honking great idea -- let's do more of those!\n"
     ]
    }
   ],
   "source": [
    "# Imma just leave it here\n",
    "# call it an Easter Egg\n",
    "\n",
    "import this"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9cd9648-5a13-446f-a775-76616ace03c1",
   "metadata": {},
   "source": [
    "# Next"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f30f9979-d5c1-4fa6-875d-76e2c017ec76",
   "metadata": {},
   "source": [
    "* Let's play with the MovieLens dataset some more."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}