exercises/day4/Day_4_exercise_5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting IMDB "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will make one last visit to the IMDB data. This time, you will get to explore the data by plotting graphs. Pandas will do the parsing for you!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The questions to answer are:\n",
    "    \n",
    "- Are movies getting better or worse? That is, how do the movies' rating relate to the the year they were produced? \n",
    "- What rating does most movies have - how many good movies are there?\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use two types of plots: `scatter` and `histogram`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting started - reading the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you might have thought when parsing the IMDB data yourself, getting it right is not that easy. Pandas agree, it's much more complicated than reading the Orange tree data. But let's just start by making a try:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "movies = pandas.read_table('../../downloads/250.imdb', index_col=0)\n",
    "movies.head()  # use .head() to print only the first part of the table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the commands above and take a look at the result. Try to figure out what went wrong!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, the lines are not split at the `|`, like they should be. To fix it, use the keyword argument `sep`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies = pandas.read_table('../../downloads/250.imdb', sep='|', index_col=0)\n",
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Better! But what about the index column..? It looks like the `#Votes` is what identifies a movie. What would be a better identifier for a movie?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I would go for `Title`; each movie has a title, and two movies are not supposed to have the same title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies = pandas.read_table('../../downloads/250.imdb', sep='|', index_col=6)\n",
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could also tell Pandas to create its own indexing for the movies. For instance, if you suspect that there are one movie with the same title. To do this, just leave out the `index_col`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies = pandas.read_table('../../downloads/250.imdb', sep='|')\n",
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it looks like we're getting somewhere!\n",
    "But the columns does not look quite right. Take a look yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are too many whitespaces! Having whitespaces in the columnames will make it much harder for you to work with the data, since it's very easy to miss them or to forget about them. That might give you annoying errors, like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies.Rating  # no whitespaces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies[' Rating']  # forgetting the last whitespace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas did not manage to get this right for us, so here's one way of fixing this issue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies.columns = movies.columns.str.strip(' #')  # Remove '#' and whitespaces\n",
    "# ... and try it:\n",
    "movies.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the complete code for parsing the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "movies = pandas.read_table('../../downloads/250.imdb', sep='|', index_col=6)\n",
    "movies.columns = movies.columns.str.strip(' #')\n",
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, a note on plotting. If your plots don't show up, try the following:\n",
    "\n",
    "- if using a notebook, run: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```py\n",
    "%pylab inline \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- if using the Python interpretor, use this:\n",
    "\n",
    "```py\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# ... code to generate the plots ...\n",
    "\n",
    "plt.show()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Are movies getting better or worse?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To solve this, we will use a `scatter` plot. Our dataframe is called `movies`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create one, just change `kind` to `scatter`:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```py\n",
    "movies.plot(x=..., y=..., kind='scatter',fontsize=14, figsize=(12,10))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What to put on the x and y axis? Take a moment and see if you can figure it out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want the x axis to represent the time, so we set `x` to `'Year'`.\n",
    "\n",
    "The y axis should represent how good a movie is, so let's use the rating:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies.plot(x='Year', y='Rating', kind='scatter',fontsize=14, figsize=(12,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You've got a graph! Have a look at it and decide for yourself whether movies seem to be getting better or not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. What rating does most movies have? How many good movies are there?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're interested in the rating, and specifically the frequency of each rating. Let's use a historgram for this. A histogram looks like this, with the frequency of a given column shown on the y axis:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../../img/histplot.png\" width=60%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This histogram tells us that values around 1 is more frequent than the value 4 or -2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making a histogram is not more diffult than changing the `kind` to `hist`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```py\n",
    "movies.plot(kind='hist', y=...)\n",
    "```\n",
    "\n",
    "Notice that were skipping the x axis here. Histograms usually show the frequency on one axis, so it only needs you to specify one axis. It will ignore what you put as x value, just give it the y!\n",
    "\n",
    "Try to plot a historgram of the rating of the movies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies.plot(kind='hist', y='Rating')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You did it! That's all! Spend a minute or two looking at your graphs. Also feel free to play around with them. If you want to plot more, some ideas are given below. If you have had enough, take a break and prepare for the course project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Want more?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Find out if people vote more for good movies or not. How does the number of votes correlate with the rating?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Which years were most movies produced?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are plenty of other tutorials online, here are a few:\n",
    "    \n",
    "\n",
    "-  https://www.youtube.com/playlist?list=PLQVvvaa0QuDc-3szzjeP6N6b0aDrrKyL- A nice youtube tutorial series\n",
    "\n",
    "- https://swcarpentry.github.io/python-intermediate-mosquitoes/01-intro-python.html\n",
    "\n",
    "- https://www.tutorialspoint.com/python_pandas/index.htm\n",
    "\n",
    "- https://www.tutorialspoint.com/python_pandas/python_pandas_visualization.htm (The visualization section of the tutorial above)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}