4 - Exploratory Plotting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step in exploratory data analysis is plotting. Let's go through the world's fastest introduction to the core plotting library in Python, **matplotlib**.\n",
    "\n",
    "We are going to look briefly at some of the underlying concepts in matplotlib to aid in orientation. Having a mental model of how matplotlib works will greatly aid your ability to unlock the power of matplotlib. That said, one of the best ways to make great plots and to customize matplotlib is to go to the [gallery](https://matplotlib.org/gallery.html), find a plot that you want to emulate, and read the code. Another recent but great resource is the [python graph gallery](https://python-graph-gallery.com/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two interfaces available when working with matplotlib -- an interactive interface and an object-oriented interface. Both are available through the `matplotlib.pyplot` namespace.\n",
    "\n",
    "When working interactively, matplotlib maintains a global state for your current plot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's set some Notebook display options. For more information on this, check [here](http://ipython.readthedocs.io/en/stable/interactive/plotting.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the canonical import for matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's use the interactive interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(range(10), range(10))\n",
    "\n",
    "plt.xlabel(\"X Axis\")\n",
    "plt.xticks([])\n",
    "plt.title(\"The Image Title\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I don't *really* recommend using the interactive interface in practice or for anything serious. \n",
    "\n",
    "Alternatively, you can use the object-oriented interface. This usually entails a call to **plt.subplots**.\n",
    "\n",
    "This call returns instances of two objects -- a Figure object and an Axes object. Understanding the [anatomy of a matplotlib plot](http://matplotlib.org/faq/usage_faq.html#parts-of-a-figure) and these two underlying concepts will help unlock the power of **matplotlib**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Figure is the object that contains the Axes. To be concrete, we'll turn the figure color gray."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig.set_facecolor('gray')\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Axes contain the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax.scatter(\n",
    "    (.25, .25, .75, .75), (.25, .75, .25, .75), \n",
    "    s=64**2,\n",
    "    c=['red', 'green', 'blue', 'orange']\n",
    ")\n",
    "\n",
    "ax.set_xticks([])\n",
    "ax.set_yticks([])\n",
    "\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Figure can contain multiple Axes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=2, ncols=1, figsize=(10, 12))\n",
    "\n",
    "axes[0].plot(range(10), range(10))\n",
    "axes[1].plot(range(10), range(10, 0, -1))\n",
    "\n",
    "axes[0].set_title(\"Line 1\")\n",
    "axes[1].set_title(\"Line 2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## High-Level Plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the reasons that this abstraction and the distinction between figures and axes can be useful is in understanding how to operate with some higher-level plotting libraries like **pandas** and **seaborn**.\n",
    "\n",
    "Pandas itself has a plotting API that utilizes matplotlib under the hood. Let's take a look at some of this functionality using the canonical Titanic dataset with information on passenger survival in this famous shipwreck."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "titanic = pd.read_csv(\"data/titanic.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we know, bar plots can be useful for displaying and comparing measurable quantities, such as counts or volumes. Let's say we want to make a bar plot of the number of survivors by the class of their ticket. \n",
    "\n",
    "Pandas provides access to all of the plotting functionality through the **DataFrame.plot** attribute. The plot attribute is exposed on GroupBy objects as well. Here, we'll group by the class of the ticket and then call `...plot.bar` to get a nice bar plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = (titanic.groupby('pclass')\n",
    "      .survived.sum()\n",
    "      .plot.bar())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that pandas returned to us the matplotlib axes. We could go on and customize these axes should we so desire. Let's rotate **tick labels**, for example.\n",
    "\n",
    "Note that each axes holds a reference to the figure, which we use to display in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for label in ax.xaxis.get_ticklabels():\n",
    "    label.set_rotation(0)\n",
    "    \n",
    "ax.figure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use this pattern to plot multiple things on an axes using a mix of pure matplotlib and higher-level plotting packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This plot is a little small. One trick I like to use is explicitly passing in an axes that I have already created. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(12, 6))\n",
    "\n",
    "ax = (titanic.groupby('pclass')\n",
    "      .survived.sum()\n",
    "      .plot.bar(ax=ax))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we might want to make the tick and axes labels bigger. We could do this by munging the axes or we could take advantage of the matplotlib style context manager."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "font_config = {\n",
    "    'font.size': 16,\n",
    "}\n",
    "\n",
    "with plt.style.context(font_config):\n",
    "    fig, ax = plt.subplots(figsize=(12, 6))\n",
    "\n",
    "    ax = (titanic.groupby('pclass')\n",
    "          .survived.sum()\n",
    "          .plot.bar(ax=ax))\n",
    "    \n",
    "    for tick in ax.xaxis.get_ticklabels():\n",
    "        tick.set_rotation(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also set matplotlib style state globally using the `rc` function. See here for more information. TODO: add link."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rc('font', size=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a few more interesting bar plots we can make using pandas. We might want to see the number of survivors broken down with class by gender, and plot it as a horizontal bar chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(12, 6))\n",
    "\n",
    "ax = (titanic.groupby(['sex', 'pclass'])\n",
    "      .survived.sum().plot.barh(ax=ax))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look use a cross-tab of the those who did and did not survive by class and gender to make a stacked bar chart, which is a little more economical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "death_counts = pd.crosstab([titanic.pclass, titanic.sex], titanic.survived.astype(bool))\n",
    "death_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = death_counts.plot.barh(stacked=True) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also look at the survival *rate* of the groups, by adjusting for the number of people in each group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = (death_counts.div(death_counts.sum(1), axis=0)\n",
    "      .plot.barh(stacked=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This, obviously, isn't quite what we want for the legend. We might Consider increasing the margin at the bottom and making the legend horizontal. Let's go ahead and clean up the labels too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = (death_counts.div(death_counts.sum(1), axis=0)\n",
    "      .plot.barh(stacked=True))\n",
    "\n",
    "ax.margins(x=0)\n",
    "ax.set_ylim(-1.75, 5.5)\n",
    "\n",
    "ax.legend(loc=(.15, .05), ncol=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Health Inspection Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look back at the Chicago Health Inspection data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from load_data import dta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dta = dta.drop([\"violations\"], axis='columns').join(\n",
    "    dta.violations.str.split(\"|\", expand=True)\n",
    "        .unstack()\n",
    "        .dropna()\n",
    "        .str.strip()\n",
    "        .reset_index(level=0, drop=True)\n",
    "        .to_frame()\n",
    "        .rename(columns={0: 'violations'}),\n",
    "    how='right'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing we may wonder is if the violations are fairly evenly distributed or if some are much more common than others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "violation_num = dta.violations.str.extract(\"(\\d+)(?=\\.)\", expand=False).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(14, 10))\n",
    "\n",
    "ax = violation_num.groupby(violation_num).size().plot.bar(ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise:\n",
    "\n",
    "See if there are any meaningful differences in the number of violations by quarter. Recall that you can use the `dt` accessor for datetime functionality on a Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load solutions/violation_by_quarter.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting Geographic Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One the most interesting features that we have here are geographical. There are a few options for plotting maps in Python (and in Notebooks).\n",
    "\n",
    "The first thing we can try is plotting with matplotlib only. If you don't need much else, this can work really well to get a sense of how things are, if your data are sufficiently dense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dta.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dta = dta.dropna(subset=[\"latitude\", \"longitude\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with plt.style.context({\n",
    "    'axes.facecolor': 'black',\n",
    "   }):\n",
    "    fig, ax = plt.subplots(figsize=(16, 12))\n",
    "\n",
    "    ax = dta.plot.scatter(x='longitude', y='latitude', color='white', s=.02, alpha=.6, ax=ax)\n",
    "    ax.xaxis.set_ticklabels([])\n",
    "    ax.yaxis.set_ticklabels([])\n",
    "    ax.set_xlabel('')\n",
    "    ax.set_ylabel('')\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few things to note. First, we used a context manager again to set some style that we know we wanted to adhere to. Second, we used the `DataFrame.plot.scatter` method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we thought there may be some geographic clustering to the restaurant incident reporting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cmap = plt.cm.get_cmap('RdYlBu')\n",
    "\n",
    "\n",
    "with plt.style.context({\n",
    "    'axes.facecolor': 'black',\n",
    "    'legend.facecolor': 'inherit',\n",
    "    'legend.markerscale': 32,\n",
    "    'text.color': 'white',\n",
    "   }):\n",
    "    fig, ax = plt.subplots(figsize=(16, 12))\n",
    "\n",
    "    grouper = dta.groupby('facility_type')\n",
    "    for i, (name, group) in enumerate(grouper):\n",
    "        if len(group) < 5000:\n",
    "            continue\n",
    "        ax = group.plot.scatter(x='longitude', y='latitude', s=.2, color=cmap(i), alpha=.6, ax=ax, label=name)\n",
    "    ax.xaxis.set_ticklabels([])\n",
    "    ax.yaxis.set_ticklabels([])\n",
    "    ax.set_xlabel('')\n",
    "    ax.set_ylabel('')\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we attempt to uncover some more detail from our map, by plotting the different types of estbalishments using different colors. We use matplotlib's **color maps** under the `plt.cmap` namespace to do this. We also add a **legend** and style it. In general, you use the `label` argument to add a name to plot element and the call `legend` on the axes to display it.\n",
    "\n",
    "Incidentally, the matplotlib documentation has a [fantastic page](https://matplotlib.org/users/colormaps.html) about choosing color maps purposefully for different situations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For some more fine-grained control over maps, you may want to check out packages like [cartopy](http://scitools.org.uk/cartopy/), [folium](http://python-visualization.github.io/folium/), and [bqplot](https://github.com/bloomberg/bqplot)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}