003_data_visualization_01_pyplot_pandas_seaborn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Disclaimer**: `matplotlib` was very recently updated to version 2.0.0, where they changed a lot of things related to styles. Some packages build on top of `matplotlib` still haven't adapted to the changes and some things can be glitchy. New versions of `pandas` and `seaborn` will fix things, but it will most probably happen in a few months."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl\n",
    "\n",
    "\n",
    "# %matplotlib notebook\n",
    "%matplotlib inline\n",
    "\n",
    "# increase default resolution of figures\n",
    "mpl.rcParams['figure.dpi'] = 110\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# About importance of visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "anscombe = pd.read_csv('data/anscombe.csv')\n",
    "anscombe.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "anscombe.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**Note**: Sometimes in other people's scripts you can see the following import (or you might've even used it yourself):\n",
    "\n",
    ">        from pylab import *\n",
    "    \n",
    ">This will import all the functions from `matplotlib.pyplot` and most `numpy` into your namespace, and you can access them directly in a MATLAB-like manner, e.g. `plot(x)` or `sin(x)`. This is generally discouraged, because it pollutes the namespace and you're not always sure what you're using. But if you come from MATLAB, you might want to use it for easier transition. You can find more details <a href=\"http://stackoverflow.com/questions/16849483/which-is-the-recommended-way-to-plot-matplotlib-or-pylab\">here</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Line plot, scatter\n",
    "We start by doing simple plots with `plot` function from the `matplotlib.pyplot` package. You can pass a list or an array of numbers, for example, a sine wave:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# create 500 samples from 0 to 1, like it is 1 second sampled with 500 Hz frequency\n",
    "t = np.linspace(0,1,500)\n",
    "# create a sine wave with frequency 5 Hz\n",
    "signal = np.sin(t*2*np.pi*5)\n",
    "plt.plot(signal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case the X axis will be just an index of the element: we have 500 elements. If you specify 2 arguments, the first one will become X axis, and second -- Y axis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.plot(t,signal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`plot`, like many functions in the `pyplot` package, has many settings. We will consider some of them here.\n",
    ">You can check them all on the [`pyplot` documentation page](http://matplotlib.org/api/pyplot_summary.html), where all functions are listed. Other great sources of insight about how this package works and what you can do with it are `matplotlib` [examples](http://matplotlib.org/examples/index.html) and [gallery](http://matplotlib.org/gallery.html)\n",
    "\n",
    "For example, you specify `linestyle` to make dashed lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.plot(t,signal,linestyle='dashed')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also set `linewidth` and `color`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.plot(t, signal, linestyle='dotted', linewidth=2.5, color='green')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selecting `linestyle` and `color` is so frequent that `pyplot` has a shortcut for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# -- is for dashed, 'r' for red\n",
    "plt.plot(t, signal, '--r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you call `plot` function several times in the same cell in Jupyter, they will all be on the same figure, and `matplotlib` will automatically change colors (we will learn later how to make figures with several subplots and control where each plot is drawn):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.plot(t, signal)\n",
    "plt.plot(t+1, signal+1)\n",
    "plt.plot(t, signal+2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, you don't actually want to see any line, but instead want to see the markers, consider the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# creare an array with 100 random points (normal distribution)\n",
    "random_points = np.random.randn(100)\n",
    "plt.plot(random_points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case plotting the line may not make sense, so I can set `linestyle=''` or `='none'` and specify `marker`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.plot(random_points, linestyle='none', marker='o')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It especially makes sense when you want to plot a scatter where points have to sequential relationship with each other (try to remove `linestyle=''` and see what happens):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# random points 100 by 2 array\n",
    "random_points = np.random.randn(100,2)\n",
    "# plot first column as X and second as Y\n",
    "plt.plot(random_points[:,0],random_points[:,1],linestyle='',marker='o')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is what we call a *scatterplot* and you can create it with `plot` function, like we just did. But there is also a specialized function to create scatterplots:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "plt.scatter(random_points[:,0],random_points[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although in this case the output seems the same, `scatter` offers some advangates. When you do `plot`, all the points are associated with the same *line object*, and they cannot have different parameters, like different colors (you'd have to call `plot` for each group of dots to specify different colors). With scatter you can pass a vector of colors (or sizes) and each point will have separate color.\n",
    "\n",
    "Let's see it with example. Load Iris dataset (it is a pretty famous dataset, you can read about it [on wiki](https://en.wikipedia.org/wiki/Iris_flower_data_set))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "iris = pd.read_csv('data/iris.csv')\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot sepal length against sepal width as a scatterplot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.scatter(iris['sepal length'], iris['sepal width'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's color it by species. What we need is an array of colors for each point. We can do it by making a `replace` on the species column, to replace each species with a color and then save it to a separate column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "species_color_mapping = {'setosa':'blue', 'versicolor':'black', 'virginica':'red'}\n",
    "species_color_mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "iris['color'] = iris['species'].replace(species_color_mapping)\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.scatter(iris['sepal length'], iris['sepal width'], color=iris['color'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Colors can also be represented in other formats (like RGB values), or you can even make colors proportional to a certain variable. To do it, pass a set of numbers as `c` argument and define a colormap (`cmap`) accoring to which values will be mapped on to the colors. For example, we can make colors proportional to another variable, petal length:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.scatter(iris['sepal length'], iris['sepal width'], c=iris['petal length'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, matplotlib will use colormap that is called `viridis`, but there are [many other options](http://matplotlib.org/users/colormaps.html). For example, in this case it makes sense to use any *sequential* colormap, which are best for representing continuous uniform values. Let's use `winter`, which uses range from blue to green. And we can also show color legend with `plt.colorbar()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.scatter(iris['sepal length'], iris['sepal width'], c=iris['petal length'], cmap='winter')\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also pass an array of sizes. For instance, we can make the size proportional to another column, petal width:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.scatter(iris['sepal length'], iris['sepal width'], \n",
    "            c=iris['petal length'], cmap='winter', s=iris['petal width']*50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In right circumstances, this approach can help represent several dimensions on the same figure. It can create really powerful and comprehensive graphics if used correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Labeling plots\n",
    "Making a legend on your plots is usually a very good idea. In `matplotlib` when you plot anything, you can pass `label` argument, and it will later show up on the legend. You have to keep in mind that by default a single plotting function will create a single label. So if we wanted to plot our iris scatterplot with different species in different colors and make the legend accordingly, if is easier to make a loop, rather than `scatterplot` approach we tried before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# iterating through groupby object and plotting each group\n",
    "for name, data in iris.groupby('species'):\n",
    "    plt.plot(data['sepal length'], data['sepal width'], 'o', label=name)\n",
    "    \n",
    "# show legend\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distribution plots\n",
    "After line plots and scatterplot, distribution plots are arguably the most used ones. And among them, the most used probably the histogram:\n",
    "\n",
    "> As we discussed in one of the first lessons, by default Jupyter will show you the output of the last line in the cell, which is very convenient. `hist` function returns bins and heights for each bar it plots, so that you can use this information later. In this case however it clutters the view, so I put `;` after calling `hist` function to suppress output. Try removing it and seeing what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "random_points = np.random.randn(1000,2)\n",
    "plt.hist(random_points[:,0]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `hist` will split the data into 10 bins. We can specify any number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.hist(random_points[:,0],bins=50);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you pass a 2-dimensional array, hist will make a histogram of each column separately, side by side:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "plt.hist(random_points,20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Histograms are basically a collection of bars. You can create some bars of your own with `bar` function. First you specify positions on the X axis and then heights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.bar([0,1,2,3], [10,11,12,13])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`yerr` argument allows to set the errorbars:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.bar([0,1,2,3], [10,11,12,13], yerr=[1,2,1,3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is quite frequent to use barplots in papers, for example we might plot means and standard deviations for each of the species in the iris dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.bar(range(3),iris.groupby('species')['sepal length'].mean(),\n",
    "        yerr=iris.groupby('species')['sepal length'].std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is, however, usually not a very good idea. Barplots only show 2 estimators of the data: usually mean and standard deviation (or standard error of mean). And this presentation can be really misleading. Consider anscombe quartet data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "anscombe = pd.read_csv('data/anscombe.csv')\n",
    "anscombe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.bar(range(4),anscombe[['y1','y2','y3','y4']].mean(),\n",
    "        yerr=anscombe[['y1','y2','y3','y4']].std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boxplot are usually much better for this job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.boxplot(anscombe[['y1','y2','y3','y4']].values);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font color='DarkSeaGreen'>Exercise</font>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Customizing figures\n",
    "`pyplot` let's you easily customize many of the figure features, such as limits of Y and X axes, axes labels, ticks and ticklabels, plot titles and many other. In the following examples we will explore more types of plots and customize them to a certain degree.\n",
    "\n",
    "This is a violin plot: it is similar to boxplot, but it shows the distribution of samples more clearly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.violinplot(anscombe[['y1','y2','y3','y4']].values);\n",
    "\n",
    "# set X axis ticks and labels\n",
    "# (in this function you first specify the ticks (the positions of the ticks)\n",
    "# and then the labels for these position; we will see wait to tweak\n",
    "# this more precisely later)\n",
    "plt.xticks([1,2,3,4], ['y1','y2','y3','y4']);\n",
    "\n",
    "# set Y axis limits\n",
    "plt.ylim([0,15]);\n",
    "\n",
    "# set x label\n",
    "plt.ylabel('Values')\n",
    "\n",
    "# set figure title\n",
    "plt.title('Anscombe quartet')\n",
    "\n",
    "# turn on grid\n",
    "plt.grid('on')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**Note** that `matplotlib`'s boxplot and violinplot both are examples of *messy* tools, meaning that they both want messy data as an input. By default they will plot a box (or a violin) for each column of the input array. We will see ways to plot tidy data later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Heatmap** is a way to visualize a matrix of values. You can create one with `imshow` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "grid = np.random.randn(5, 5)\n",
    "grid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.imshow(grid)\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let' visualize something more interesting: total births for each month for each year:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "births = pd.read_csv('data/births.csv')\n",
    "births.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "births_year_month = births.pivot_table(index='year',columns='month',values='births',aggfunc=np.sum)\n",
    "births_year_month.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll take first 12 years in the dataset, so that the final figure is square:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "births_year_month = births_year_month.iloc[:12]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.imshow(births_year_month)\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's customize the figure: change colormap to something more appropriate (sequential), and label axes and ticks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.imshow(births_year_month, interpolation=None, cmap='Reds')\n",
    "plt.colorbar()\n",
    "\n",
    "plt.xticks(range(len(births_year_month.columns)), \n",
    "           births_year_month.columns)\n",
    "\n",
    "plt.yticks(range(len(births_year_month.index)), \n",
    "           births_year_month.index)\n",
    "\n",
    "plt.xlabel('Month')\n",
    "plt.ylabel('Year')\n",
    "plt.title('Total births for each month from ' + str(births_year_month.index[0]) + \n",
    "          ' to ' + str(births_year_month.index[-1]));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font color='DarkSeaGreen'>Exercise</font>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plotting with `pandas`\n",
    "`matplotlib` is a great package, it is very powerful and well organized. However, it can require a lot of code to plot certain types of data, e.g. tidy data. It is just not build for this. But because of its nice structure, `matplotlib` became the basis for other plotting tools. In fact, `pandas` itself can interface with `matplotlib` and allow you to plot certain things really quickly. For example, you can quickly plot 2 columns, and pandas will automatically label everything it can accoring to the names of the columns, index, etc. Let's plot births for males and females for each year (calculated with `pivot_table`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "births_year_gender = births.pivot_table(values='births', index='year', columns='gender', aggfunc=np.sum)\n",
    "births_year_gender.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "births_year_gender.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The great thing about `pandas` basing their plotting tools on `matplotlib` is that you can still tweak everything using `matplotlib`, if you need it. For example, we can label Y axis and add a title:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "births_year_gender.plot()\n",
    "plt.title('Change of total yearly births for males and females')\n",
    "plt.ylabel('Total births')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the beginning we plotted with `pyplot`, but what is `pyplot` and how it relates to `matplotlib`? In technical terms, `pyplot` is an API (*application programming interface*) to `matplotlib`, which means that it provides certain commands (such as `plt.plot`, `plt.scatter`, etc) to access functionality of `matplotlib` in a certain way. We could say that `pandas` provides another API for `matplotlib`. It doesn't implement anything new in `matplotlib`, it just creates a high-level access point to a some combination of commands, which is convenient to use in certain case. In particular, `pandas` provides a way to plot tidy data. \n",
    "\n",
    "Let's take a look. If you call method `boxplot` on a `DataFrame`, it will create boxplots for each of the columns, which is not very different from `plt.boxplot` (except that `pandas` will use column labels to label boxes):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "iris.boxplot();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can, however, specify an argument `by` and provide it a groupping column. The `boxplot` method will do something different:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "iris.boxplot(by='species');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of plotting columns, it creates boxes for each of the groups, and splits columns in separate subplots. You can also specify just a subset of columns to plot, and some other parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "iris.boxplot(column=['petal length', 'petal width'], by='species', grid='off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`pandas` has a [nice tutorial](http://pandas.pydata.org/pandas-docs/stable/visualization.html) on its visualization tools. `pandas` plotting API is not 100% mature, but it can simplify exploratory visualization a lot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Seaborn -- plotting tidy data\n",
    "`seaborn` ([website](https://seaborn.pydata.org/index.html)) is a visualization package developed on top of `matplotlib`, and it is made specifically for working with tidy data, and creating very appealing figures. It has extensive [gallery](https://seaborn.pydata.org/examples/index.html) with lots of examples and a very good and concise [tutorial](https://seaborn.pydata.org/tutorial.html). Because it is build on top of `matplotlib`, you can still use `matplotlib` to tweak and adjust things, which is great.\n",
    "\n",
    "Let's import `seaborn`:\n",
    "\n",
    ">There are 2 ways of importing `seaborn`. If you run `import seaborn as sns`, you will get a full range of commands from the package, but `seaborn` will also change default style of your figures. Usually, they start to look better. But there are times when you don't want to change the style, but only want to take some elements from `seaborn` package. In this case you can run `import seaborn.apionly as sns`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Immediately default style of `matplotlib` as changed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# repeat plot from the beginning of the notebook\n",
    "plt.plot(t,signal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`seaborn` is really good at plotting complicated and noisy data, because it will automatically calculate confidence intervals for you. Let's say you have many signals you want to average and plot with errors (e.g. change is neuronal firing rate before a stimulus for many trials, or change in BOLD signal, etc):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# create 10 sine waves with added noise\n",
    "x = np.linspace(0, 15, 31)\n",
    "data = np.sin(x) + np.random.rand(10, 31) + np.random.randn(10, 1)\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I can plot all the waves with basic `plot` function, but it is difficult to estimate errors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.plot(data.T,'k',linewidth=1);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `seaborn` `tsplot` (*time-series plot*) I can plot it representing errors in a variety of ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# by default it plots means with 95% confidence interval\n",
    "sns.tsplot(data=data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.tsplot(data=data, err_style=\"ci_bars\", interpolate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# multiple confidence bands (68% and 95%)\n",
    "sns.tsplot(data=data, ci=[68, 95])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# you can show each trace\n",
    "sns.tsplot(data=data, err_style=\"unit_traces\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# or bootstrapped traces\n",
    "sns.tsplot(data=data, err_style=\"boot_traces\", n_boot=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But where `seaborn` trully shines is tidy data comparison. `seaborn` has several functions for representing categorical data (which is data split by group): `boxplot`, `violinplot`, `stripplot`, `swarmplot`, `pointplot`. All of them have really good documentation with lots of example, and I invite you to check it out [here](https://seaborn.pydata.org/api.html) (section Categorical plots) and try them out. We will see a few examples.\n",
    "\n",
    "First, let's make the same boxplot we made with `pandas` before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.boxplot(data=iris, x='species', y='sepal length')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boxplots are nice, they are a big improvement over barplots, because they show much more estimators of the data (i.e. median, quartiles, range, outliers). But nothing can beat showing raw data. Stripplot and swarmplot do exactly that, stripplot plots all the samples with some random jitter along X axis (so that samples don't overlap so much if they have the same Y values), while swarmplot will make sure that samples don't overlap by putting them always next to each other:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.stripplot(data=iris, x='species', y='sepal length', jitter=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.swarmplot(data=iris, x='species', y='sepal length')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When sample size allows, showing raw data is almost always prefferable. However, when groups have >200-300 samples each, it becomes increasingle difficult. In this case `violinplot` provides a good alternative, because they try approximate the distributions, which is even better at high sampling sizes.\n",
    "\n",
    "`seaborn` has a higher-level function called `factorplot`, which can help you very quickly create visualizations for tidy dataset, splitting it exactly the way you want. Let's load food preference dataset and see a few examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "food_df = pd.read_csv('data/Paolo.csv')\n",
    "\n",
    "## do some cleanup\n",
    "food_df.drop('Unnamed: 0', axis='columns', inplace=True)\n",
    "food_df['cond'].replace({1: 'high vs high', 2: 'low vs low', \n",
    "                    3: 'high vs low', 4: 'low vs high'}, inplace=True)\n",
    "food_df['congr'].replace({0: 'same', 1: 'different'}, inplace=True)\n",
    "food_df['session'].replace({0: 'fed', 1: 'hungry'}, inplace=True)\n",
    "\n",
    "food_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`factorplot` allows you to select any kind of categorical plot (e.g. `boxplot`, `swarmplot`, etc), and make groups in different subplots. Let's start easy: make a `pointplot` of reaction time, splitting for session type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.factorplot(data=food_df,x='session',y='rt',kind='point')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`pointplot` shows mean and 95% confidence interval, it is basically a version of `barplot`. To get a better idea about the distributions, we can select `box` or `violin` in the `factorplot` option `kind`. Because sample size is so high (each group will contain ~7500 reaction times), `swarm` is not a good idea here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.factorplot(data=food_df,x='session',y='rt',kind='violin')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is nice, however, we really want to look at the data for each subject separately. `factorplot` has agruments `col` and `row`, which allow you to split the data accoring to more grouppin variables and plot them on separate subplots in columns (`col`) or rows (`row`), e.g.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.factorplot(data=food_df,x='session',y='rt',kind='violin',col='subj_num')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.factorplot(data=food_df,x='session',y='rt',kind='violin',col='subj_num',row='congr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.factorplot(data=food_df,x='session',y='rt',kind='point',row='subj_num',hue='congr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's summarize\n",
    "1. `matplotlib.pyplot` is great for plotting simple things, like 1D or 2D arrays, doing simple visualizations, etc. However, it frequently requires *messy input* rather than tidy. (You can pivot your tables to make them into necessary format, but it can be cumbersome to get exactly what you want);\n",
    "2. `matplotlib` itself has a lower level interface, which allows you to tweak pretty much anything on your figures. This requires some getting used to, because there are many commands, so there is a learning curve. The good thing is that this will work on `matplotlib.pyplot` figures, but also on other figures, created with `pandas` or `seaborn`, because they internally use the machinery of `matplotlib`;\n",
    "3. `pandas` provides an API to `matplotlib`, which allows you to plot `Series` and `DataFrames` quickly, some things will work with tidy data as well. This is not 100% mature system, it is currently being developed, but many things work well;\n",
    "4. `seaborn` is a specialized module for plotting tidy data, and it also usually creates figures of great quality. Its API is quite straightforward to learn, because you usually just need to think about what you want to get in the final figure. It can be used for exploratory visualization and for publications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving figures\n",
    "Matplotlib is really good at saving figures to different formats. You can check the list of filetypes it supports by default like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# list supported graphics types\n",
    "plt.figure().canvas.get_supported_filetypes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save the figure to several formats. Generally, if you want to save a lot of figures for quick examination, it is better to use one of the raster formats, `png` is usually the best choice for figures. However, if you save for publication, using vector format, such as `pdf` or `svg` is better (if they accept vector graphics). An added benefit of vector graphics is that you can edit any part of it with vector grapfics editor, such as [Inkscape](https://inkscape.org/en/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.swarmplot(data=iris, x='species', y='sepal length')\n",
    "plt.title('Iris sepal length for each species')\n",
    "\n",
    "plt.savefig('iris_sepal_length_swarm.png', dpi=300)\n",
    "plt.savefig('iris_sepal_length_swarm.pdf')\n",
    "plt.savefig('iris_sepal_length_swarm.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Where to go from here\n",
    "[Python Data Science Handbook](http://nbviewer.jupyter.org/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/Index.ipynb) by Jake VanderPlas, section *Visualization with Matplotlib*\n",
    "\n",
    "\n",
    "[Matplotlib examples gallery](https://matplotlib.org/gallery.html)\n",
    "\n",
    "[Matplotlib tutorial](http://matplotlib.org/users/tutorials.html)\n",
    "\n",
    "[Seaborn examples gallery](http://seaborn.pydata.org/examples/index.html)\n",
    "\n",
    "[Seaborn tutorial](http://seaborn.pydata.org/tutorial.html)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "py36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}