notebooks/chapter10_signal/03_autocorrelation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This is one of the 100 recipes of the [IPython Cookbook](http://ipython-books.github.io/), the definitive guide to high-performance scientific computing and data science in Python.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10.3. Computing the autocorrelation of a time series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the *Babies* dataset on the book's website. (http://ipython-books.github.io)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. We import the packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. We read the data with Pandas. The dataset contains one CSV file per year. Each file contains all baby names given that year with the respective frequencies. We load the data in a dictionary, containing one `DataFrame` per year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "files = [file for file in os.listdir('data/') \n",
    "         if file.startswith('yob')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "years = np.array(sorted([int(file[3:7]) \n",
    "                         for file in files]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = {year: \n",
    "        pd.read_csv('data/yob{y:d}.txt'.format(y=year), \n",
    "                    index_col=0, header=None, \n",
    "                    names=['First name', 'Gender', 'Number']) \n",
    "        for year in years}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data[2012].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. We write functions to retrieve the frequencies of baby names as a function of the name, gender, and birth year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_value(name, gender, year):\n",
    "    \"\"\"Return the number of babies born a given year, with a \n",
    "    given gender and a given name.\"\"\"\n",
    "    try:\n",
    "        return data[year][data[year]['Gender'] == gender] \\\n",
    "               ['Number'][name]\n",
    "    except KeyError:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_evolution(name, gender):\n",
    "    \"\"\"Return the evolution of a baby name over the years.\"\"\"\n",
    "    return np.array([get_value(name, gender, year) \n",
    "                     for year in years])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. Let's define a function that computes the autocorrelation of a signal. This function is essentially based on NumPy's `correlate` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def autocorr(x):\n",
    "    result = np.correlate(x, x, mode='full')\n",
    "    return result[result.size/2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. Now, we create a function that displays the evolution of a baby name, as well as its autocorrelation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def autocorr_name(name, gender, color):\n",
    "    x = get_evolution(name, gender)\n",
    "    z = autocorr(x)\n",
    "    # Evolution of the name.\n",
    "    plt.subplot(121);\n",
    "    plt.plot(years, x, '-o'+color, label=name);\n",
    "    plt.title(\"Baby names\");\n",
    "    # Autocorrelation.\n",
    "    plt.subplot(122);\n",
    "    plt.plot(z / float(z.max()), '-'+color, label=name);\n",
    "    plt.legend();\n",
    "    plt.title(\"Autocorrelation\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. Let's take a look at two female names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,4));\n",
    "autocorr_name('Olivia', 'F', 'k');\n",
    "autocorr_name('Maria', 'F', 'y');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The autocorrelation of Olivia is decaying much faster than Maria's. This is mainly because of the steep increase of the name Olivia at the end of the twentieth century. By contrast, the name Maria is evolving more slowly globally, and its autocorrelation is decaying somewhat slower."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> You'll find all the explanations, figures, references, and much more in the book (to be released later this summer).\n",
    "\n",
    "> [IPython Cookbook](http://ipython-books.github.io/), by [Cyrille Rossant](http://cyrille.rossant.net), Packt Publishing, 2014 (500 pages)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}