Lecture 09.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:669a5df7acb5ebbe52c3481a059d8a6cbe13ae3f1c9b360f791bf41ab31133b2"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Today we'll cover the following basic data structures in Pandas:\n",
      "\n",
      "1. [Series](#Series)\n",
      "2. [DataFrame](#DataFrame)\n",
      "3. [Panel](#Panel)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Import modules\n",
      "import numpy as np\n",
      "import pandas as pd\n",
      "import sys"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Print versions of Python and modules using which this notebook was built\n",
      "print 'Python version ' + sys.version\n",
      "print 'Numpy version: ' + np.__version__\n",
      "print 'Pandas version: ' + pd.__version__"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Python version 2.7.7 |Anaconda 2.0.1 (x86_64)| (default, Jun  2 2014, 12:48:16) \n",
        "[GCC 4.0.1 (Apple Inc. build 5493)]\n",
        "Numpy version: 1.8.1\n",
        "Pandas version: 0.14.0\n"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Series"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A pandas Series object is like a fixed length dict mapping the `index` values to the corresponding data values."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops_dict = {'China': 1.36e+09, 'India': 1.29e+09, 'USA': 322e+06, \n",
      "                     'Indonesia': 250e+06, 'Brazil': 202e+06, 'Pakistan': 186e+06,\n",
      "                     'Nigeria': 175e+06, 'Bangladesh': 156e+06, 'Russia': 142e+06,\n",
      "                     'Japan': 126e+06}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops = pd.Series(country_pops_dict)  # creates a Pandas Series from Python dict"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print country_pops"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Bangladesh     156000000\n",
        "Brazil         202000000\n",
        "China         1360000000\n",
        "India         1290000000\n",
        "Indonesia      250000000\n",
        "Japan          126000000\n",
        "Nigeria        175000000\n",
        "Pakistan       186000000\n",
        "Russia         142000000\n",
        "USA            322000000\n",
        "dtype: float64\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "However, Series objects are also numpy array-like: you can select a value or set of values."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops['Japan']  # get one data value"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "126000000.0"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops[['USA', 'Russia']]  # get multiples values from indices "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "USA       322000000\n",
        "Russia    142000000\n",
        "dtype: float64"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops / 1e6  # in millions"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "Bangladesh     156\n",
        "Brazil         202\n",
        "China         1360\n",
        "India         1290\n",
        "Indonesia      250\n",
        "Japan          126\n",
        "Nigeria        175\n",
        "Pakistan       186\n",
        "Russia         142\n",
        "USA            322\n",
        "dtype: float64"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops[country_pops > 1e9]  # countries with population over a billion"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "China    1360000000\n",
        "India    1290000000\n",
        "dtype: float64"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_area_dict = {'Russia': 1.708e+07,  # areas in sq km of 10 largest countries\n",
      "                    'Canada': 9.985e+06, \n",
      "                    'USA': 9.631e+06,\n",
      "                    'China': 9.597e+06,\n",
      "                    'Brazil': 8.515e+06,\n",
      "                    'Australia': 7.741e+06,\n",
      "                    'India': 3.287e+06,\n",
      "                    'Argentina': 2.767e+06,\n",
      "                    'Kazakhstan': 2.725e+06,\n",
      "                    'Algeria': 2.382e+06}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_area = pd.Series(country_area_dict)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops / country_area  # diff sized Series are automatically aligned by indices (NaNs are automatically supplied)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 12,
       "text": [
        "Algeria              NaN\n",
        "Argentina            NaN\n",
        "Australia            NaN\n",
        "Bangladesh           NaN\n",
        "Brazil         23.722842\n",
        "Canada               NaN\n",
        "China         141.710951\n",
        "India         392.455126\n",
        "Indonesia            NaN\n",
        "Japan                NaN\n",
        "Kazakhstan           NaN\n",
        "Nigeria              NaN\n",
        "Pakistan             NaN\n",
        "Russia          8.313817\n",
        "USA            33.433704\n",
        "dtype: float64"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops.name = 'Country population'  # Series can have a name"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops.index.name = 'Country'  # so can its index"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print country_pops"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Country\n",
        "Bangladesh     156000000\n",
        "Brazil         202000000\n",
        "China         1360000000\n",
        "India         1290000000\n",
        "Indonesia      250000000\n",
        "Japan          126000000\n",
        "Nigeria        175000000\n",
        "Pakistan       186000000\n",
        "Russia         142000000\n",
        "USA            322000000\n",
        "Name: Country population, dtype: float64\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ser1 = pd.Series([1, 2, 3], index=['a', 'b', 'c'])  # can also creates Series from lists"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print ser1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "a    1\n",
        "b    2\n",
        "c    3\n",
        "dtype: int64\n"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops.index = [name.upper() for name in country_pops.index]  # index can be modified"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print country_pops  # now country names are in upper case (note that index.name got deleted)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "BANGLADESH     156000000\n",
        "BRAZIL         202000000\n",
        "CHINA         1360000000\n",
        "INDIA         1290000000\n",
        "INDONESIA      250000000\n",
        "JAPAN          126000000\n",
        "NIGERIA        175000000\n",
        "PAKISTAN       186000000\n",
        "RUSSIA         142000000\n",
        "USA            322000000\n",
        "Name: Country population, dtype: float64\n"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops.sort()  # sort in place based on values"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 20
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print country_pops"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "JAPAN          126000000\n",
        "RUSSIA         142000000\n",
        "BANGLADESH     156000000\n",
        "NIGERIA        175000000\n",
        "PAKISTAN       186000000\n",
        "BRAZIL         202000000\n",
        "INDONESIA      250000000\n",
        "USA            322000000\n",
        "INDIA         1290000000\n",
        "CHINA         1360000000\n",
        "Name: Country population, dtype: float64\n"
       ]
      }
     ],
     "prompt_number": 21
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops.sort_index()  # return a copy with indices in sorted order"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 22,
       "text": [
        "BANGLADESH     156000000\n",
        "BRAZIL         202000000\n",
        "CHINA         1360000000\n",
        "INDIA         1290000000\n",
        "INDONESIA      250000000\n",
        "JAPAN          126000000\n",
        "NIGERIA        175000000\n",
        "PAKISTAN       186000000\n",
        "RUSSIA         142000000\n",
        "USA            322000000\n",
        "Name: Country population, dtype: float64"
       ]
      }
     ],
     "prompt_number": 22
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "country_pops  # original Series is still sorted by values"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 23,
       "text": [
        "JAPAN          126000000\n",
        "RUSSIA         142000000\n",
        "BANGLADESH     156000000\n",
        "NIGERIA        175000000\n",
        "PAKISTAN       186000000\n",
        "BRAZIL         202000000\n",
        "INDONESIA      250000000\n",
        "USA            322000000\n",
        "INDIA         1290000000\n",
        "CHINA         1360000000\n",
        "Name: Country population, dtype: float64"
       ]
      }
     ],
     "prompt_number": 23
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "DataFrame"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let us read in the famous [Iris data set](http://en.wikipedia.org/wiki/Iris_flower_data_set#Data_set) from the pandas github repository.\n",
      "\n",
      "`pd.read_csv()` return a Pandas DataFrame object."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data = pd.read_csv('https://raw.githubusercontent.com/pydata/pandas/master/pandas/tests/data/iris.csv')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.head(10)  # first 10 rows"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SepalLength</th>\n",
        "      <th>SepalWidth</th>\n",
        "      <th>PetalLength</th>\n",
        "      <th>PetalWidth</th>\n",
        "      <th>Name</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 5.1</td>\n",
        "      <td> 3.5</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 4.9</td>\n",
        "      <td> 3.0</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 4.7</td>\n",
        "      <td> 3.2</td>\n",
        "      <td> 1.3</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 4.6</td>\n",
        "      <td> 3.1</td>\n",
        "      <td> 1.5</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 5.0</td>\n",
        "      <td> 3.6</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5</th>\n",
        "      <td> 5.4</td>\n",
        "      <td> 3.9</td>\n",
        "      <td> 1.7</td>\n",
        "      <td> 0.4</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6</th>\n",
        "      <td> 4.6</td>\n",
        "      <td> 3.4</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.3</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7</th>\n",
        "      <td> 5.0</td>\n",
        "      <td> 3.4</td>\n",
        "      <td> 1.5</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8</th>\n",
        "      <td> 4.4</td>\n",
        "      <td> 2.9</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9</th>\n",
        "      <td> 4.9</td>\n",
        "      <td> 3.1</td>\n",
        "      <td> 1.5</td>\n",
        "      <td> 0.1</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 25,
       "text": [
        "   SepalLength  SepalWidth  PetalLength  PetalWidth         Name\n",
        "0          5.1         3.5          1.4         0.2  Iris-setosa\n",
        "1          4.9         3.0          1.4         0.2  Iris-setosa\n",
        "2          4.7         3.2          1.3         0.2  Iris-setosa\n",
        "3          4.6         3.1          1.5         0.2  Iris-setosa\n",
        "4          5.0         3.6          1.4         0.2  Iris-setosa\n",
        "5          5.4         3.9          1.7         0.4  Iris-setosa\n",
        "6          4.6         3.4          1.4         0.3  Iris-setosa\n",
        "7          5.0         3.4          1.5         0.2  Iris-setosa\n",
        "8          4.4         2.9          1.4         0.2  Iris-setosa\n",
        "9          4.9         3.1          1.5         0.1  Iris-setosa"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.columns  # columns attribute has columns names"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 26,
       "text": [
        "Index([u'SepalLength', u'SepalWidth', u'PetalLength', u'PetalWidth', u'Name'], dtype='object')"
       ]
      }
     ],
     "prompt_number": 26
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Note that column names are stored as [Python unicode strings](https://docs.python.org/2/howto/unicode.html#the-unicode-type)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data['SepalLength']  # returns a column as pd.Series"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 27,
       "text": [
        "0     5.1\n",
        "1     4.9\n",
        "2     4.7\n",
        "3     4.6\n",
        "4     5.0\n",
        "5     5.4\n",
        "6     4.6\n",
        "7     5.0\n",
        "8     4.4\n",
        "9     4.9\n",
        "10    5.4\n",
        "11    4.8\n",
        "12    4.8\n",
        "13    4.3\n",
        "14    5.8\n",
        "...\n",
        "135    7.7\n",
        "136    6.3\n",
        "137    6.4\n",
        "138    6.0\n",
        "139    6.9\n",
        "140    6.7\n",
        "141    6.9\n",
        "142    5.8\n",
        "143    6.8\n",
        "144    6.7\n",
        "145    6.7\n",
        "146    6.3\n",
        "147    6.5\n",
        "148    6.2\n",
        "149    5.9\n",
        "Name: SepalLength, Length: 150, dtype: float64"
       ]
      }
     ],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.Name  # columns can also be accessed as attributes"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 28,
       "text": [
        "0     Iris-setosa\n",
        "1     Iris-setosa\n",
        "2     Iris-setosa\n",
        "3     Iris-setosa\n",
        "4     Iris-setosa\n",
        "5     Iris-setosa\n",
        "6     Iris-setosa\n",
        "7     Iris-setosa\n",
        "8     Iris-setosa\n",
        "9     Iris-setosa\n",
        "10    Iris-setosa\n",
        "11    Iris-setosa\n",
        "12    Iris-setosa\n",
        "13    Iris-setosa\n",
        "14    Iris-setosa\n",
        "...\n",
        "135    Iris-virginica\n",
        "136    Iris-virginica\n",
        "137    Iris-virginica\n",
        "138    Iris-virginica\n",
        "139    Iris-virginica\n",
        "140    Iris-virginica\n",
        "141    Iris-virginica\n",
        "142    Iris-virginica\n",
        "143    Iris-virginica\n",
        "144    Iris-virginica\n",
        "145    Iris-virginica\n",
        "146    Iris-virginica\n",
        "147    Iris-virginica\n",
        "148    Iris-virginica\n",
        "149    Iris-virginica\n",
        "Name: Name, Length: 150, dtype: object"
       ]
      }
     ],
     "prompt_number": 28
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.head(10)[[0, 2]]  # columns 0 (SepalLength) and 2 (PetalLength)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SepalLength</th>\n",
        "      <th>PetalLength</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 5.1</td>\n",
        "      <td> 1.4</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 4.9</td>\n",
        "      <td> 1.4</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 4.7</td>\n",
        "      <td> 1.3</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 4.6</td>\n",
        "      <td> 1.5</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 5.0</td>\n",
        "      <td> 1.4</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5</th>\n",
        "      <td> 5.4</td>\n",
        "      <td> 1.7</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6</th>\n",
        "      <td> 4.6</td>\n",
        "      <td> 1.4</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7</th>\n",
        "      <td> 5.0</td>\n",
        "      <td> 1.5</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8</th>\n",
        "      <td> 4.4</td>\n",
        "      <td> 1.4</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9</th>\n",
        "      <td> 4.9</td>\n",
        "      <td> 1.5</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 29,
       "text": [
        "   SepalLength  PetalLength\n",
        "0          5.1          1.4\n",
        "1          4.9          1.4\n",
        "2          4.7          1.3\n",
        "3          4.6          1.5\n",
        "4          5.0          1.4\n",
        "5          5.4          1.7\n",
        "6          4.6          1.4\n",
        "7          5.0          1.5\n",
        "8          4.4          1.4\n",
        "9          4.9          1.5"
       ]
      }
     ],
     "prompt_number": 29
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "However, *slicing and boolean indexing extracts rows*, not columns. You will probably find this a bit inconsistent but that's the way it is."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.head(10)[:2]  # rows 0 through 2 (inclusive), not columns"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SepalLength</th>\n",
        "      <th>SepalWidth</th>\n",
        "      <th>PetalLength</th>\n",
        "      <th>PetalWidth</th>\n",
        "      <th>Name</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 5.1</td>\n",
        "      <td> 3.5</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 4.9</td>\n",
        "      <td> 3.0</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 30,
       "text": [
        "   SepalLength  SepalWidth  PetalLength  PetalWidth         Name\n",
        "0          5.1         3.5          1.4         0.2  Iris-setosa\n",
        "1          4.9         3.0          1.4         0.2  Iris-setosa"
       ]
      }
     ],
     "prompt_number": 30
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# return rows where PetalLength is at least 90% of the maximum PetalLength\n",
      "iris_data[iris_data['PetalLength'] > 0.9*np.max(iris_data['PetalLength'])]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SepalLength</th>\n",
        "      <th>SepalWidth</th>\n",
        "      <th>PetalLength</th>\n",
        "      <th>PetalWidth</th>\n",
        "      <th>Name</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>105</th>\n",
        "      <td> 7.6</td>\n",
        "      <td> 3.0</td>\n",
        "      <td> 6.6</td>\n",
        "      <td> 2.1</td>\n",
        "      <td> Iris-virginica</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>107</th>\n",
        "      <td> 7.3</td>\n",
        "      <td> 2.9</td>\n",
        "      <td> 6.3</td>\n",
        "      <td> 1.8</td>\n",
        "      <td> Iris-virginica</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>117</th>\n",
        "      <td> 7.7</td>\n",
        "      <td> 3.8</td>\n",
        "      <td> 6.7</td>\n",
        "      <td> 2.2</td>\n",
        "      <td> Iris-virginica</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>118</th>\n",
        "      <td> 7.7</td>\n",
        "      <td> 2.6</td>\n",
        "      <td> 6.9</td>\n",
        "      <td> 2.3</td>\n",
        "      <td> Iris-virginica</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>122</th>\n",
        "      <td> 7.7</td>\n",
        "      <td> 2.8</td>\n",
        "      <td> 6.7</td>\n",
        "      <td> 2.0</td>\n",
        "      <td> Iris-virginica</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>131</th>\n",
        "      <td> 7.9</td>\n",
        "      <td> 3.8</td>\n",
        "      <td> 6.4</td>\n",
        "      <td> 2.0</td>\n",
        "      <td> Iris-virginica</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 31,
       "text": [
        "     SepalLength  SepalWidth  PetalLength  PetalWidth            Name\n",
        "105          7.6         3.0          6.6         2.1  Iris-virginica\n",
        "107          7.3         2.9          6.3         1.8  Iris-virginica\n",
        "117          7.7         3.8          6.7         2.2  Iris-virginica\n",
        "118          7.7         2.6          6.9         2.3  Iris-virginica\n",
        "122          7.7         2.8          6.7         2.0  Iris-virginica\n",
        "131          7.9         3.8          6.4         2.0  Iris-virginica"
       ]
      }
     ],
     "prompt_number": 31
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.ix[[11, 22, 33]]  # rows 11, 22, and 33 (note the use of the ix() method)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SepalLength</th>\n",
        "      <th>SepalWidth</th>\n",
        "      <th>PetalLength</th>\n",
        "      <th>PetalWidth</th>\n",
        "      <th>Name</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>11</th>\n",
        "      <td> 4.8</td>\n",
        "      <td> 3.4</td>\n",
        "      <td> 1.6</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>22</th>\n",
        "      <td> 4.6</td>\n",
        "      <td> 3.6</td>\n",
        "      <td> 1.0</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>33</th>\n",
        "      <td> 5.5</td>\n",
        "      <td> 4.2</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 32,
       "text": [
        "    SepalLength  SepalWidth  PetalLength  PetalWidth         Name\n",
        "11          4.8         3.4          1.6         0.2  Iris-setosa\n",
        "22          4.6         3.6          1.0         0.2  Iris-setosa\n",
        "33          5.5         4.2          1.4         0.2  Iris-setosa"
       ]
      }
     ],
     "prompt_number": 32
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "names = iris_data['Name']  # extract Name column\n",
      "del iris_data['Name']  # delete Name column\n",
      "iris_data.head(10)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SepalLength</th>\n",
        "      <th>SepalWidth</th>\n",
        "      <th>PetalLength</th>\n",
        "      <th>PetalWidth</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 5.1</td>\n",
        "      <td> 3.5</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 4.9</td>\n",
        "      <td> 3.0</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 4.7</td>\n",
        "      <td> 3.2</td>\n",
        "      <td> 1.3</td>\n",
        "      <td> 0.2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 4.6</td>\n",
        "      <td> 3.1</td>\n",
        "      <td> 1.5</td>\n",
        "      <td> 0.2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 5.0</td>\n",
        "      <td> 3.6</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5</th>\n",
        "      <td> 5.4</td>\n",
        "      <td> 3.9</td>\n",
        "      <td> 1.7</td>\n",
        "      <td> 0.4</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6</th>\n",
        "      <td> 4.6</td>\n",
        "      <td> 3.4</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.3</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7</th>\n",
        "      <td> 5.0</td>\n",
        "      <td> 3.4</td>\n",
        "      <td> 1.5</td>\n",
        "      <td> 0.2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8</th>\n",
        "      <td> 4.4</td>\n",
        "      <td> 2.9</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9</th>\n",
        "      <td> 4.9</td>\n",
        "      <td> 3.1</td>\n",
        "      <td> 1.5</td>\n",
        "      <td> 0.1</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 33,
       "text": [
        "   SepalLength  SepalWidth  PetalLength  PetalWidth\n",
        "0          5.1         3.5          1.4         0.2\n",
        "1          4.9         3.0          1.4         0.2\n",
        "2          4.7         3.2          1.3         0.2\n",
        "3          4.6         3.1          1.5         0.2\n",
        "4          5.0         3.6          1.4         0.2\n",
        "5          5.4         3.9          1.7         0.4\n",
        "6          4.6         3.4          1.4         0.3\n",
        "7          5.0         3.4          1.5         0.2\n",
        "8          4.4         2.9          1.4         0.2\n",
        "9          4.9         3.1          1.5         0.1"
       ]
      }
     ],
     "prompt_number": 33
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data['Species'] = names  # put it back bt with a different column name"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 34
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.ix[[11, 22, 33]][['PetalWidth', 'Species']]  # rows 11, 22, 33 and columns PetalWidth and Species"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PetalWidth</th>\n",
        "      <th>Species</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>11</th>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>22</th>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>33</th>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 35,
       "text": [
        "    PetalWidth      Species\n",
        "11         0.2  Iris-setosa\n",
        "22         0.2  Iris-setosa\n",
        "33         0.2  Iris-setosa"
       ]
      }
     ],
     "prompt_number": 35
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.ix[[11, 22, 33], ['PetalWidth', 'Species']]  # another way, a bit shorter"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>PetalWidth</th>\n",
        "      <th>Species</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>11</th>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>22</th>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>33</th>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 36,
       "text": [
        "    PetalWidth      Species\n",
        "11         0.2  Iris-setosa\n",
        "22         0.2  Iris-setosa\n",
        "33         0.2  Iris-setosa"
       ]
      }
     ],
     "prompt_number": 36
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.head()  # default arg for head() is 5"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SepalLength</th>\n",
        "      <th>SepalWidth</th>\n",
        "      <th>PetalLength</th>\n",
        "      <th>PetalWidth</th>\n",
        "      <th>Species</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 5.1</td>\n",
        "      <td> 3.5</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 4.9</td>\n",
        "      <td> 3.0</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 4.7</td>\n",
        "      <td> 3.2</td>\n",
        "      <td> 1.3</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 4.6</td>\n",
        "      <td> 3.1</td>\n",
        "      <td> 1.5</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 5.0</td>\n",
        "      <td> 3.6</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "      <td> Iris-setosa</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 37,
       "text": [
        "   SepalLength  SepalWidth  PetalLength  PetalWidth      Species\n",
        "0          5.1         3.5          1.4         0.2  Iris-setosa\n",
        "1          4.9         3.0          1.4         0.2  Iris-setosa\n",
        "2          4.7         3.2          1.3         0.2  Iris-setosa\n",
        "3          4.6         3.1          1.5         0.2  Iris-setosa\n",
        "4          5.0         3.6          1.4         0.2  Iris-setosa"
       ]
      }
     ],
     "prompt_number": 37
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_vals = iris_data.values  # just the numeric values in the DataFrame, returned as a numpy array"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 38
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print type(iris_vals)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<type 'numpy.ndarray'>\n"
       ]
      }
     ],
     "prompt_number": 39
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_vals.shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 40,
       "text": [
        "(150, 5)"
       ]
      }
     ],
     "prompt_number": 40
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.ix[:, :4].sum()  # sums of all columns from 0 through 4 (now excluded since we're using ix())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 41,
       "text": [
        "SepalLength    876.5\n",
        "SepalWidth     458.1\n",
        "PetalLength    563.8\n",
        "PetalWidth     179.8\n",
        "dtype: float64"
       ]
      }
     ],
     "prompt_number": 41
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.ix[:, :4].mean()  # means of the same columns"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 42,
       "text": [
        "SepalLength    5.843333\n",
        "SepalWidth     3.054000\n",
        "PetalLength    3.758667\n",
        "PetalWidth     1.198667\n",
        "dtype: float64"
       ]
      }
     ],
     "prompt_number": 42
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.ix[:6, :4]  # rows 1 thru 6 (excluded) and cols 0 thru 4 (excluded)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SepalLength</th>\n",
        "      <th>SepalWidth</th>\n",
        "      <th>PetalLength</th>\n",
        "      <th>PetalWidth</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 5.1</td>\n",
        "      <td> 3.5</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 4.9</td>\n",
        "      <td> 3.0</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 4.7</td>\n",
        "      <td> 3.2</td>\n",
        "      <td> 1.3</td>\n",
        "      <td> 0.2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 4.6</td>\n",
        "      <td> 3.1</td>\n",
        "      <td> 1.5</td>\n",
        "      <td> 0.2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 5.0</td>\n",
        "      <td> 3.6</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5</th>\n",
        "      <td> 5.4</td>\n",
        "      <td> 3.9</td>\n",
        "      <td> 1.7</td>\n",
        "      <td> 0.4</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6</th>\n",
        "      <td> 4.6</td>\n",
        "      <td> 3.4</td>\n",
        "      <td> 1.4</td>\n",
        "      <td> 0.3</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 43,
       "text": [
        "   SepalLength  SepalWidth  PetalLength  PetalWidth\n",
        "0          5.1         3.5          1.4         0.2\n",
        "1          4.9         3.0          1.4         0.2\n",
        "2          4.7         3.2          1.3         0.2\n",
        "3          4.6         3.1          1.5         0.2\n",
        "4          5.0         3.6          1.4         0.2\n",
        "5          5.4         3.9          1.7         0.4\n",
        "6          4.6         3.4          1.4         0.3"
       ]
      }
     ],
     "prompt_number": 43
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iris_data.ix[:6, :4].mean(axis=1)  # take mean across axis 1, i.e. horizontal"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 44,
       "text": [
        "0    2.550\n",
        "1    2.375\n",
        "2    2.350\n",
        "3    2.350\n",
        "4    2.550\n",
        "5    2.850\n",
        "6    2.425\n",
        "dtype: float64"
       ]
      }
     ],
     "prompt_number": 44
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# create a DataFrame with duplicate column names and row index values\n",
      "df = pd.DataFrame(np.random.randn(3, 5), columns=['one', 'two', 'three', 'fournfive', 'fournfive'],\n",
      "                  index=['John','John','Mike'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 45
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>one</th>\n",
        "      <th>two</th>\n",
        "      <th>three</th>\n",
        "      <th>fournfive</th>\n",
        "      <th>fournfive</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>-0.706945</td>\n",
        "      <td> 0.266734</td>\n",
        "      <td>-0.312746</td>\n",
        "      <td> 0.769932</td>\n",
        "      <td>-0.796752</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td> 1.207626</td>\n",
        "      <td>-0.372847</td>\n",
        "      <td> 2.171378</td>\n",
        "      <td>-1.794998</td>\n",
        "      <td> 1.065065</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td> 1.574784</td>\n",
        "      <td>-0.921769</td>\n",
        "      <td>-0.040932</td>\n",
        "      <td>-0.300894</td>\n",
        "      <td>-1.293681</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 46,
       "text": [
        "           one       two     three  fournfive  fournfive\n",
        "John -0.706945  0.266734 -0.312746   0.769932  -0.796752\n",
        "John  1.207626 -0.372847  2.171378  -1.794998   1.065065\n",
        "Mike  1.574784 -0.921769 -0.040932  -0.300894  -1.293681"
       ]
      }
     ],
     "prompt_number": 46
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.ix['John']  # returns all rows with index 'John'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>one</th>\n",
        "      <th>two</th>\n",
        "      <th>three</th>\n",
        "      <th>fournfive</th>\n",
        "      <th>fournfive</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>-0.706945</td>\n",
        "      <td> 0.266734</td>\n",
        "      <td>-0.312746</td>\n",
        "      <td> 0.769932</td>\n",
        "      <td>-0.796752</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td> 1.207626</td>\n",
        "      <td>-0.372847</td>\n",
        "      <td> 2.171378</td>\n",
        "      <td>-1.794998</td>\n",
        "      <td> 1.065065</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 47,
       "text": [
        "           one       two     three  fournfive  fournfive\n",
        "John -0.706945  0.266734 -0.312746   0.769932  -0.796752\n",
        "John  1.207626 -0.372847  2.171378  -1.794998   1.065065"
       ]
      }
     ],
     "prompt_number": 47
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df['fournfive']  # returns all columns with index 'fournfive'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>fournfive</th>\n",
        "      <th>fournfive</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td> 0.769932</td>\n",
        "      <td>-0.796752</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>-1.794998</td>\n",
        "      <td> 1.065065</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td>-0.300894</td>\n",
        "      <td>-1.293681</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 48,
       "text": [
        "      fournfive  fournfive\n",
        "John   0.769932  -0.796752\n",
        "John  -1.794998   1.065065\n",
        "Mike  -0.300894  -1.293681"
       ]
      }
     ],
     "prompt_number": 48
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.T  # transpose"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>John</th>\n",
        "      <th>John</th>\n",
        "      <th>Mike</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>one</th>\n",
        "      <td>-0.706945</td>\n",
        "      <td> 1.207626</td>\n",
        "      <td> 1.574784</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>two</th>\n",
        "      <td> 0.266734</td>\n",
        "      <td>-0.372847</td>\n",
        "      <td>-0.921769</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>three</th>\n",
        "      <td>-0.312746</td>\n",
        "      <td> 2.171378</td>\n",
        "      <td>-0.040932</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>fournfive</th>\n",
        "      <td> 0.769932</td>\n",
        "      <td>-1.794998</td>\n",
        "      <td>-0.300894</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>fournfive</th>\n",
        "      <td>-0.796752</td>\n",
        "      <td> 1.065065</td>\n",
        "      <td>-1.293681</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 49,
       "text": [
        "               John      John      Mike\n",
        "one       -0.706945  1.207626  1.574784\n",
        "two        0.266734 -0.372847 -0.921769\n",
        "three     -0.312746  2.171378 -0.040932\n",
        "fournfive  0.769932 -1.794998 -0.300894\n",
        "fournfive -0.796752  1.065065 -1.293681"
       ]
      }
     ],
     "prompt_number": 49
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.ix[['John'], ['one', 'three']] = np.nan  # introduce missing values (NaN = Not A Number)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 50
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>one</th>\n",
        "      <th>two</th>\n",
        "      <th>three</th>\n",
        "      <th>fournfive</th>\n",
        "      <th>fournfive</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>      NaN</td>\n",
        "      <td> 0.266734</td>\n",
        "      <td>      NaN</td>\n",
        "      <td> 0.769932</td>\n",
        "      <td>-0.796752</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>      NaN</td>\n",
        "      <td>-0.372847</td>\n",
        "      <td>      NaN</td>\n",
        "      <td>-1.794998</td>\n",
        "      <td> 1.065065</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td> 1.574784</td>\n",
        "      <td>-0.921769</td>\n",
        "      <td>-0.040932</td>\n",
        "      <td>-0.300894</td>\n",
        "      <td>-1.293681</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 51,
       "text": [
        "           one       two     three  fournfive  fournfive\n",
        "John       NaN  0.266734       NaN   0.769932  -0.796752\n",
        "John       NaN -0.372847       NaN  -1.794998   1.065065\n",
        "Mike  1.574784 -0.921769 -0.040932  -0.300894  -1.293681"
       ]
      }
     ],
     "prompt_number": 51
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.isnull()  # return booleans indicating where the missing values are"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>one</th>\n",
        "      <th>two</th>\n",
        "      <th>three</th>\n",
        "      <th>fournfive</th>\n",
        "      <th>fournfive</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>  True</td>\n",
        "      <td> False</td>\n",
        "      <td>  True</td>\n",
        "      <td> False</td>\n",
        "      <td> False</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>  True</td>\n",
        "      <td> False</td>\n",
        "      <td>  True</td>\n",
        "      <td> False</td>\n",
        "      <td> False</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td> False</td>\n",
        "      <td> False</td>\n",
        "      <td> False</td>\n",
        "      <td> False</td>\n",
        "      <td> False</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 52,
       "text": [
        "        one    two  three fournfive fournfive\n",
        "John   True  False   True     False     False\n",
        "John   True  False   True     False     False\n",
        "Mike  False  False  False     False     False"
       ]
      }
     ],
     "prompt_number": 52
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.notnull()  # boolean complement of isnull()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>one</th>\n",
        "      <th>two</th>\n",
        "      <th>three</th>\n",
        "      <th>fournfive</th>\n",
        "      <th>fournfive</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td> False</td>\n",
        "      <td> True</td>\n",
        "      <td> False</td>\n",
        "      <td> True</td>\n",
        "      <td> True</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td> False</td>\n",
        "      <td> True</td>\n",
        "      <td> False</td>\n",
        "      <td> True</td>\n",
        "      <td> True</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td>  True</td>\n",
        "      <td> True</td>\n",
        "      <td>  True</td>\n",
        "      <td> True</td>\n",
        "      <td> True</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 53,
       "text": [
        "        one   two  three fournfive fournfive\n",
        "John  False  True  False      True      True\n",
        "John  False  True  False      True      True\n",
        "Mike   True  True   True      True      True"
       ]
      }
     ],
     "prompt_number": 53
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.dropna()  # returns a copy with rows containing missing vals dropped"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>one</th>\n",
        "      <th>two</th>\n",
        "      <th>three</th>\n",
        "      <th>fournfive</th>\n",
        "      <th>fournfive</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td> 1.574784</td>\n",
        "      <td>-0.921769</td>\n",
        "      <td>-0.040932</td>\n",
        "      <td>-0.300894</td>\n",
        "      <td>-1.293681</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 54,
       "text": [
        "           one       two     three  fournfive  fournfive\n",
        "Mike  1.574784 -0.921769 -0.040932  -0.300894  -1.293681"
       ]
      }
     ],
     "prompt_number": 54
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.dropna(axis=1)  # drop cols containing missing vals"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>two</th>\n",
        "      <th>fournfive</th>\n",
        "      <th>fournfive</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td> 0.266734</td>\n",
        "      <td> 0.769932</td>\n",
        "      <td>-0.796752</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>-0.372847</td>\n",
        "      <td>-1.794998</td>\n",
        "      <td> 1.065065</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td>-0.921769</td>\n",
        "      <td>-0.300894</td>\n",
        "      <td>-1.293681</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 55,
       "text": [
        "           two  fournfive  fournfive\n",
        "John  0.266734   0.769932  -0.796752\n",
        "John -0.372847  -1.794998   1.065065\n",
        "Mike -0.921769  -0.300894  -1.293681"
       ]
      }
     ],
     "prompt_number": 55
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.fillna(0)  # returns a copy with missing vals replaced by 0"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>one</th>\n",
        "      <th>two</th>\n",
        "      <th>three</th>\n",
        "      <th>fournfive</th>\n",
        "      <th>fournfive</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.266734</td>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.769932</td>\n",
        "      <td>-0.796752</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td>-0.372847</td>\n",
        "      <td> 0.000000</td>\n",
        "      <td>-1.794998</td>\n",
        "      <td> 1.065065</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td> 1.574784</td>\n",
        "      <td>-0.921769</td>\n",
        "      <td>-0.040932</td>\n",
        "      <td>-0.300894</td>\n",
        "      <td>-1.293681</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 56,
       "text": [
        "           one       two     three  fournfive  fournfive\n",
        "John  0.000000  0.266734  0.000000   0.769932  -0.796752\n",
        "John  0.000000 -0.372847  0.000000  -1.794998   1.065065\n",
        "Mike  1.574784 -0.921769 -0.040932  -0.300894  -1.293681"
       ]
      }
     ],
     "prompt_number": 56
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df  # as we said, dropna() and fillna() return copies"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>one</th>\n",
        "      <th>two</th>\n",
        "      <th>three</th>\n",
        "      <th>fournfive</th>\n",
        "      <th>fournfive</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>      NaN</td>\n",
        "      <td> 0.266734</td>\n",
        "      <td>      NaN</td>\n",
        "      <td> 0.769932</td>\n",
        "      <td>-0.796752</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>      NaN</td>\n",
        "      <td>-0.372847</td>\n",
        "      <td>      NaN</td>\n",
        "      <td>-1.794998</td>\n",
        "      <td> 1.065065</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td> 1.574784</td>\n",
        "      <td>-0.921769</td>\n",
        "      <td>-0.040932</td>\n",
        "      <td>-0.300894</td>\n",
        "      <td>-1.293681</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 57,
       "text": [
        "           one       two     three  fournfive  fournfive\n",
        "John       NaN  0.266734       NaN   0.769932  -0.796752\n",
        "John       NaN -0.372847       NaN  -1.794998   1.065065\n",
        "Mike  1.574784 -0.921769 -0.040932  -0.300894  -1.293681"
       ]
      }
     ],
     "prompt_number": 57
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.ix[['Mike'], ['two', 'fournfive']] = np.nan  # introduce 3 mores NaNs in Mike's row\n",
      "df"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>one</th>\n",
        "      <th>two</th>\n",
        "      <th>three</th>\n",
        "      <th>fournfive</th>\n",
        "      <th>fournfive</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>      NaN</td>\n",
        "      <td> 0.266734</td>\n",
        "      <td>      NaN</td>\n",
        "      <td> 0.769932</td>\n",
        "      <td>-0.796752</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>      NaN</td>\n",
        "      <td>-0.372847</td>\n",
        "      <td>      NaN</td>\n",
        "      <td>-1.794998</td>\n",
        "      <td> 1.065065</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td> 1.574784</td>\n",
        "      <td>      NaN</td>\n",
        "      <td>-0.040932</td>\n",
        "      <td>      NaN</td>\n",
        "      <td>      NaN</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 58,
       "text": [
        "           one       two     three  fournfive  fournfive\n",
        "John       NaN  0.266734       NaN   0.769932  -0.796752\n",
        "John       NaN -0.372847       NaN  -1.794998   1.065065\n",
        "Mike  1.574784       NaN -0.040932        NaN        NaN"
       ]
      }
     ],
     "prompt_number": 58
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.dropna(thresh=3)  # only Mike will get dropped since our threshold is 3 missing values"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>one</th>\n",
        "      <th>two</th>\n",
        "      <th>three</th>\n",
        "      <th>fournfive</th>\n",
        "      <th>fournfive</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>NaN</td>\n",
        "      <td> 0.266734</td>\n",
        "      <td>NaN</td>\n",
        "      <td> 0.769932</td>\n",
        "      <td>-0.796752</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td>NaN</td>\n",
        "      <td>-0.372847</td>\n",
        "      <td>NaN</td>\n",
        "      <td>-1.794998</td>\n",
        "      <td> 1.065065</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 59,
       "text": [
        "      one       two  three  fournfive  fournfive\n",
        "John  NaN  0.266734    NaN   0.769932  -0.796752\n",
        "John  NaN -0.372847    NaN  -1.794998   1.065065"
       ]
      }
     ],
     "prompt_number": 59
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Panel"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[Panel data](http://en.wikipedia.org/wiki/Panel_data) is an econometric term (for what would be called Longitudinal data in other fields such as Biostatistics or Social Sciences).\n",
      "\n",
      "Pandas, the name of the Python module we're studying, is itself partly derived from Pan[el]-da[ta]-s."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "vals1 = np.array([[60000, 29],\n",
      "                  [45000, 45],\n",
      "                  [70000, 35]])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 60
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df1 = pd.DataFrame(vals1, columns=['income','age'], index=['Mike', 'John', 'Jane'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 61
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>income</th>\n",
        "      <th>age</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td> 60000</td>\n",
        "      <td> 29</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td> 45000</td>\n",
        "      <td> 45</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Jane</th>\n",
        "      <td> 70000</td>\n",
        "      <td> 35</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 62,
       "text": [
        "      income  age\n",
        "Mike   60000   29\n",
        "John   45000   45\n",
        "Jane   70000   35"
       ]
      }
     ],
     "prompt_number": 62
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "vals2 = np.array([[61000, 30],\n",
      "                  [46000, 46],\n",
      "                  [73000, 36]])\n",
      "vals3 = np.array([[63000, 31],\n",
      "                  [47000, 47],\n",
      "                  [74000, 37]])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 63
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df2 = pd.DataFrame(vals2, columns=['income','age'], index=['Mike', 'John', 'Jane'])\n",
      "df3 = pd.DataFrame(vals3, columns=['income','age'], index=['Mike', 'John', 'Jane'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 64
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "panel = pd.Panel({'2010': df1, '2011': df2, '2012': df3})"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 65
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "panel"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 66,
       "text": [
        "<class 'pandas.core.panel.Panel'>\n",
        "Dimensions: 3 (items) x 3 (major_axis) x 2 (minor_axis)\n",
        "Items axis: 2010 to 2012\n",
        "Major_axis axis: Mike to Jane\n",
        "Minor_axis axis: income to age"
       ]
      }
     ],
     "prompt_number": 66
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can think of a Panel as stacked up DataFrames. The `items` axis (axis 0) gives individual DataFrames, the `major` axis (axis 1) corresponds to index (rows) of individual DataFrames, and the `minor` axis corresponds to columns of individual DataFrames. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "panel['2011']  # df for year 2011"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>income</th>\n",
        "      <th>age</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td> 61000</td>\n",
        "      <td> 30</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td> 46000</td>\n",
        "      <td> 46</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>Jane</th>\n",
        "      <td> 73000</td>\n",
        "      <td> 36</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 67,
       "text": [
        "      income  age\n",
        "Mike   61000   30\n",
        "John   46000   46\n",
        "Jane   73000   36"
       ]
      }
     ],
     "prompt_number": 67
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "panel.ix['2011':'2012']  # a sub-panel with data only for years 2011 and 2012"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 68,
       "text": [
        "<class 'pandas.core.panel.Panel'>\n",
        "Dimensions: 2 (items) x 3 (major_axis) x 2 (minor_axis)\n",
        "Items axis: 2011 to 2012\n",
        "Major_axis axis: Mike to Jane\n",
        "Minor_axis axis: income to age"
       ]
      }
     ],
     "prompt_number": 68
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "panel.ix[:, 'Jane', :]  # Jane's data only"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>2010</th>\n",
        "      <th>2011</th>\n",
        "      <th>2012</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>income</th>\n",
        "      <td> 70000</td>\n",
        "      <td> 73000</td>\n",
        "      <td> 74000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>age</th>\n",
        "      <td>    35</td>\n",
        "      <td>    36</td>\n",
        "      <td>    37</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 69,
       "text": [
        "         2010   2011   2012\n",
        "income  70000  73000  74000\n",
        "age        35     36     37"
       ]
      }
     ],
     "prompt_number": 69
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "panel.ix['2011', 'Mike':'John', :]  # Mike thru John's data for the year 2011"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>income</th>\n",
        "      <th>age</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>Mike</th>\n",
        "      <td> 61000</td>\n",
        "      <td> 30</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>John</th>\n",
        "      <td> 46000</td>\n",
        "      <td> 46</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 70,
       "text": [
        "      income  age\n",
        "Mike   61000   30\n",
        "John   46000   46"
       ]
      }
     ],
     "prompt_number": 70
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "another_panel = panel.ix['2011':, 'John':, :]  # sub-panel with data for years 2011 and onwards, for John and Jane\n",
      "another_panel"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 71,
       "text": [
        "<class 'pandas.core.panel.Panel'>\n",
        "Dimensions: 2 (items) x 2 (major_axis) x 2 (minor_axis)\n",
        "Items axis: 2011 to 2012\n",
        "Major_axis axis: John to Jane\n",
        "Minor_axis axis: income to age"
       ]
      }
     ],
     "prompt_number": 71
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As the [pandas documentation](http://pandas.pydata.org/pandas-docs/stable/dsintro.html#from-dict-of-dataframe-objects) says:\n",
      "    \n",
      "\"Unfortunately Panel, being less commonly used than Series and DataFrame, has been slightly neglected feature-wise. A number of methods and options available in DataFrame are not available in Panel. This will get worked on, of course, in future releases. And faster if you join me in working on the codebase.\""
     ]
    }
   ],
   "metadata": {}
  }
 ]
}