Chapter 07 Grouping for Aggregation, Filtration and Transformation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 7: Grouping for Aggregation, Filtration and Transformation\n",
    "## Recipes\n",
    "* [Defining an aggregation](#Defining-an-aggregation)\n",
    "* [Grouping and aggregating with multiple columns and functions](#Grouping-and-aggregating-with-multiple-columns-and-functions)\n",
    "* [Removing the MultiIndex after grouping](#Removing-the-MultiIndex-after-grouping)\n",
    "* [Customizing an aggregation function](#Customizing-an-aggregation-function)\n",
    "* [Customizing aggregating functions with \\*args and \\*\\*kwargs](#Customizing-aggregating-functions-with-*args-and-**kwargs)\n",
    "* [Examining the groupby object](#Examining-a-groupby-object)\n",
    "* [Filtering for states with a minority majority](#Filtering-for-states-with-a-minority-majority)\n",
    "* [Transforming through a weight loss bet](#Transforming-through-a-weight-loss)\n",
    "* [Calculating weighted mean SAT scores per state with apply](#Calculating-weighted-mean-SAT-scores-per-state-with-apply)\n",
    "* [Grouping by continuous variables](#Grouping-by-continuous-variables)\n",
    "* [Counting the total number of flights between cities](#Counting-the-total-number-of-flights-between-cities)\n",
    "* [Finding the longest streak of on-time flights](#Finding-the-longest-streak-of-on-time-flights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining an aggregation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MONTH</th>\n",
       "      <th>DAY</th>\n",
       "      <th>WEEKDAY</th>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>ORG_AIR</th>\n",
       "      <th>DEST_AIR</th>\n",
       "      <th>SCHED_DEP</th>\n",
       "      <th>DEP_DELAY</th>\n",
       "      <th>AIR_TIME</th>\n",
       "      <th>DIST</th>\n",
       "      <th>SCHED_ARR</th>\n",
       "      <th>ARR_DELAY</th>\n",
       "      <th>DIVERTED</th>\n",
       "      <th>CANCELLED</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>SLC</td>\n",
       "      <td>1625</td>\n",
       "      <td>58.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>590</td>\n",
       "      <td>1905</td>\n",
       "      <td>65.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>UA</td>\n",
       "      <td>DEN</td>\n",
       "      <td>IAD</td>\n",
       "      <td>823</td>\n",
       "      <td>7.0</td>\n",
       "      <td>154.0</td>\n",
       "      <td>1452</td>\n",
       "      <td>1333</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>MQ</td>\n",
       "      <td>DFW</td>\n",
       "      <td>VPS</td>\n",
       "      <td>1305</td>\n",
       "      <td>36.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>641</td>\n",
       "      <td>1453</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>AA</td>\n",
       "      <td>DFW</td>\n",
       "      <td>DCA</td>\n",
       "      <td>1555</td>\n",
       "      <td>7.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>1192</td>\n",
       "      <td>1935</td>\n",
       "      <td>-7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>MCI</td>\n",
       "      <td>1720</td>\n",
       "      <td>48.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>1363</td>\n",
       "      <td>2225</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MONTH  DAY  WEEKDAY AIRLINE ORG_AIR DEST_AIR  SCHED_DEP  DEP_DELAY  \\\n",
       "0      1    1        4      WN     LAX      SLC       1625       58.0   \n",
       "1      1    1        4      UA     DEN      IAD        823        7.0   \n",
       "2      1    1        4      MQ     DFW      VPS       1305       36.0   \n",
       "3      1    1        4      AA     DFW      DCA       1555        7.0   \n",
       "4      1    1        4      WN     LAX      MCI       1720       48.0   \n",
       "\n",
       "   AIR_TIME  DIST  SCHED_ARR  ARR_DELAY  DIVERTED  CANCELLED  \n",
       "0      94.0   590       1905       65.0         0          0  \n",
       "1     154.0  1452       1333      -13.0         0          0  \n",
       "2      85.0   641       1453       35.0         0          0  \n",
       "3     126.0  1192       1935       -7.0         0          0  \n",
       "4     166.0  1363       2225       39.0         0          0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights = pd.read_csv('data/flights.csv')\n",
    "flights.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIRLINE\n",
       "AA    5.542661\n",
       "AS   -0.833333\n",
       "B6    8.692593\n",
       "DL    0.339691\n",
       "EV    7.034580\n",
       "Name: ARR_DELAY, dtype: float64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.groupby('AIRLINE')['ARR_DELAY'].agg('mean').head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ARR_DELAY</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AIRLINE</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AA</th>\n",
       "      <td>5.542661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AS</th>\n",
       "      <td>-0.833333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B6</th>\n",
       "      <td>8.692593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DL</th>\n",
       "      <td>0.339691</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EV</th>\n",
       "      <td>7.034580</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         ARR_DELAY\n",
       "AIRLINE           \n",
       "AA        5.542661\n",
       "AS       -0.833333\n",
       "B6        8.692593\n",
       "DL        0.339691\n",
       "EV        7.034580"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.groupby('AIRLINE').agg({'ARR_DELAY':'mean'}).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIRLINE\n",
       "AA    5.542661\n",
       "AS   -0.833333\n",
       "B6    8.692593\n",
       "DL    0.339691\n",
       "EV    7.034580\n",
       "Name: ARR_DELAY, dtype: float64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.groupby('AIRLINE')['ARR_DELAY'].agg(np.mean).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIRLINE\n",
       "AA    5.542661\n",
       "AS   -0.833333\n",
       "B6    8.692593\n",
       "DL    0.339691\n",
       "EV    7.034580\n",
       "Name: ARR_DELAY, dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.groupby('AIRLINE')['ARR_DELAY'].mean().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How it works..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.groupby.DataFrameGroupBy"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped = flights.groupby('AIRLINE')\n",
    "type(grouped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## There's more"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py:842: RuntimeWarning: invalid value encountered in sqrt\n",
      "  f = lambda x: func(x, *args, **kwargs)\n",
      "/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py:3015: RuntimeWarning: invalid value encountered in sqrt\n",
      "  output = func(group, *args, **kwargs)\n"
     ]
    },
    {
     "ename": "Exception",
     "evalue": "Must produce aggregated value",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36magg_series\u001b[0;34m(self, obj, func)\u001b[0m\n\u001b[1;32m   2177\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2178\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_aggregate_series_fast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2179\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m_aggregate_series_fast\u001b[0;34m(self, obj, func)\u001b[0m\n\u001b[1;32m   2197\u001b[0m                                     dummy)\n\u001b[0;32m-> 2198\u001b[0;31m         \u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcounts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrouper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_result\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2199\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcounts\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/src/reduce.pyx\u001b[0m in \u001b[0;36mpandas._libs.lib.SeriesGrouper.get_result (pandas/_libs/lib.c:39105)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/src/reduce.pyx\u001b[0m in \u001b[0;36mpandas._libs.lib.SeriesGrouper.get_result (pandas/_libs/lib.c:38973)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/src/reduce.pyx\u001b[0m in \u001b[0;36mpandas._libs.lib._get_result_array (pandas/_libs/lib.c:32039)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: function does not reduce",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36maggregate\u001b[0;34m(self, func_or_funcs, *args, **kwargs)\u001b[0m\n\u001b[1;32m   2882\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2883\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_python_agg_general\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc_or_funcs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2884\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m_python_agg_general\u001b[0;34m(self, func, *args, **kwargs)\u001b[0m\n\u001b[1;32m    847\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 848\u001b[0;31m                 \u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcounts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrouper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0magg_series\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    849\u001b[0m                 \u001b[0moutput\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_try_cast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumeric_only\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36magg_series\u001b[0;34m(self, obj, func)\u001b[0m\n\u001b[1;32m   2179\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2180\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_aggregate_series_pure_python\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2181\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m_aggregate_series_pure_python\u001b[0;34m(self, obj, func)\u001b[0m\n\u001b[1;32m   2214\u001b[0m                         isinstance(res, list)):\n\u001b[0;32m-> 2215\u001b[0;31m                     \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Function does not reduce'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2216\u001b[0m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mempty\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mngroups\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'O'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Function does not reduce",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mException\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-8-2bcc9ccfec77>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mflights\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroupby\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'AIRLINE'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'ARR_DELAY'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0magg\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36maggregate\u001b[0;34m(self, func_or_funcs, *args, **kwargs)\u001b[0m\n\u001b[1;32m   2883\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_python_agg_general\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc_or_funcs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2884\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2885\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_aggregate_named\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc_or_funcs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2886\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2887\u001b[0m             \u001b[0mindex\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mIndex\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msorted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrouper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnames\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m_aggregate_named\u001b[0;34m(self, func, *args, **kwargs)\u001b[0m\n\u001b[1;32m   3015\u001b[0m             \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3016\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mSeries\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mIndex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mndarray\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3017\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Must produce aggregated value'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3018\u001b[0m             \u001b[0mresult\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_try_cast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgroup\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3019\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mException\u001b[0m: Must produce aggregated value"
     ]
    }
   ],
   "source": [
    "flights.groupby('AIRLINE')['ARR_DELAY'].agg(np.sqrt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grouping and aggregating with multiple columns and functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MONTH</th>\n",
       "      <th>DAY</th>\n",
       "      <th>WEEKDAY</th>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>ORG_AIR</th>\n",
       "      <th>DEST_AIR</th>\n",
       "      <th>SCHED_DEP</th>\n",
       "      <th>DEP_DELAY</th>\n",
       "      <th>AIR_TIME</th>\n",
       "      <th>DIST</th>\n",
       "      <th>SCHED_ARR</th>\n",
       "      <th>ARR_DELAY</th>\n",
       "      <th>DIVERTED</th>\n",
       "      <th>CANCELLED</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>SLC</td>\n",
       "      <td>1625</td>\n",
       "      <td>58.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>590</td>\n",
       "      <td>1905</td>\n",
       "      <td>65.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>UA</td>\n",
       "      <td>DEN</td>\n",
       "      <td>IAD</td>\n",
       "      <td>823</td>\n",
       "      <td>7.0</td>\n",
       "      <td>154.0</td>\n",
       "      <td>1452</td>\n",
       "      <td>1333</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>MQ</td>\n",
       "      <td>DFW</td>\n",
       "      <td>VPS</td>\n",
       "      <td>1305</td>\n",
       "      <td>36.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>641</td>\n",
       "      <td>1453</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>AA</td>\n",
       "      <td>DFW</td>\n",
       "      <td>DCA</td>\n",
       "      <td>1555</td>\n",
       "      <td>7.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>1192</td>\n",
       "      <td>1935</td>\n",
       "      <td>-7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>MCI</td>\n",
       "      <td>1720</td>\n",
       "      <td>48.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>1363</td>\n",
       "      <td>2225</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MONTH  DAY  WEEKDAY AIRLINE ORG_AIR DEST_AIR  SCHED_DEP  DEP_DELAY  \\\n",
       "0      1    1        4      WN     LAX      SLC       1625       58.0   \n",
       "1      1    1        4      UA     DEN      IAD        823        7.0   \n",
       "2      1    1        4      MQ     DFW      VPS       1305       36.0   \n",
       "3      1    1        4      AA     DFW      DCA       1555        7.0   \n",
       "4      1    1        4      WN     LAX      MCI       1720       48.0   \n",
       "\n",
       "   AIR_TIME  DIST  SCHED_ARR  ARR_DELAY  DIVERTED  CANCELLED  \n",
       "0      94.0   590       1905       65.0         0          0  \n",
       "1     154.0  1452       1333      -13.0         0          0  \n",
       "2      85.0   641       1453       35.0         0          0  \n",
       "3     126.0  1192       1935       -7.0         0          0  \n",
       "4     166.0  1363       2225       39.0         0          0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights = pd.read_csv('data/flights.csv')\n",
    "flights.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIRLINE  WEEKDAY\n",
       "AA       1          41\n",
       "         2           9\n",
       "         3          16\n",
       "         4          20\n",
       "         5          18\n",
       "         6          21\n",
       "         7          29\n",
       "Name: CANCELLED, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The number of cancelled flights for every airline per day weekday\n",
    "flights.groupby(['AIRLINE', 'WEEKDAY'])['CANCELLED'].agg('sum').head(7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">CANCELLED</th>\n",
       "      <th colspan=\"2\" halign=\"left\">DIVERTED</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>WEEKDAY</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"7\" valign=\"top\">AA</th>\n",
       "      <th>1</th>\n",
       "      <td>41</td>\n",
       "      <td>0.032106</td>\n",
       "      <td>6</td>\n",
       "      <td>0.004699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>0.007341</td>\n",
       "      <td>2</td>\n",
       "      <td>0.001631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>0.011949</td>\n",
       "      <td>2</td>\n",
       "      <td>0.001494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20</td>\n",
       "      <td>0.015004</td>\n",
       "      <td>5</td>\n",
       "      <td>0.003751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>18</td>\n",
       "      <td>0.014151</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>21</td>\n",
       "      <td>0.018667</td>\n",
       "      <td>9</td>\n",
       "      <td>0.008000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>29</td>\n",
       "      <td>0.021837</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000753</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                CANCELLED           DIVERTED          \n",
       "                      sum      mean      sum      mean\n",
       "AIRLINE WEEKDAY                                       \n",
       "AA      1              41  0.032106        6  0.004699\n",
       "        2               9  0.007341        2  0.001631\n",
       "        3              16  0.011949        2  0.001494\n",
       "        4              20  0.015004        5  0.003751\n",
       "        5              18  0.014151        1  0.000786\n",
       "        6              21  0.018667        9  0.008000\n",
       "        7              29  0.021837        1  0.000753"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Find the number and percentage of cancelled and diverted flights for every airline per weekday\n",
    "flights.groupby(['AIRLINE', 'WEEKDAY'])['CANCELLED', 'DIVERTED'].agg(['sum', 'mean']).head(7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">CANCELLED</th>\n",
       "      <th colspan=\"2\" halign=\"left\">AIR_TIME</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "      <th>size</th>\n",
       "      <th>mean</th>\n",
       "      <th>var</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ORG_AIR</th>\n",
       "      <th>DEST_AIR</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">ATL</th>\n",
       "      <th>ABE</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>31</td>\n",
       "      <td>96.387097</td>\n",
       "      <td>45.778495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ABQ</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16</td>\n",
       "      <td>170.500000</td>\n",
       "      <td>87.866667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ABY</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19</td>\n",
       "      <td>28.578947</td>\n",
       "      <td>6.590643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ACY</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6</td>\n",
       "      <td>91.333333</td>\n",
       "      <td>11.466667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AEX</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40</td>\n",
       "      <td>78.725000</td>\n",
       "      <td>47.332692</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 CANCELLED              AIR_TIME           \n",
       "                       sum mean size        mean        var\n",
       "ORG_AIR DEST_AIR                                           \n",
       "ATL     ABE              0  0.0   31   96.387097  45.778495\n",
       "        ABQ              0  0.0   16  170.500000  87.866667\n",
       "        ABY              0  0.0   19   28.578947   6.590643\n",
       "        ACY              0  0.0    6   91.333333  11.466667\n",
       "        AEX              0  0.0   40   78.725000  47.332692"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For each origin to destination flight, find the total number of flights, \n",
    "# the number and percentage of cancelled flights and the average and variance of the airtime. \n",
    "group_cols = ['ORG_AIR', 'DEST_AIR']\n",
    "agg_dict = {'CANCELLED':['sum', 'mean', 'size'], \n",
    "            'AIR_TIME':['mean', 'var']}\n",
    "flights.groupby(group_cols).agg(agg_dict).head()\n",
    "# flights.groupby(['ORG_AIR', 'DEST_AIR']).agg({'CANCELLED': ['sum', 'mean', 'size'], \n",
    "#                                               'AIR_TIME':['mean', 'var']}).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Removing the MultiIndex after grouping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MONTH</th>\n",
       "      <th>DAY</th>\n",
       "      <th>WEEKDAY</th>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>ORG_AIR</th>\n",
       "      <th>DEST_AIR</th>\n",
       "      <th>SCHED_DEP</th>\n",
       "      <th>DEP_DELAY</th>\n",
       "      <th>AIR_TIME</th>\n",
       "      <th>DIST</th>\n",
       "      <th>SCHED_ARR</th>\n",
       "      <th>ARR_DELAY</th>\n",
       "      <th>DIVERTED</th>\n",
       "      <th>CANCELLED</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>SLC</td>\n",
       "      <td>1625</td>\n",
       "      <td>58.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>590</td>\n",
       "      <td>1905</td>\n",
       "      <td>65.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>UA</td>\n",
       "      <td>DEN</td>\n",
       "      <td>IAD</td>\n",
       "      <td>823</td>\n",
       "      <td>7.0</td>\n",
       "      <td>154.0</td>\n",
       "      <td>1452</td>\n",
       "      <td>1333</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>MQ</td>\n",
       "      <td>DFW</td>\n",
       "      <td>VPS</td>\n",
       "      <td>1305</td>\n",
       "      <td>36.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>641</td>\n",
       "      <td>1453</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>AA</td>\n",
       "      <td>DFW</td>\n",
       "      <td>DCA</td>\n",
       "      <td>1555</td>\n",
       "      <td>7.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>1192</td>\n",
       "      <td>1935</td>\n",
       "      <td>-7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>MCI</td>\n",
       "      <td>1720</td>\n",
       "      <td>48.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>1363</td>\n",
       "      <td>2225</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MONTH  DAY  WEEKDAY AIRLINE ORG_AIR DEST_AIR  SCHED_DEP  DEP_DELAY  \\\n",
       "0      1    1        4      WN     LAX      SLC       1625       58.0   \n",
       "1      1    1        4      UA     DEN      IAD        823        7.0   \n",
       "2      1    1        4      MQ     DFW      VPS       1305       36.0   \n",
       "3      1    1        4      AA     DFW      DCA       1555        7.0   \n",
       "4      1    1        4      WN     LAX      MCI       1720       48.0   \n",
       "\n",
       "   AIR_TIME  DIST  SCHED_ARR  ARR_DELAY  DIVERTED  CANCELLED  \n",
       "0      94.0   590       1905       65.0         0          0  \n",
       "1     154.0  1452       1333      -13.0         0          0  \n",
       "2      85.0   641       1453       35.0         0          0  \n",
       "3     126.0  1192       1935       -7.0         0          0  \n",
       "4     166.0  1363       2225       39.0         0          0  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights = pd.read_csv('data/flights.csv')\n",
    "flights.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">DIST</th>\n",
       "      <th colspan=\"2\" halign=\"left\">ARR_DELAY</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>WEEKDAY</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">AA</th>\n",
       "      <th>1</th>\n",
       "      <td>1455386</td>\n",
       "      <td>1139</td>\n",
       "      <td>-60</td>\n",
       "      <td>551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1358256</td>\n",
       "      <td>1107</td>\n",
       "      <td>-52</td>\n",
       "      <td>725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1496665</td>\n",
       "      <td>1117</td>\n",
       "      <td>-45</td>\n",
       "      <td>473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1452394</td>\n",
       "      <td>1089</td>\n",
       "      <td>-46</td>\n",
       "      <td>349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1427749</td>\n",
       "      <td>1122</td>\n",
       "      <td>-41</td>\n",
       "      <td>732</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    DIST       ARR_DELAY     \n",
       "                     sum  mean       min  max\n",
       "AIRLINE WEEKDAY                              \n",
       "AA      1        1455386  1139       -60  551\n",
       "        2        1358256  1107       -52  725\n",
       "        3        1496665  1117       -45  473\n",
       "        4        1452394  1089       -46  349\n",
       "        5        1427749  1122       -41  732"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "airline_info = flights.groupby(['AIRLINE', 'WEEKDAY'])\\\n",
    "                      .agg({'DIST':['sum', 'mean'], \n",
    "                                    'ARR_DELAY':['min', 'max']}).astype(int)\n",
    "airline_info.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['DIST', 'DIST', 'ARR_DELAY', 'ARR_DELAY'], dtype='object')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "level0 = airline_info.columns.get_level_values(0)\n",
    "level0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['sum', 'mean', 'min', 'max'], dtype='object')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "level1 = airline_info.columns.get_level_values(1)\n",
    "level1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "airline_info.columns = level0 + '_' + level1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>DIST_sum</th>\n",
       "      <th>DIST_mean</th>\n",
       "      <th>ARR_DELAY_min</th>\n",
       "      <th>ARR_DELAY_max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>WEEKDAY</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"7\" valign=\"top\">AA</th>\n",
       "      <th>1</th>\n",
       "      <td>1455386</td>\n",
       "      <td>1139</td>\n",
       "      <td>-60</td>\n",
       "      <td>551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1358256</td>\n",
       "      <td>1107</td>\n",
       "      <td>-52</td>\n",
       "      <td>725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1496665</td>\n",
       "      <td>1117</td>\n",
       "      <td>-45</td>\n",
       "      <td>473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1452394</td>\n",
       "      <td>1089</td>\n",
       "      <td>-46</td>\n",
       "      <td>349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1427749</td>\n",
       "      <td>1122</td>\n",
       "      <td>-41</td>\n",
       "      <td>732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1265340</td>\n",
       "      <td>1124</td>\n",
       "      <td>-50</td>\n",
       "      <td>858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1461906</td>\n",
       "      <td>1100</td>\n",
       "      <td>-49</td>\n",
       "      <td>626</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 DIST_sum  DIST_mean  ARR_DELAY_min  ARR_DELAY_max\n",
       "AIRLINE WEEKDAY                                                   \n",
       "AA      1         1455386       1139            -60            551\n",
       "        2         1358256       1107            -52            725\n",
       "        3         1496665       1117            -45            473\n",
       "        4         1452394       1089            -46            349\n",
       "        5         1427749       1122            -41            732\n",
       "        6         1265340       1124            -50            858\n",
       "        7         1461906       1100            -49            626"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "airline_info.head(7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>WEEKDAY</th>\n",
       "      <th>DIST_sum</th>\n",
       "      <th>DIST_mean</th>\n",
       "      <th>ARR_DELAY_min</th>\n",
       "      <th>ARR_DELAY_max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AA</td>\n",
       "      <td>1</td>\n",
       "      <td>1455386</td>\n",
       "      <td>1139</td>\n",
       "      <td>-60</td>\n",
       "      <td>551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AA</td>\n",
       "      <td>2</td>\n",
       "      <td>1358256</td>\n",
       "      <td>1107</td>\n",
       "      <td>-52</td>\n",
       "      <td>725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AA</td>\n",
       "      <td>3</td>\n",
       "      <td>1496665</td>\n",
       "      <td>1117</td>\n",
       "      <td>-45</td>\n",
       "      <td>473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AA</td>\n",
       "      <td>4</td>\n",
       "      <td>1452394</td>\n",
       "      <td>1089</td>\n",
       "      <td>-46</td>\n",
       "      <td>349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AA</td>\n",
       "      <td>5</td>\n",
       "      <td>1427749</td>\n",
       "      <td>1122</td>\n",
       "      <td>-41</td>\n",
       "      <td>732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>AA</td>\n",
       "      <td>6</td>\n",
       "      <td>1265340</td>\n",
       "      <td>1124</td>\n",
       "      <td>-50</td>\n",
       "      <td>858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>AA</td>\n",
       "      <td>7</td>\n",
       "      <td>1461906</td>\n",
       "      <td>1100</td>\n",
       "      <td>-49</td>\n",
       "      <td>626</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  AIRLINE  WEEKDAY  DIST_sum  DIST_mean  ARR_DELAY_min  ARR_DELAY_max\n",
       "0      AA        1   1455386       1139            -60            551\n",
       "1      AA        2   1358256       1107            -52            725\n",
       "2      AA        3   1496665       1117            -45            473\n",
       "3      AA        4   1452394       1089            -46            349\n",
       "4      AA        5   1427749       1122            -41            732\n",
       "5      AA        6   1265340       1124            -50            858\n",
       "6      AA        7   1461906       1100            -49            626"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "airline_info.reset_index().head(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## There's more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>DIST</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AA</td>\n",
       "      <td>1114.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AS</td>\n",
       "      <td>1066.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B6</td>\n",
       "      <td>1772.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>DL</td>\n",
       "      <td>866.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>EV</td>\n",
       "      <td>460.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>F9</td>\n",
       "      <td>970.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>HA</td>\n",
       "      <td>2615.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>MQ</td>\n",
       "      <td>404.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NK</td>\n",
       "      <td>1047.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>OO</td>\n",
       "      <td>511.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>UA</td>\n",
       "      <td>1231.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>US</td>\n",
       "      <td>1181.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>VX</td>\n",
       "      <td>1240.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>WN</td>\n",
       "      <td>810.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   AIRLINE    DIST\n",
       "0       AA  1114.0\n",
       "1       AS  1066.0\n",
       "2       B6  1772.0\n",
       "3       DL   866.0\n",
       "4       EV   460.0\n",
       "5       F9   970.0\n",
       "6       HA  2615.0\n",
       "7       MQ   404.0\n",
       "8       NK  1047.0\n",
       "9       OO   511.0\n",
       "10      UA  1231.0\n",
       "11      US  1181.0\n",
       "12      VX  1240.0\n",
       "13      WN   810.0"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.groupby(['AIRLINE'], as_index=False)['DIST'].agg('mean').round(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>DIST</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>WN</td>\n",
       "      <td>809.985626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>UA</td>\n",
       "      <td>1230.918891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>MQ</td>\n",
       "      <td>404.229041</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AA</td>\n",
       "      <td>1114.347865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F9</td>\n",
       "      <td>969.593014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>EV</td>\n",
       "      <td>460.237453</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>OO</td>\n",
       "      <td>511.239375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>NK</td>\n",
       "      <td>1047.428100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>US</td>\n",
       "      <td>1181.226625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>AS</td>\n",
       "      <td>1065.884115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>DL</td>\n",
       "      <td>866.448448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>VX</td>\n",
       "      <td>1240.296073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>B6</td>\n",
       "      <td>1771.882136</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>HA</td>\n",
       "      <td>2615.178571</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   AIRLINE         DIST\n",
       "0       WN   809.985626\n",
       "1       UA  1230.918891\n",
       "2       MQ   404.229041\n",
       "3       AA  1114.347865\n",
       "4       F9   969.593014\n",
       "5       EV   460.237453\n",
       "6       OO   511.239375\n",
       "7       NK  1047.428100\n",
       "8       US  1181.226625\n",
       "9       AS  1065.884115\n",
       "10      DL   866.448448\n",
       "11      VX  1240.296073\n",
       "12      B6  1771.882136\n",
       "13      HA  2615.178571"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.groupby(['AIRLINE'], as_index=False, sort=False)['DIST'].agg('mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Customizing an aggregation function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>INSTNM</th>\n",
       "      <th>CITY</th>\n",
       "      <th>STABBR</th>\n",
       "      <th>HBCU</th>\n",
       "      <th>MENONLY</th>\n",
       "      <th>WOMENONLY</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>SATMTMID</th>\n",
       "      <th>DISTANCEONLY</th>\n",
       "      <th>...</th>\n",
       "      <th>UGDS_2MOR</th>\n",
       "      <th>UGDS_NRA</th>\n",
       "      <th>UGDS_UNKN</th>\n",
       "      <th>PPTUG_EF</th>\n",
       "      <th>CURROPER</th>\n",
       "      <th>PCTPELL</th>\n",
       "      <th>PCTFLOAN</th>\n",
       "      <th>UG25ABV</th>\n",
       "      <th>MD_EARN_WNE_P10</th>\n",
       "      <th>GRAD_DEBT_MDN_SUPP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alabama A &amp; M University</td>\n",
       "      <td>Normal</td>\n",
       "      <td>AL</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>424.0</td>\n",
       "      <td>420.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0059</td>\n",
       "      <td>0.0138</td>\n",
       "      <td>0.0656</td>\n",
       "      <td>1</td>\n",
       "      <td>0.7356</td>\n",
       "      <td>0.8284</td>\n",
       "      <td>0.1049</td>\n",
       "      <td>30300</td>\n",
       "      <td>33888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>University of Alabama at Birmingham</td>\n",
       "      <td>Birmingham</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>570.0</td>\n",
       "      <td>565.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0368</td>\n",
       "      <td>0.0179</td>\n",
       "      <td>0.0100</td>\n",
       "      <td>0.2607</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3460</td>\n",
       "      <td>0.5214</td>\n",
       "      <td>0.2422</td>\n",
       "      <td>39700</td>\n",
       "      <td>21941.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Amridge University</td>\n",
       "      <td>Montgomery</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.2715</td>\n",
       "      <td>0.4536</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6801</td>\n",
       "      <td>0.7795</td>\n",
       "      <td>0.8540</td>\n",
       "      <td>40100</td>\n",
       "      <td>23370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>University of Alabama in Huntsville</td>\n",
       "      <td>Huntsville</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>595.0</td>\n",
       "      <td>590.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0172</td>\n",
       "      <td>0.0332</td>\n",
       "      <td>0.0350</td>\n",
       "      <td>0.2146</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3072</td>\n",
       "      <td>0.4596</td>\n",
       "      <td>0.2640</td>\n",
       "      <td>45500</td>\n",
       "      <td>24097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Alabama State University</td>\n",
       "      <td>Montgomery</td>\n",
       "      <td>AL</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>425.0</td>\n",
       "      <td>430.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0098</td>\n",
       "      <td>0.0243</td>\n",
       "      <td>0.0137</td>\n",
       "      <td>0.0892</td>\n",
       "      <td>1</td>\n",
       "      <td>0.7347</td>\n",
       "      <td>0.7554</td>\n",
       "      <td>0.1270</td>\n",
       "      <td>26600</td>\n",
       "      <td>33118.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                INSTNM        CITY STABBR  HBCU  MENONLY  \\\n",
       "0             Alabama A & M University      Normal     AL   1.0      0.0   \n",
       "1  University of Alabama at Birmingham  Birmingham     AL   0.0      0.0   \n",
       "2                   Amridge University  Montgomery     AL   0.0      0.0   \n",
       "3  University of Alabama in Huntsville  Huntsville     AL   0.0      0.0   \n",
       "4             Alabama State University  Montgomery     AL   1.0      0.0   \n",
       "\n",
       "   WOMENONLY  RELAFFIL  SATVRMID  SATMTMID  DISTANCEONLY         ...          \\\n",
       "0        0.0         0     424.0     420.0           0.0         ...           \n",
       "1        0.0         0     570.0     565.0           0.0         ...           \n",
       "2        0.0         1       NaN       NaN           1.0         ...           \n",
       "3        0.0         0     595.0     590.0           0.0         ...           \n",
       "4        0.0         0     425.0     430.0           0.0         ...           \n",
       "\n",
       "   UGDS_2MOR  UGDS_NRA  UGDS_UNKN  PPTUG_EF  CURROPER  PCTPELL  PCTFLOAN  \\\n",
       "0     0.0000    0.0059     0.0138    0.0656         1   0.7356    0.8284   \n",
       "1     0.0368    0.0179     0.0100    0.2607         1   0.3460    0.5214   \n",
       "2     0.0000    0.0000     0.2715    0.4536         1   0.6801    0.7795   \n",
       "3     0.0172    0.0332     0.0350    0.2146         1   0.3072    0.4596   \n",
       "4     0.0098    0.0243     0.0137    0.0892         1   0.7347    0.7554   \n",
       "\n",
       "   UG25ABV  MD_EARN_WNE_P10  GRAD_DEBT_MDN_SUPP  \n",
       "0   0.1049            30300               33888  \n",
       "1   0.2422            39700             21941.5  \n",
       "2   0.8540            40100               23370  \n",
       "3   0.2640            45500               24097  \n",
       "4   0.1270            26600             33118.5  \n",
       "\n",
       "[5 rows x 27 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college = pd.read_csv('data/college.csv')\n",
    "college.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>STABBR</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AK</th>\n",
       "      <td>2493.0</td>\n",
       "      <td>4052.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AL</th>\n",
       "      <td>2790.0</td>\n",
       "      <td>4658.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AR</th>\n",
       "      <td>1644.0</td>\n",
       "      <td>3143.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AS</th>\n",
       "      <td>1276.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AZ</th>\n",
       "      <td>4130.0</td>\n",
       "      <td>14894.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          mean      std\n",
       "STABBR                 \n",
       "AK      2493.0   4052.0\n",
       "AL      2790.0   4658.0\n",
       "AR      1644.0   3143.0\n",
       "AS      1276.0      NaN\n",
       "AZ      4130.0  14894.0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college.groupby('STABBR')['UGDS'].agg(['mean', 'std']).round(0).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def max_deviation(s):\n",
    "    std_score = (s - s.mean()) / s.std()\n",
    "    return std_score.abs().max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "STABBR\n",
       "AK    2.6\n",
       "AL    5.8\n",
       "AR    6.3\n",
       "AS    NaN\n",
       "AZ    9.9\n",
       "Name: UGDS, dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college.groupby('STABBR')['UGDS'].agg(max_deviation).round(1).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## There's more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UGDS</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>SATMTMID</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>STABBR</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AK</th>\n",
       "      <td>2.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AL</th>\n",
       "      <td>5.8</td>\n",
       "      <td>1.6</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AR</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.2</td>\n",
       "      <td>2.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AS</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AZ</th>\n",
       "      <td>9.9</td>\n",
       "      <td>1.9</td>\n",
       "      <td>1.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        UGDS  SATVRMID  SATMTMID\n",
       "STABBR                          \n",
       "AK       2.6       NaN       NaN\n",
       "AL       5.8       1.6       1.8\n",
       "AR       6.3       2.2       2.3\n",
       "AS       NaN       NaN       NaN\n",
       "AZ       9.9       1.9       1.4"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college.groupby('STABBR')['UGDS', 'SATVRMID', 'SATMTMID'].agg(max_deviation).round(1).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">UGDS</th>\n",
       "      <th colspan=\"3\" halign=\"left\">SATVRMID</th>\n",
       "      <th colspan=\"3\" halign=\"left\">SATMTMID</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>max_deviation</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>max_deviation</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>max_deviation</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>STABBR</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">AK</th>\n",
       "      <th>0</th>\n",
       "      <td>2.1</td>\n",
       "      <td>3508.9</td>\n",
       "      <td>4539.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.1</td>\n",
       "      <td>123.3</td>\n",
       "      <td>132.9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>555.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>503.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">AL</th>\n",
       "      <th>0</th>\n",
       "      <td>5.2</td>\n",
       "      <td>3248.8</td>\n",
       "      <td>5102.4</td>\n",
       "      <td>1.6</td>\n",
       "      <td>514.9</td>\n",
       "      <td>56.5</td>\n",
       "      <td>1.7</td>\n",
       "      <td>515.8</td>\n",
       "      <td>56.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.4</td>\n",
       "      <td>979.7</td>\n",
       "      <td>870.8</td>\n",
       "      <td>1.5</td>\n",
       "      <td>498.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>485.6</td>\n",
       "      <td>61.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AR</th>\n",
       "      <th>0</th>\n",
       "      <td>5.8</td>\n",
       "      <td>1793.7</td>\n",
       "      <td>3401.6</td>\n",
       "      <td>1.9</td>\n",
       "      <td>481.1</td>\n",
       "      <td>37.9</td>\n",
       "      <td>2.0</td>\n",
       "      <td>503.6</td>\n",
       "      <td>39.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         UGDS                      SATVRMID               \\\n",
       "                max_deviation    mean     std max_deviation   mean   std   \n",
       "STABBR RELAFFIL                                                            \n",
       "AK     0                  2.1  3508.9  4539.5           NaN    NaN   NaN   \n",
       "       1                  1.1   123.3   132.9           NaN  555.0   NaN   \n",
       "AL     0                  5.2  3248.8  5102.4           1.6  514.9  56.5   \n",
       "       1                  2.4   979.7   870.8           1.5  498.0  53.0   \n",
       "AR     0                  5.8  1793.7  3401.6           1.9  481.1  37.9   \n",
       "\n",
       "                     SATMTMID               \n",
       "                max_deviation   mean   std  \n",
       "STABBR RELAFFIL                             \n",
       "AK     0                  NaN    NaN   NaN  \n",
       "       1                  NaN  503.0   NaN  \n",
       "AL     0                  1.7  515.8  56.7  \n",
       "       1                  1.4  485.6  61.4  \n",
       "AR     0                  2.0  503.6  39.0  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college.groupby(['STABBR', 'RELAFFIL'])['UGDS', 'SATVRMID', 'SATMTMID']\\\n",
    "       .agg([max_deviation, 'mean', 'std']).round(1).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'max_deviation'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_deviation.__name__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "max_deviation.__name__ = 'Max Deviation'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">UGDS</th>\n",
       "      <th colspan=\"3\" halign=\"left\">SATVRMID</th>\n",
       "      <th colspan=\"3\" halign=\"left\">SATMTMID</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Max Deviation</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>Max Deviation</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>Max Deviation</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>STABBR</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">AK</th>\n",
       "      <th>0</th>\n",
       "      <td>2.1</td>\n",
       "      <td>3508.9</td>\n",
       "      <td>4539.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.1</td>\n",
       "      <td>123.3</td>\n",
       "      <td>132.9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>555.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>503.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">AL</th>\n",
       "      <th>0</th>\n",
       "      <td>5.2</td>\n",
       "      <td>3248.8</td>\n",
       "      <td>5102.4</td>\n",
       "      <td>1.6</td>\n",
       "      <td>514.9</td>\n",
       "      <td>56.5</td>\n",
       "      <td>1.7</td>\n",
       "      <td>515.8</td>\n",
       "      <td>56.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.4</td>\n",
       "      <td>979.7</td>\n",
       "      <td>870.8</td>\n",
       "      <td>1.5</td>\n",
       "      <td>498.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>485.6</td>\n",
       "      <td>61.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AR</th>\n",
       "      <th>0</th>\n",
       "      <td>5.8</td>\n",
       "      <td>1793.7</td>\n",
       "      <td>3401.6</td>\n",
       "      <td>1.9</td>\n",
       "      <td>481.1</td>\n",
       "      <td>37.9</td>\n",
       "      <td>2.0</td>\n",
       "      <td>503.6</td>\n",
       "      <td>39.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         UGDS                      SATVRMID               \\\n",
       "                Max Deviation    mean     std Max Deviation   mean   std   \n",
       "STABBR RELAFFIL                                                            \n",
       "AK     0                  2.1  3508.9  4539.5           NaN    NaN   NaN   \n",
       "       1                  1.1   123.3   132.9           NaN  555.0   NaN   \n",
       "AL     0                  5.2  3248.8  5102.4           1.6  514.9  56.5   \n",
       "       1                  2.4   979.7   870.8           1.5  498.0  53.0   \n",
       "AR     0                  5.8  1793.7  3401.6           1.9  481.1  37.9   \n",
       "\n",
       "                     SATMTMID               \n",
       "                Max Deviation   mean   std  \n",
       "STABBR RELAFFIL                             \n",
       "AK     0                  NaN    NaN   NaN  \n",
       "       1                  NaN  503.0   NaN  \n",
       "AL     0                  1.7  515.8  56.7  \n",
       "       1                  1.4  485.6  61.4  \n",
       "AR     0                  2.0  503.6  39.0  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college.groupby(['STABBR', 'RELAFFIL'])['UGDS', 'SATVRMID', 'SATMTMID']\\\n",
    "       .agg([max_deviation, 'mean', 'std']).round(1).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Customizing aggregating functions with \\*args and \\*\\*kwargs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "college = pd.read_csv('data/college.csv')\n",
    "grouped = college.groupby(['STABBR', 'RELAFFIL'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Signature (arg, *args, **kwargs)>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import inspect\n",
    "inspect.signature(grouped.agg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to do it..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pct_between_1_3k(s):\n",
    "    return s.between(1000, 3000).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "STABBR  RELAFFIL\n",
       "AK      0           0.142857\n",
       "        1           0.000000\n",
       "AL      0           0.236111\n",
       "        1           0.333333\n",
       "AR      0           0.279412\n",
       "        1           0.111111\n",
       "AS      0           1.000000\n",
       "AZ      0           0.096774\n",
       "        1           0.000000\n",
       "Name: UGDS, dtype: float64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college.groupby(['STABBR', 'RELAFFIL'])['UGDS'].agg(pct_between_1_3k).head(9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pct_between(s, low, high):\n",
    "    return s.between(low, high).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "STABBR  RELAFFIL\n",
       "AK      0           0.428571\n",
       "        1           0.000000\n",
       "AL      0           0.458333\n",
       "        1           0.375000\n",
       "AR      0           0.397059\n",
       "        1           0.166667\n",
       "AS      0           1.000000\n",
       "AZ      0           0.233871\n",
       "        1           0.111111\n",
       "Name: UGDS, dtype: float64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college.groupby(['STABBR', 'RELAFFIL'])['UGDS'].agg(pct_between, 1000, 10000).head(9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How it works..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "STABBR  RELAFFIL\n",
       "AK      0           0.428571\n",
       "        1           0.000000\n",
       "AL      0           0.458333\n",
       "        1           0.375000\n",
       "AR      0           0.397059\n",
       "        1           0.166667\n",
       "AS      0           1.000000\n",
       "AZ      0           0.233871\n",
       "        1           0.111111\n",
       "Name: UGDS, dtype: float64"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college.groupby(['STABBR', 'RELAFFIL'])['UGDS'].agg(pct_between, high=10000, low=1000).head(9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "STABBR  RELAFFIL\n",
       "AK      0           0.428571\n",
       "        1           0.000000\n",
       "AL      0           0.458333\n",
       "        1           0.375000\n",
       "AR      0           0.397059\n",
       "        1           0.166667\n",
       "AS      0           1.000000\n",
       "AZ      0           0.233871\n",
       "        1           0.111111\n",
       "Name: UGDS, dtype: float64"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college.groupby(['STABBR', 'RELAFFIL'])['UGDS'].agg(pct_between, 1000, high=10000).head(9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## There's more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "pct_between() missing 2 required positional arguments: 'low' and 'high'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-39-3e3e18919cf9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mcollege\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroupby\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'STABBR'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'RELAFFIL'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'UGDS'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0magg\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'mean'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpct_between\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlow\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhigh\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36maggregate\u001b[0;34m(self, func_or_funcs, *args, **kwargs)\u001b[0m\n\u001b[1;32m   2871\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc_or_funcs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'__iter__'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2872\u001b[0m             ret = self._aggregate_multiple_funcs(func_or_funcs,\n\u001b[0;32m-> 2873\u001b[0;31m                                                  (_level or 0) + 1)\n\u001b[0m\u001b[1;32m   2874\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2875\u001b[0m             \u001b[0mcyfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_cython_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc_or_funcs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m_aggregate_multiple_funcs\u001b[0;34m(self, arg, _level)\u001b[0m\n\u001b[1;32m   2944\u001b[0m                 \u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reset_cache\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2945\u001b[0m                 \u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_selection\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2946\u001b[0;31m             \u001b[0mresults\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maggregate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2947\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2948\u001b[0m         if isinstance(list(compat.itervalues(results))[0],\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36maggregate\u001b[0;34m(self, func_or_funcs, *args, **kwargs)\u001b[0m\n\u001b[1;32m   2878\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2879\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrouper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnkeys\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2880\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_python_agg_general\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc_or_funcs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2881\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2882\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m_python_agg_general\u001b[0;34m(self, func, *args, **kwargs)\u001b[0m\n\u001b[1;32m    852\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    853\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 854\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_python_apply_general\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    855\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    856\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrouper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_filter_empty_groups\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m_python_apply_general\u001b[0;34m(self, f)\u001b[0m\n\u001b[1;32m    718\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_python_apply_general\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    719\u001b[0m         keys, values, mutated = self.grouper.apply(f, self._selected_obj,\n\u001b[0;32m--> 720\u001b[0;31m                                                    self.axis)\n\u001b[0m\u001b[1;32m    721\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    722\u001b[0m         return self._wrap_applied_output(\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36mapply\u001b[0;34m(self, f, data, axis)\u001b[0m\n\u001b[1;32m   1800\u001b[0m             \u001b[0;31m# group might be modified\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1801\u001b[0m             \u001b[0mgroup_axes\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_get_axes\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1802\u001b[0;31m             \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1803\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0m_is_indexed_like\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgroup_axes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1804\u001b[0m                 \u001b[0mmutated\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m    840\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_python_agg_general\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    841\u001b[0m         \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_builtin_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 842\u001b[0;31m         \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    843\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    844\u001b[0m         \u001b[0;31m# iterate through \"columns\" ex exclusions to populate output dict\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: pct_between() missing 2 required positional arguments: 'low' and 'high'"
     ]
    }
   ],
   "source": [
    "college.groupby(['STABBR', 'RELAFFIL'])['UGDS'].agg(['mean', pct_between], low=100, high=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_agg_func(func, name, *args, **kwargs):\n",
    "    def wrapper(x):\n",
    "        return func(x, *args, **kwargs)\n",
    "    wrapper.__name__ = name\n",
    "    return wrapper\n",
    "\n",
    "my_agg1 = make_agg_func(pct_between, 'pct_1_3k', low=1000, high=3000)\n",
    "my_agg2 = make_agg_func(pct_between, 'pct_10_30k', 10000, 30000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>pct_1_3k</th>\n",
       "      <th>pct_10_30k</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>STABBR</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">AK</th>\n",
       "      <th>0</th>\n",
       "      <td>3508.857143</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.142857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>123.333333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">AL</th>\n",
       "      <th>0</th>\n",
       "      <td>3248.774648</td>\n",
       "      <td>0.236111</td>\n",
       "      <td>0.083333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>979.722222</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AR</th>\n",
       "      <th>0</th>\n",
       "      <td>1793.691176</td>\n",
       "      <td>0.279412</td>\n",
       "      <td>0.014706</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        mean  pct_1_3k  pct_10_30k\n",
       "STABBR RELAFFIL                                   \n",
       "AK     0         3508.857143  0.142857    0.142857\n",
       "       1          123.333333  0.000000    0.000000\n",
       "AL     0         3248.774648  0.236111    0.083333\n",
       "       1          979.722222  0.333333    0.000000\n",
       "AR     0         1793.691176  0.279412    0.014706"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college.groupby(['STABBR', 'RELAFFIL'])['UGDS'].agg(['mean', my_agg1, my_agg2]).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examining a groupby object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.groupby.DataFrameGroupBy"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college = pd.read_csv('data/college.csv')\n",
    "grouped = college.groupby(['STABBR', 'RELAFFIL'])\n",
    "type(grouped)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['CITY', 'CURROPER', 'DISTANCEONLY', 'GRAD_DEBT_MDN_SUPP', 'HBCU', 'INSTNM', 'MD_EARN_WNE_P10', 'MENONLY', 'PCTFLOAN', 'PCTPELL', 'PPTUG_EF', 'RELAFFIL', 'SATMTMID', 'SATVRMID', 'STABBR', 'UG25ABV', 'UGDS', 'UGDS_2MOR', 'UGDS_AIAN', 'UGDS_ASIAN', 'UGDS_BLACK', 'UGDS_HISP', 'UGDS_NHPI', 'UGDS_NRA', 'UGDS_UNKN', 'UGDS_WHITE', 'WOMENONLY', 'agg', 'aggregate', 'all', 'any', 'apply', 'backfill', 'bfill', 'boxplot', 'corr', 'corrwith', 'count', 'cov', 'cumcount', 'cummax', 'cummin', 'cumprod', 'cumsum', 'describe', 'diff', 'dtypes', 'expanding', 'ffill', 'fillna', 'filter', 'first', 'get_group', 'groups', 'head', 'hist', 'idxmax', 'idxmin', 'indices', 'last', 'mad', 'max', 'mean', 'median', 'min', 'ndim', 'ngroup', 'ngroups', 'nth', 'nunique', 'ohlc', 'pad', 'pct_change', 'plot', 'prod', 'quantile', 'rank', 'resample', 'rolling', 'sem', 'shift', 'size', 'skew', 'std', 'sum', 'tail', 'take', 'transform', 'tshift', 'var']\n"
     ]
    }
   ],
   "source": [
    "print([attr for attr in dir(grouped) if not attr.startswith('_')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "112"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.ngroups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('AK', 0), ('AK', 1), ('AL', 0), ('AL', 1), ('AR', 0), ('AR', 1)]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "groups = list(grouped.groups.keys())\n",
    "groups[:6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>INSTNM</th>\n",
       "      <th>CITY</th>\n",
       "      <th>STABBR</th>\n",
       "      <th>HBCU</th>\n",
       "      <th>MENONLY</th>\n",
       "      <th>WOMENONLY</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>SATMTMID</th>\n",
       "      <th>DISTANCEONLY</th>\n",
       "      <th>...</th>\n",
       "      <th>UGDS_2MOR</th>\n",
       "      <th>UGDS_NRA</th>\n",
       "      <th>UGDS_UNKN</th>\n",
       "      <th>PPTUG_EF</th>\n",
       "      <th>CURROPER</th>\n",
       "      <th>PCTPELL</th>\n",
       "      <th>PCTFLOAN</th>\n",
       "      <th>UG25ABV</th>\n",
       "      <th>MD_EARN_WNE_P10</th>\n",
       "      <th>GRAD_DEBT_MDN_SUPP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>712</th>\n",
       "      <td>The Baptist College of Florida</td>\n",
       "      <td>Graceville</td>\n",
       "      <td>FL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>545.0</td>\n",
       "      <td>465.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0308</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0507</td>\n",
       "      <td>0.2291</td>\n",
       "      <td>1</td>\n",
       "      <td>0.5878</td>\n",
       "      <td>0.5602</td>\n",
       "      <td>0.3531</td>\n",
       "      <td>30800</td>\n",
       "      <td>20052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>713</th>\n",
       "      <td>Barry University</td>\n",
       "      <td>Miami</td>\n",
       "      <td>FL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>470.0</td>\n",
       "      <td>462.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0164</td>\n",
       "      <td>0.0741</td>\n",
       "      <td>0.0841</td>\n",
       "      <td>0.1518</td>\n",
       "      <td>1</td>\n",
       "      <td>0.5045</td>\n",
       "      <td>0.6733</td>\n",
       "      <td>0.4361</td>\n",
       "      <td>44100</td>\n",
       "      <td>28250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>714</th>\n",
       "      <td>Gooding Institute of Nurse Anesthesia</td>\n",
       "      <td>Panama City</td>\n",
       "      <td>FL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PrivacySuppressed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>715</th>\n",
       "      <td>Bethune-Cookman University</td>\n",
       "      <td>Daytona Beach</td>\n",
       "      <td>FL</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>405.0</td>\n",
       "      <td>395.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0198</td>\n",
       "      <td>0.0205</td>\n",
       "      <td>0.0190</td>\n",
       "      <td>0.0523</td>\n",
       "      <td>1</td>\n",
       "      <td>0.7758</td>\n",
       "      <td>0.8867</td>\n",
       "      <td>0.0647</td>\n",
       "      <td>29400</td>\n",
       "      <td>36250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>724</th>\n",
       "      <td>Johnson University Florida</td>\n",
       "      <td>Kissimmee</td>\n",
       "      <td>FL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>480.0</td>\n",
       "      <td>470.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0045</td>\n",
       "      <td>0.0045</td>\n",
       "      <td>0.0136</td>\n",
       "      <td>0.1636</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6689</td>\n",
       "      <td>0.7384</td>\n",
       "      <td>0.2185</td>\n",
       "      <td>26300</td>\n",
       "      <td>20199</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    INSTNM           CITY STABBR  HBCU  \\\n",
       "712         The Baptist College of Florida     Graceville     FL   0.0   \n",
       "713                       Barry University          Miami     FL   0.0   \n",
       "714  Gooding Institute of Nurse Anesthesia    Panama City     FL   0.0   \n",
       "715             Bethune-Cookman University  Daytona Beach     FL   1.0   \n",
       "724             Johnson University Florida      Kissimmee     FL   0.0   \n",
       "\n",
       "     MENONLY  WOMENONLY  RELAFFIL  SATVRMID  SATMTMID  DISTANCEONLY  \\\n",
       "712      0.0        0.0         1     545.0     465.0           0.0   \n",
       "713      0.0        0.0         1     470.0     462.0           0.0   \n",
       "714      0.0        0.0         1       NaN       NaN           0.0   \n",
       "715      0.0        0.0         1     405.0     395.0           0.0   \n",
       "724      0.0        0.0         1     480.0     470.0           0.0   \n",
       "\n",
       "            ...          UGDS_2MOR  UGDS_NRA  UGDS_UNKN  PPTUG_EF  CURROPER  \\\n",
       "712         ...             0.0308    0.0000     0.0507    0.2291         1   \n",
       "713         ...             0.0164    0.0741     0.0841    0.1518         1   \n",
       "714         ...                NaN       NaN        NaN       NaN         0   \n",
       "715         ...             0.0198    0.0205     0.0190    0.0523         1   \n",
       "724         ...             0.0045    0.0045     0.0136    0.1636         1   \n",
       "\n",
       "     PCTPELL  PCTFLOAN  UG25ABV  MD_EARN_WNE_P10  GRAD_DEBT_MDN_SUPP  \n",
       "712   0.5878    0.5602   0.3531            30800               20052  \n",
       "713   0.5045    0.6733   0.4361            44100               28250  \n",
       "714      NaN       NaN      NaN              NaN   PrivacySuppressed  \n",
       "715   0.7758    0.8867   0.0647            29400               36250  \n",
       "724   0.6689    0.7384   0.2185            26300               20199  \n",
       "\n",
       "[5 rows x 27 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.get_group(('FL', 1)).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('AK', 0)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>INSTNM</th>\n",
       "      <th>CITY</th>\n",
       "      <th>STABBR</th>\n",
       "      <th>HBCU</th>\n",
       "      <th>MENONLY</th>\n",
       "      <th>WOMENONLY</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>SATMTMID</th>\n",
       "      <th>DISTANCEONLY</th>\n",
       "      <th>...</th>\n",
       "      <th>UGDS_2MOR</th>\n",
       "      <th>UGDS_NRA</th>\n",
       "      <th>UGDS_UNKN</th>\n",
       "      <th>PPTUG_EF</th>\n",
       "      <th>CURROPER</th>\n",
       "      <th>PCTPELL</th>\n",
       "      <th>PCTFLOAN</th>\n",
       "      <th>UG25ABV</th>\n",
       "      <th>MD_EARN_WNE_P10</th>\n",
       "      <th>GRAD_DEBT_MDN_SUPP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>University of Alaska Anchorage</td>\n",
       "      <td>Anchorage</td>\n",
       "      <td>AK</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0980</td>\n",
       "      <td>0.0181</td>\n",
       "      <td>0.0457</td>\n",
       "      <td>0.4539</td>\n",
       "      <td>1</td>\n",
       "      <td>0.2385</td>\n",
       "      <td>0.2647</td>\n",
       "      <td>0.4386</td>\n",
       "      <td>42500</td>\n",
       "      <td>19449.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>University of Alaska Fairbanks</td>\n",
       "      <td>Fairbanks</td>\n",
       "      <td>AK</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0401</td>\n",
       "      <td>0.0110</td>\n",
       "      <td>0.3060</td>\n",
       "      <td>0.3887</td>\n",
       "      <td>1</td>\n",
       "      <td>0.2263</td>\n",
       "      <td>0.2550</td>\n",
       "      <td>0.4519</td>\n",
       "      <td>36200</td>\n",
       "      <td>19355</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                            INSTNM       CITY STABBR  HBCU  MENONLY  \\\n",
       "60  University of Alaska Anchorage  Anchorage     AK   0.0      0.0   \n",
       "62  University of Alaska Fairbanks  Fairbanks     AK   0.0      0.0   \n",
       "\n",
       "    WOMENONLY  RELAFFIL  SATVRMID  SATMTMID  DISTANCEONLY         ...          \\\n",
       "60        0.0         0       NaN       NaN           0.0         ...           \n",
       "62        0.0         0       NaN       NaN           0.0         ...           \n",
       "\n",
       "    UGDS_2MOR  UGDS_NRA  UGDS_UNKN  PPTUG_EF  CURROPER  PCTPELL  PCTFLOAN  \\\n",
       "60     0.0980    0.0181     0.0457    0.4539         1   0.2385    0.2647   \n",
       "62     0.0401    0.0110     0.3060    0.3887         1   0.2263    0.2550   \n",
       "\n",
       "    UG25ABV  MD_EARN_WNE_P10  GRAD_DEBT_MDN_SUPP  \n",
       "60   0.4386            42500             19449.5  \n",
       "62   0.4519            36200               19355  \n",
       "\n",
       "[2 rows x 27 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('AK', 1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>INSTNM</th>\n",
       "      <th>CITY</th>\n",
       "      <th>STABBR</th>\n",
       "      <th>HBCU</th>\n",
       "      <th>MENONLY</th>\n",
       "      <th>WOMENONLY</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>SATMTMID</th>\n",
       "      <th>DISTANCEONLY</th>\n",
       "      <th>...</th>\n",
       "      <th>UGDS_2MOR</th>\n",
       "      <th>UGDS_NRA</th>\n",
       "      <th>UGDS_UNKN</th>\n",
       "      <th>PPTUG_EF</th>\n",
       "      <th>CURROPER</th>\n",
       "      <th>PCTPELL</th>\n",
       "      <th>PCTFLOAN</th>\n",
       "      <th>UG25ABV</th>\n",
       "      <th>MD_EARN_WNE_P10</th>\n",
       "      <th>GRAD_DEBT_MDN_SUPP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>Alaska Bible College</td>\n",
       "      <td>Palmer</td>\n",
       "      <td>AK</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0370</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1481</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3571</td>\n",
       "      <td>0.2857</td>\n",
       "      <td>0.4286</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PrivacySuppressed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>Alaska Pacific University</td>\n",
       "      <td>Anchorage</td>\n",
       "      <td>AK</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>555.0</td>\n",
       "      <td>503.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0945</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0873</td>\n",
       "      <td>0.3745</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3152</td>\n",
       "      <td>0.5297</td>\n",
       "      <td>0.4910</td>\n",
       "      <td>47000</td>\n",
       "      <td>23250</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                       INSTNM       CITY STABBR  HBCU  MENONLY  WOMENONLY  \\\n",
       "61       Alaska Bible College     Palmer     AK   0.0      0.0        0.0   \n",
       "64  Alaska Pacific University  Anchorage     AK   0.0      0.0        0.0   \n",
       "\n",
       "    RELAFFIL  SATVRMID  SATMTMID  DISTANCEONLY         ...          UGDS_2MOR  \\\n",
       "61         1       NaN       NaN           0.0         ...             0.0370   \n",
       "64         1     555.0     503.0           0.0         ...             0.0945   \n",
       "\n",
       "    UGDS_NRA  UGDS_UNKN  PPTUG_EF  CURROPER  PCTPELL  PCTFLOAN  UG25ABV  \\\n",
       "61       0.0     0.0000    0.1481         1   0.3571    0.2857   0.4286   \n",
       "64       0.0     0.0873    0.3745         1   0.3152    0.5297   0.4910   \n",
       "\n",
       "    MD_EARN_WNE_P10  GRAD_DEBT_MDN_SUPP  \n",
       "61              NaN   PrivacySuppressed  \n",
       "64            47000               23250  \n",
       "\n",
       "[2 rows x 27 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('AL', 0)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>INSTNM</th>\n",
       "      <th>CITY</th>\n",
       "      <th>STABBR</th>\n",
       "      <th>HBCU</th>\n",
       "      <th>MENONLY</th>\n",
       "      <th>WOMENONLY</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>SATMTMID</th>\n",
       "      <th>DISTANCEONLY</th>\n",
       "      <th>...</th>\n",
       "      <th>UGDS_2MOR</th>\n",
       "      <th>UGDS_NRA</th>\n",
       "      <th>UGDS_UNKN</th>\n",
       "      <th>PPTUG_EF</th>\n",
       "      <th>CURROPER</th>\n",
       "      <th>PCTPELL</th>\n",
       "      <th>PCTFLOAN</th>\n",
       "      <th>UG25ABV</th>\n",
       "      <th>MD_EARN_WNE_P10</th>\n",
       "      <th>GRAD_DEBT_MDN_SUPP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alabama A &amp; M University</td>\n",
       "      <td>Normal</td>\n",
       "      <td>AL</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>424.0</td>\n",
       "      <td>420.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0059</td>\n",
       "      <td>0.0138</td>\n",
       "      <td>0.0656</td>\n",
       "      <td>1</td>\n",
       "      <td>0.7356</td>\n",
       "      <td>0.8284</td>\n",
       "      <td>0.1049</td>\n",
       "      <td>30300</td>\n",
       "      <td>33888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>University of Alabama at Birmingham</td>\n",
       "      <td>Birmingham</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>570.0</td>\n",
       "      <td>565.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0368</td>\n",
       "      <td>0.0179</td>\n",
       "      <td>0.0100</td>\n",
       "      <td>0.2607</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3460</td>\n",
       "      <td>0.5214</td>\n",
       "      <td>0.2422</td>\n",
       "      <td>39700</td>\n",
       "      <td>21941.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                INSTNM        CITY STABBR  HBCU  MENONLY  \\\n",
       "0             Alabama A & M University      Normal     AL   1.0      0.0   \n",
       "1  University of Alabama at Birmingham  Birmingham     AL   0.0      0.0   \n",
       "\n",
       "   WOMENONLY  RELAFFIL  SATVRMID  SATMTMID  DISTANCEONLY         ...          \\\n",
       "0        0.0         0     424.0     420.0           0.0         ...           \n",
       "1        0.0         0     570.0     565.0           0.0         ...           \n",
       "\n",
       "   UGDS_2MOR  UGDS_NRA  UGDS_UNKN  PPTUG_EF  CURROPER  PCTPELL  PCTFLOAN  \\\n",
       "0     0.0000    0.0059     0.0138    0.0656         1   0.7356    0.8284   \n",
       "1     0.0368    0.0179     0.0100    0.2607         1   0.3460    0.5214   \n",
       "\n",
       "   UG25ABV  MD_EARN_WNE_P10  GRAD_DEBT_MDN_SUPP  \n",
       "0   0.1049            30300               33888  \n",
       "1   0.2422            39700             21941.5  \n",
       "\n",
       "[2 rows x 27 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('AL', 1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>INSTNM</th>\n",
       "      <th>CITY</th>\n",
       "      <th>STABBR</th>\n",
       "      <th>HBCU</th>\n",
       "      <th>MENONLY</th>\n",
       "      <th>WOMENONLY</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>SATMTMID</th>\n",
       "      <th>DISTANCEONLY</th>\n",
       "      <th>...</th>\n",
       "      <th>UGDS_2MOR</th>\n",
       "      <th>UGDS_NRA</th>\n",
       "      <th>UGDS_UNKN</th>\n",
       "      <th>PPTUG_EF</th>\n",
       "      <th>CURROPER</th>\n",
       "      <th>PCTPELL</th>\n",
       "      <th>PCTFLOAN</th>\n",
       "      <th>UG25ABV</th>\n",
       "      <th>MD_EARN_WNE_P10</th>\n",
       "      <th>GRAD_DEBT_MDN_SUPP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Amridge University</td>\n",
       "      <td>Montgomery</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2715</td>\n",
       "      <td>0.4536</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6801</td>\n",
       "      <td>0.7795</td>\n",
       "      <td>0.8540</td>\n",
       "      <td>40100</td>\n",
       "      <td>23370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Birmingham Southern College</td>\n",
       "      <td>Birmingham</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>560.0</td>\n",
       "      <td>560.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0051</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0051</td>\n",
       "      <td>0.0017</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1920</td>\n",
       "      <td>0.4809</td>\n",
       "      <td>0.0152</td>\n",
       "      <td>44200</td>\n",
       "      <td>27000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                         INSTNM        CITY STABBR  HBCU  MENONLY  WOMENONLY  \\\n",
       "2            Amridge University  Montgomery     AL   0.0      0.0        0.0   \n",
       "10  Birmingham Southern College  Birmingham     AL   0.0      0.0        0.0   \n",
       "\n",
       "    RELAFFIL  SATVRMID  SATMTMID  DISTANCEONLY         ...          UGDS_2MOR  \\\n",
       "2          1       NaN       NaN           1.0         ...             0.0000   \n",
       "10         1     560.0     560.0           0.0         ...             0.0051   \n",
       "\n",
       "    UGDS_NRA  UGDS_UNKN  PPTUG_EF  CURROPER  PCTPELL  PCTFLOAN  UG25ABV  \\\n",
       "2        0.0     0.2715    0.4536         1   0.6801    0.7795   0.8540   \n",
       "10       0.0     0.0051    0.0017         1   0.1920    0.4809   0.0152   \n",
       "\n",
       "    MD_EARN_WNE_P10  GRAD_DEBT_MDN_SUPP  \n",
       "2             40100               23370  \n",
       "10            44200               27000  \n",
       "\n",
       "[2 rows x 27 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('AR', 0)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>INSTNM</th>\n",
       "      <th>CITY</th>\n",
       "      <th>STABBR</th>\n",
       "      <th>HBCU</th>\n",
       "      <th>MENONLY</th>\n",
       "      <th>WOMENONLY</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>SATMTMID</th>\n",
       "      <th>DISTANCEONLY</th>\n",
       "      <th>...</th>\n",
       "      <th>UGDS_2MOR</th>\n",
       "      <th>UGDS_NRA</th>\n",
       "      <th>UGDS_UNKN</th>\n",
       "      <th>PPTUG_EF</th>\n",
       "      <th>CURROPER</th>\n",
       "      <th>PCTPELL</th>\n",
       "      <th>PCTFLOAN</th>\n",
       "      <th>UG25ABV</th>\n",
       "      <th>MD_EARN_WNE_P10</th>\n",
       "      <th>GRAD_DEBT_MDN_SUPP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>University of Arkansas at Little Rock</td>\n",
       "      <td>Little Rock</td>\n",
       "      <td>AR</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>470.0</td>\n",
       "      <td>510.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0755</td>\n",
       "      <td>0.0283</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>0.4126</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3941</td>\n",
       "      <td>0.4775</td>\n",
       "      <td>0.4062</td>\n",
       "      <td>33900</td>\n",
       "      <td>21736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>University of Arkansas for Medical Sciences</td>\n",
       "      <td>Little Rock</td>\n",
       "      <td>AR</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0281</td>\n",
       "      <td>0.0070</td>\n",
       "      <td>0.0169</td>\n",
       "      <td>0.2433</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3944</td>\n",
       "      <td>0.6144</td>\n",
       "      <td>0.5133</td>\n",
       "      <td>61400</td>\n",
       "      <td>12500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          INSTNM         CITY STABBR  HBCU  \\\n",
       "128        University of Arkansas at Little Rock  Little Rock     AR   0.0   \n",
       "129  University of Arkansas for Medical Sciences  Little Rock     AR   0.0   \n",
       "\n",
       "     MENONLY  WOMENONLY  RELAFFIL  SATVRMID  SATMTMID  DISTANCEONLY  \\\n",
       "128      0.0        0.0         0     470.0     510.0           0.0   \n",
       "129      0.0        0.0         0       NaN       NaN           0.0   \n",
       "\n",
       "            ...          UGDS_2MOR  UGDS_NRA  UGDS_UNKN  PPTUG_EF  CURROPER  \\\n",
       "128         ...             0.0755    0.0283     0.0003    0.4126         1   \n",
       "129         ...             0.0281    0.0070     0.0169    0.2433         1   \n",
       "\n",
       "     PCTPELL  PCTFLOAN  UG25ABV  MD_EARN_WNE_P10  GRAD_DEBT_MDN_SUPP  \n",
       "128   0.3941    0.4775   0.4062            33900               21736  \n",
       "129   0.3944    0.6144   0.5133            61400               12500  \n",
       "\n",
       "[2 rows x 27 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "i = 0\n",
    "for name, group in grouped:\n",
    "    print(name)\n",
    "    display(group.head(2))\n",
    "    i += 1\n",
    "    if i == 5:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>INSTNM</th>\n",
       "      <th>CITY</th>\n",
       "      <th>STABBR</th>\n",
       "      <th>HBCU</th>\n",
       "      <th>MENONLY</th>\n",
       "      <th>WOMENONLY</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>SATMTMID</th>\n",
       "      <th>DISTANCEONLY</th>\n",
       "      <th>...</th>\n",
       "      <th>UGDS_2MOR</th>\n",
       "      <th>UGDS_NRA</th>\n",
       "      <th>UGDS_UNKN</th>\n",
       "      <th>PPTUG_EF</th>\n",
       "      <th>CURROPER</th>\n",
       "      <th>PCTPELL</th>\n",
       "      <th>PCTFLOAN</th>\n",
       "      <th>UG25ABV</th>\n",
       "      <th>MD_EARN_WNE_P10</th>\n",
       "      <th>GRAD_DEBT_MDN_SUPP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alabama A &amp; M University</td>\n",
       "      <td>Normal</td>\n",
       "      <td>AL</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>424.0</td>\n",
       "      <td>420.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0059</td>\n",
       "      <td>0.0138</td>\n",
       "      <td>0.0656</td>\n",
       "      <td>1</td>\n",
       "      <td>0.7356</td>\n",
       "      <td>0.8284</td>\n",
       "      <td>0.1049</td>\n",
       "      <td>30300</td>\n",
       "      <td>33888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>University of Alabama at Birmingham</td>\n",
       "      <td>Birmingham</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>570.0</td>\n",
       "      <td>565.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0368</td>\n",
       "      <td>0.0179</td>\n",
       "      <td>0.0100</td>\n",
       "      <td>0.2607</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3460</td>\n",
       "      <td>0.5214</td>\n",
       "      <td>0.2422</td>\n",
       "      <td>39700</td>\n",
       "      <td>21941.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Amridge University</td>\n",
       "      <td>Montgomery</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.2715</td>\n",
       "      <td>0.4536</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6801</td>\n",
       "      <td>0.7795</td>\n",
       "      <td>0.8540</td>\n",
       "      <td>40100</td>\n",
       "      <td>23370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Birmingham Southern College</td>\n",
       "      <td>Birmingham</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>560.0</td>\n",
       "      <td>560.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0051</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0051</td>\n",
       "      <td>0.0017</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1920</td>\n",
       "      <td>0.4809</td>\n",
       "      <td>0.0152</td>\n",
       "      <td>44200</td>\n",
       "      <td>27000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Prince Institute-Southeast</td>\n",
       "      <td>Elmhurst</td>\n",
       "      <td>IL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>0.7857</td>\n",
       "      <td>0.9375</td>\n",
       "      <td>0.6569</td>\n",
       "      <td>PrivacySuppressed</td>\n",
       "      <td>20992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>University of Alaska Anchorage</td>\n",
       "      <td>Anchorage</td>\n",
       "      <td>AK</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0980</td>\n",
       "      <td>0.0181</td>\n",
       "      <td>0.0457</td>\n",
       "      <td>0.4539</td>\n",
       "      <td>1</td>\n",
       "      <td>0.2385</td>\n",
       "      <td>0.2647</td>\n",
       "      <td>0.4386</td>\n",
       "      <td>42500</td>\n",
       "      <td>19449.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 INSTNM        CITY STABBR  HBCU  MENONLY  \\\n",
       "0              Alabama A & M University      Normal     AL   1.0      0.0   \n",
       "1   University of Alabama at Birmingham  Birmingham     AL   0.0      0.0   \n",
       "2                    Amridge University  Montgomery     AL   0.0      0.0   \n",
       "10          Birmingham Southern College  Birmingham     AL   0.0      0.0   \n",
       "43           Prince Institute-Southeast    Elmhurst     IL   0.0      0.0   \n",
       "60       University of Alaska Anchorage   Anchorage     AK   0.0      0.0   \n",
       "\n",
       "    WOMENONLY  RELAFFIL  SATVRMID  SATMTMID  DISTANCEONLY         ...          \\\n",
       "0         0.0         0     424.0     420.0           0.0         ...           \n",
       "1         0.0         0     570.0     565.0           0.0         ...           \n",
       "2         0.0         1       NaN       NaN           1.0         ...           \n",
       "10        0.0         1     560.0     560.0           0.0         ...           \n",
       "43        0.0         0       NaN       NaN           0.0         ...           \n",
       "60        0.0         0       NaN       NaN           0.0         ...           \n",
       "\n",
       "    UGDS_2MOR  UGDS_NRA  UGDS_UNKN  PPTUG_EF  CURROPER  PCTPELL  PCTFLOAN  \\\n",
       "0      0.0000    0.0059     0.0138    0.0656         1   0.7356    0.8284   \n",
       "1      0.0368    0.0179     0.0100    0.2607         1   0.3460    0.5214   \n",
       "2      0.0000    0.0000     0.2715    0.4536         1   0.6801    0.7795   \n",
       "10     0.0051    0.0000     0.0051    0.0017         1   0.1920    0.4809   \n",
       "43     0.0000    0.0000     0.0000    0.0000         1   0.7857    0.9375   \n",
       "60     0.0980    0.0181     0.0457    0.4539         1   0.2385    0.2647   \n",
       "\n",
       "    UG25ABV    MD_EARN_WNE_P10  GRAD_DEBT_MDN_SUPP  \n",
       "0    0.1049              30300               33888  \n",
       "1    0.2422              39700             21941.5  \n",
       "2    0.8540              40100               23370  \n",
       "10   0.0152              44200               27000  \n",
       "43   0.6569  PrivacySuppressed               20992  \n",
       "60   0.4386              42500             19449.5  \n",
       "\n",
       "[6 rows x 27 columns]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.head(2).head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## There's more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>CITY</th>\n",
       "      <th>CURROPER</th>\n",
       "      <th>DISTANCEONLY</th>\n",
       "      <th>GRAD_DEBT_MDN_SUPP</th>\n",
       "      <th>HBCU</th>\n",
       "      <th>INSTNM</th>\n",
       "      <th>MD_EARN_WNE_P10</th>\n",
       "      <th>MENONLY</th>\n",
       "      <th>PCTFLOAN</th>\n",
       "      <th>PCTPELL</th>\n",
       "      <th>...</th>\n",
       "      <th>UGDS_2MOR</th>\n",
       "      <th>UGDS_AIAN</th>\n",
       "      <th>UGDS_ASIAN</th>\n",
       "      <th>UGDS_BLACK</th>\n",
       "      <th>UGDS_HISP</th>\n",
       "      <th>UGDS_NHPI</th>\n",
       "      <th>UGDS_NRA</th>\n",
       "      <th>UGDS_UNKN</th>\n",
       "      <th>UGDS_WHITE</th>\n",
       "      <th>WOMENONLY</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>STABBR</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">AK</th>\n",
       "      <th>0</th>\n",
       "      <td>Fairbanks</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19355</td>\n",
       "      <td>0.0</td>\n",
       "      <td>University of Alaska Fairbanks</td>\n",
       "      <td>36200</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2550</td>\n",
       "      <td>0.2263</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0401</td>\n",
       "      <td>0.1284</td>\n",
       "      <td>0.0126</td>\n",
       "      <td>0.0210</td>\n",
       "      <td>0.0522</td>\n",
       "      <td>0.0027</td>\n",
       "      <td>0.0110</td>\n",
       "      <td>0.3060</td>\n",
       "      <td>0.4259</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Barrow</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>PrivacySuppressed</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Ilisagvik College</td>\n",
       "      <td>24900</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1323</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.6881</td>\n",
       "      <td>0.0826</td>\n",
       "      <td>0.0183</td>\n",
       "      <td>0.0092</td>\n",
       "      <td>0.0459</td>\n",
       "      <td>0.0183</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1376</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Anchorage</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>23250</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Alaska Pacific University</td>\n",
       "      <td>47000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5297</td>\n",
       "      <td>0.3152</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0945</td>\n",
       "      <td>0.1855</td>\n",
       "      <td>0.0255</td>\n",
       "      <td>0.0291</td>\n",
       "      <td>0.0364</td>\n",
       "      <td>0.0109</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0873</td>\n",
       "      <td>0.5309</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Soldotna</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>PrivacySuppressed</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Alaska Christian College</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6792</td>\n",
       "      <td>0.8868</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0147</td>\n",
       "      <td>0.7794</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0147</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1324</td>\n",
       "      <td>0.0588</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">AL</th>\n",
       "      <th>0</th>\n",
       "      <td>Birmingham</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21941.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>University of Alabama at Birmingham</td>\n",
       "      <td>39700</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5214</td>\n",
       "      <td>0.3460</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0368</td>\n",
       "      <td>0.0022</td>\n",
       "      <td>0.0518</td>\n",
       "      <td>0.2600</td>\n",
       "      <td>0.0283</td>\n",
       "      <td>0.0007</td>\n",
       "      <td>0.0179</td>\n",
       "      <td>0.0100</td>\n",
       "      <td>0.5922</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Dothan</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>PrivacySuppressed</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Alabama College of Osteopathic Medicine</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Birmingham</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>27000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Birmingham Southern College</td>\n",
       "      <td>44200</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4809</td>\n",
       "      <td>0.1920</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0051</td>\n",
       "      <td>0.0102</td>\n",
       "      <td>0.0517</td>\n",
       "      <td>0.1102</td>\n",
       "      <td>0.0195</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0051</td>\n",
       "      <td>0.7983</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Huntsville</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>36173.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Strayer University-Huntsville Campus</td>\n",
       "      <td>49200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                       CITY  CURROPER  DISTANCEONLY GRAD_DEBT_MDN_SUPP  HBCU  \\\n",
       "STABBR RELAFFIL                                                                \n",
       "AK     0          Fairbanks         1           0.0              19355   0.0   \n",
       "       0             Barrow         1           0.0  PrivacySuppressed   0.0   \n",
       "       1          Anchorage         1           0.0              23250   0.0   \n",
       "       1           Soldotna         1           0.0  PrivacySuppressed   0.0   \n",
       "AL     0         Birmingham         1           0.0            21941.5   0.0   \n",
       "       0             Dothan         1           0.0  PrivacySuppressed   0.0   \n",
       "       1         Birmingham         1           0.0              27000   0.0   \n",
       "       1         Huntsville         1           NaN            36173.5   NaN   \n",
       "\n",
       "                                                  INSTNM MD_EARN_WNE_P10  \\\n",
       "STABBR RELAFFIL                                                            \n",
       "AK     0                  University of Alaska Fairbanks           36200   \n",
       "       0                               Ilisagvik College           24900   \n",
       "       1                       Alaska Pacific University           47000   \n",
       "       1                        Alaska Christian College             NaN   \n",
       "AL     0             University of Alabama at Birmingham           39700   \n",
       "       0         Alabama College of Osteopathic Medicine             NaN   \n",
       "       1                     Birmingham Southern College           44200   \n",
       "       1            Strayer University-Huntsville Campus           49200   \n",
       "\n",
       "                 MENONLY  PCTFLOAN  PCTPELL    ...      UGDS_2MOR  UGDS_AIAN  \\\n",
       "STABBR RELAFFIL                                ...                             \n",
       "AK     0             0.0    0.2550   0.2263    ...         0.0401     0.1284   \n",
       "       0             0.0    0.0000   0.1323    ...         0.0000     0.6881   \n",
       "       1             0.0    0.5297   0.3152    ...         0.0945     0.1855   \n",
       "       1             0.0    0.6792   0.8868    ...         0.0147     0.7794   \n",
       "AL     0             0.0    0.5214   0.3460    ...         0.0368     0.0022   \n",
       "       0             0.0       NaN      NaN    ...            NaN        NaN   \n",
       "       1             0.0    0.4809   0.1920    ...         0.0051     0.0102   \n",
       "       1             NaN       NaN      NaN    ...            NaN        NaN   \n",
       "\n",
       "                 UGDS_ASIAN  UGDS_BLACK  UGDS_HISP  UGDS_NHPI  UGDS_NRA  \\\n",
       "STABBR RELAFFIL                                                           \n",
       "AK     0             0.0126      0.0210     0.0522     0.0027    0.0110   \n",
       "       0             0.0826      0.0183     0.0092     0.0459    0.0183   \n",
       "       1             0.0255      0.0291     0.0364     0.0109    0.0000   \n",
       "       1             0.0000      0.0000     0.0147     0.0000    0.0000   \n",
       "AL     0             0.0518      0.2600     0.0283     0.0007    0.0179   \n",
       "       0                NaN         NaN        NaN        NaN       NaN   \n",
       "       1             0.0517      0.1102     0.0195     0.0000    0.0000   \n",
       "       1                NaN         NaN        NaN        NaN       NaN   \n",
       "\n",
       "                 UGDS_UNKN  UGDS_WHITE  WOMENONLY  \n",
       "STABBR RELAFFIL                                    \n",
       "AK     0            0.3060      0.4259        0.0  \n",
       "       0            0.0000      0.1376        0.0  \n",
       "       1            0.0873      0.5309        0.0  \n",
       "       1            0.1324      0.0588        0.0  \n",
       "AL     0            0.0100      0.5922        0.0  \n",
       "       0               NaN         NaN        0.0  \n",
       "       1            0.0051      0.7983        0.0  \n",
       "       1               NaN         NaN        NaN  \n",
       "\n",
       "[8 rows x 25 columns]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.nth([1, -1]).head(8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Filtering for states with a minority majority"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "59"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college = pd.read_csv('data/college.csv', index_col='INSTNM')\n",
    "grouped = college.groupby('STABBR')\n",
    "grouped.ngroups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "285 µs ± 11.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit college['STABBR'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def check_minority(df, threshold):\n",
    "    minority_pct = 1 - df['UGDS_WHITE']\n",
    "    total_minority = (df['UGDS'] * minority_pct).sum()\n",
    "    total_ugds = df['UGDS'].sum()\n",
    "    total_minority_pct = total_minority / total_ugds\n",
    "    return total_minority_pct > threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CITY</th>\n",
       "      <th>STABBR</th>\n",
       "      <th>HBCU</th>\n",
       "      <th>MENONLY</th>\n",
       "      <th>WOMENONLY</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>SATMTMID</th>\n",
       "      <th>DISTANCEONLY</th>\n",
       "      <th>UGDS</th>\n",
       "      <th>...</th>\n",
       "      <th>UGDS_2MOR</th>\n",
       "      <th>UGDS_NRA</th>\n",
       "      <th>UGDS_UNKN</th>\n",
       "      <th>PPTUG_EF</th>\n",
       "      <th>CURROPER</th>\n",
       "      <th>PCTPELL</th>\n",
       "      <th>PCTFLOAN</th>\n",
       "      <th>UG25ABV</th>\n",
       "      <th>MD_EARN_WNE_P10</th>\n",
       "      <th>GRAD_DEBT_MDN_SUPP</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>INSTNM</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Everest College-Phoenix</th>\n",
       "      <td>Phoenix</td>\n",
       "      <td>AZ</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4102.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0373</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1026</td>\n",
       "      <td>0.4749</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8291</td>\n",
       "      <td>0.7151</td>\n",
       "      <td>0.6700</td>\n",
       "      <td>28600</td>\n",
       "      <td>9500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Collins College</th>\n",
       "      <td>Phoenix</td>\n",
       "      <td>AZ</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>83.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0241</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3855</td>\n",
       "      <td>0.3373</td>\n",
       "      <td>0</td>\n",
       "      <td>0.7205</td>\n",
       "      <td>0.8228</td>\n",
       "      <td>0.4764</td>\n",
       "      <td>25700</td>\n",
       "      <td>47000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Empire Beauty School-Paradise Valley</th>\n",
       "      <td>Phoenix</td>\n",
       "      <td>AZ</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0400</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1600</td>\n",
       "      <td>0</td>\n",
       "      <td>0.6349</td>\n",
       "      <td>0.5873</td>\n",
       "      <td>0.4651</td>\n",
       "      <td>17800</td>\n",
       "      <td>9588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Empire Beauty School-Tucson</th>\n",
       "      <td>Tucson</td>\n",
       "      <td>AZ</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0079</td>\n",
       "      <td>0.2222</td>\n",
       "      <td>1</td>\n",
       "      <td>0.7962</td>\n",
       "      <td>0.6615</td>\n",
       "      <td>0.4229</td>\n",
       "      <td>18200</td>\n",
       "      <td>9833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thunderbird School of Global Management</th>\n",
       "      <td>Glendale</td>\n",
       "      <td>AZ</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>118900</td>\n",
       "      <td>PrivacySuppressed</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             CITY STABBR  HBCU  MENONLY  \\\n",
       "INSTNM                                                                    \n",
       "Everest College-Phoenix                   Phoenix     AZ   0.0      0.0   \n",
       "Collins College                           Phoenix     AZ   0.0      0.0   \n",
       "Empire Beauty School-Paradise Valley      Phoenix     AZ   0.0      0.0   \n",
       "Empire Beauty School-Tucson                Tucson     AZ   0.0      0.0   \n",
       "Thunderbird School of Global Management  Glendale     AZ   0.0      0.0   \n",
       "\n",
       "                                         WOMENONLY  RELAFFIL  SATVRMID  \\\n",
       "INSTNM                                                                   \n",
       "Everest College-Phoenix                        0.0         1       NaN   \n",
       "Collins College                                0.0         0       NaN   \n",
       "Empire Beauty School-Paradise Valley           0.0         1       NaN   \n",
       "Empire Beauty School-Tucson                    0.0         0       NaN   \n",
       "Thunderbird School of Global Management        0.0         0       NaN   \n",
       "\n",
       "                                         SATMTMID  DISTANCEONLY    UGDS  \\\n",
       "INSTNM                                                                    \n",
       "Everest College-Phoenix                       NaN           0.0  4102.0   \n",
       "Collins College                               NaN           0.0    83.0   \n",
       "Empire Beauty School-Paradise Valley          NaN           0.0    25.0   \n",
       "Empire Beauty School-Tucson                   NaN           0.0   126.0   \n",
       "Thunderbird School of Global Management       NaN           0.0     1.0   \n",
       "\n",
       "                                                ...          UGDS_2MOR  \\\n",
       "INSTNM                                          ...                      \n",
       "Everest College-Phoenix                         ...             0.0373   \n",
       "Collins College                                 ...             0.0241   \n",
       "Empire Beauty School-Paradise Valley            ...             0.0400   \n",
       "Empire Beauty School-Tucson                     ...             0.0000   \n",
       "Thunderbird School of Global Management         ...             0.0000   \n",
       "\n",
       "                                         UGDS_NRA  UGDS_UNKN  PPTUG_EF  \\\n",
       "INSTNM                                                                   \n",
       "Everest College-Phoenix                       0.0     0.1026    0.4749   \n",
       "Collins College                               0.0     0.3855    0.3373   \n",
       "Empire Beauty School-Paradise Valley          0.0     0.0000    0.1600   \n",
       "Empire Beauty School-Tucson                   0.0     0.0079    0.2222   \n",
       "Thunderbird School of Global Management       0.0     0.0000    1.0000   \n",
       "\n",
       "                                         CURROPER  PCTPELL  PCTFLOAN  UG25ABV  \\\n",
       "INSTNM                                                                          \n",
       "Everest College-Phoenix                         0   0.8291    0.7151   0.6700   \n",
       "Collins College                                 0   0.7205    0.8228   0.4764   \n",
       "Empire Beauty School-Paradise Valley            0   0.6349    0.5873   0.4651   \n",
       "Empire Beauty School-Tucson                     1   0.7962    0.6615   0.4229   \n",
       "Thunderbird School of Global Management         0   0.0000    0.0000   0.0000   \n",
       "\n",
       "                                         MD_EARN_WNE_P10  GRAD_DEBT_MDN_SUPP  \n",
       "INSTNM                                                                        \n",
       "Everest College-Phoenix                            28600                9500  \n",
       "Collins College                                    25700               47000  \n",
       "Empire Beauty School-Paradise Valley               17800                9588  \n",
       "Empire Beauty School-Tucson                        18200                9833  \n",
       "Thunderbird School of Global Management           118900   PrivacySuppressed  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college_filtered = grouped.filter(check_minority, threshold=.5)\n",
    "college_filtered.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7535, 26)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3028, 26)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college_filtered.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college_filtered['STABBR'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## There's more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7461, 26)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college_filtered_20 = grouped.filter(check_minority, threshold=.2)\n",
    "college_filtered_20.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "57"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college_filtered_20['STABBR'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(957, 26)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college_filtered_70 = grouped.filter(check_minority, threshold=.7)\n",
    "college_filtered_70.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college_filtered_70['STABBR'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(156, 26)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college_filtered_95 = grouped.filter(check_minority, threshold=.95)\n",
    "college_filtered_95.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transforming through a weight-loss "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Month</th>\n",
       "      <th>Week</th>\n",
       "      <th>Weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 1</td>\n",
       "      <td>291</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Amy</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 1</td>\n",
       "      <td>197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 2</td>\n",
       "      <td>288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Amy</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 2</td>\n",
       "      <td>189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 3</td>\n",
       "      <td>283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Amy</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 3</td>\n",
       "      <td>189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 4</td>\n",
       "      <td>283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Amy</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 4</td>\n",
       "      <td>190</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Name Month    Week  Weight\n",
       "0  Bob   Jan  Week 1     291\n",
       "1  Amy   Jan  Week 1     197\n",
       "2  Bob   Jan  Week 2     288\n",
       "3  Amy   Jan  Week 2     189\n",
       "4  Bob   Jan  Week 3     283\n",
       "5  Amy   Jan  Week 3     189\n",
       "6  Bob   Jan  Week 4     283\n",
       "7  Amy   Jan  Week 4     190"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weight_loss = pd.read_csv('data/weight_loss.csv')\n",
    "weight_loss.query('Month == \"Jan\"')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def find_perc_loss(s):\n",
    "    return (s - s.iloc[0]) / s.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.000000\n",
       "2   -0.010309\n",
       "4   -0.027491\n",
       "6   -0.027491\n",
       "Name: Weight, dtype: float64"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bob_jan = weight_loss.query('Name==\"Bob\" and Month==\"Jan\"')\n",
    "find_perc_loss(bob_jan['Weight'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.000000\n",
       "1    0.000000\n",
       "2   -0.010309\n",
       "3   -0.040609\n",
       "4   -0.027491\n",
       "5   -0.040609\n",
       "6   -0.027491\n",
       "7   -0.035533\n",
       "Name: Weight, dtype: float64"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pcnt_loss = weight_loss.groupby(['Name', 'Month'])['Weight'].transform(find_perc_loss)\n",
    "pcnt_loss.head(8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Month</th>\n",
       "      <th>Week</th>\n",
       "      <th>Weight</th>\n",
       "      <th>Perc Weight Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 1</td>\n",
       "      <td>291</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 2</td>\n",
       "      <td>288</td>\n",
       "      <td>-0.010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 3</td>\n",
       "      <td>283</td>\n",
       "      <td>-0.027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 4</td>\n",
       "      <td>283</td>\n",
       "      <td>-0.027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Feb</td>\n",
       "      <td>Week 1</td>\n",
       "      <td>283</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Feb</td>\n",
       "      <td>Week 2</td>\n",
       "      <td>275</td>\n",
       "      <td>-0.028</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Feb</td>\n",
       "      <td>Week 3</td>\n",
       "      <td>268</td>\n",
       "      <td>-0.053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Feb</td>\n",
       "      <td>Week 4</td>\n",
       "      <td>268</td>\n",
       "      <td>-0.053</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Name Month    Week  Weight  Perc Weight Loss\n",
       "0   Bob   Jan  Week 1     291             0.000\n",
       "2   Bob   Jan  Week 2     288            -0.010\n",
       "4   Bob   Jan  Week 3     283            -0.027\n",
       "6   Bob   Jan  Week 4     283            -0.027\n",
       "8   Bob   Feb  Week 1     283             0.000\n",
       "10  Bob   Feb  Week 2     275            -0.028\n",
       "12  Bob   Feb  Week 3     268            -0.053\n",
       "14  Bob   Feb  Week 4     268            -0.053"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weight_loss['Perc Weight Loss'] = pcnt_loss.round(3)\n",
    "weight_loss.query('Name==\"Bob\" and Month in [\"Jan\", \"Feb\"]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Month</th>\n",
       "      <th>Week</th>\n",
       "      <th>Weight</th>\n",
       "      <th>Perc Weight Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 4</td>\n",
       "      <td>283</td>\n",
       "      <td>-0.027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Amy</td>\n",
       "      <td>Jan</td>\n",
       "      <td>Week 4</td>\n",
       "      <td>190</td>\n",
       "      <td>-0.036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Feb</td>\n",
       "      <td>Week 4</td>\n",
       "      <td>268</td>\n",
       "      <td>-0.053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Amy</td>\n",
       "      <td>Feb</td>\n",
       "      <td>Week 4</td>\n",
       "      <td>173</td>\n",
       "      <td>-0.089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Mar</td>\n",
       "      <td>Week 4</td>\n",
       "      <td>261</td>\n",
       "      <td>-0.026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Amy</td>\n",
       "      <td>Mar</td>\n",
       "      <td>Week 4</td>\n",
       "      <td>170</td>\n",
       "      <td>-0.017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Bob</td>\n",
       "      <td>Apr</td>\n",
       "      <td>Week 4</td>\n",
       "      <td>250</td>\n",
       "      <td>-0.042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Amy</td>\n",
       "      <td>Apr</td>\n",
       "      <td>Week 4</td>\n",
       "      <td>161</td>\n",
       "      <td>-0.053</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Name Month    Week  Weight  Perc Weight Loss\n",
       "6   Bob   Jan  Week 4     283            -0.027\n",
       "7   Amy   Jan  Week 4     190            -0.036\n",
       "14  Bob   Feb  Week 4     268            -0.053\n",
       "15  Amy   Feb  Week 4     173            -0.089\n",
       "22  Bob   Mar  Week 4     261            -0.026\n",
       "23  Amy   Mar  Week 4     170            -0.017\n",
       "30  Bob   Apr  Week 4     250            -0.042\n",
       "31  Amy   Apr  Week 4     161            -0.053"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "week4 = weight_loss.query('Week == \"Week 4\"')\n",
    "week4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Name</th>\n",
       "      <th>Amy</th>\n",
       "      <th>Bob</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Month</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Apr</th>\n",
       "      <td>-0.053</td>\n",
       "      <td>-0.042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Feb</th>\n",
       "      <td>-0.089</td>\n",
       "      <td>-0.053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jan</th>\n",
       "      <td>-0.036</td>\n",
       "      <td>-0.027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mar</th>\n",
       "      <td>-0.017</td>\n",
       "      <td>-0.026</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Name     Amy    Bob\n",
       "Month              \n",
       "Apr   -0.053 -0.042\n",
       "Feb   -0.089 -0.053\n",
       "Jan   -0.036 -0.027\n",
       "Mar   -0.017 -0.026"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "winner = week4.pivot(index='Month', columns='Name', values='Perc Weight Loss')\n",
    "winner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow0_col0 {\n",
       "            background-color:  yellow;\n",
       "        }    #T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow1_col0 {\n",
       "            background-color:  yellow;\n",
       "        }    #T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow2_col0 {\n",
       "            background-color:  yellow;\n",
       "        }    #T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow3_col1 {\n",
       "            background-color:  yellow;\n",
       "        }</style>  \n",
       "<table id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68c\" > \n",
       "<thead>    <tr> \n",
       "        <th class=\"index_name level0\" >Name</th> \n",
       "        <th class=\"col_heading level0 col0\" >Amy</th> \n",
       "        <th class=\"col_heading level0 col1\" >Bob</th> \n",
       "        <th class=\"col_heading level0 col2\" >Winner</th> \n",
       "    </tr>    <tr> \n",
       "        <th class=\"index_name level0\" >Month</th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "    </tr></thead> \n",
       "<tbody>    <tr> \n",
       "        <th id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68clevel0_row0\" class=\"row_heading level0 row0\" >Apr</th> \n",
       "        <td id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow0_col0\" class=\"data row0 col0\" >-0.053</td> \n",
       "        <td id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow0_col1\" class=\"data row0 col1\" >-0.042</td> \n",
       "        <td id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow0_col2\" class=\"data row0 col2\" >Amy</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68clevel0_row1\" class=\"row_heading level0 row1\" >Feb</th> \n",
       "        <td id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow1_col0\" class=\"data row1 col0\" >-0.089</td> \n",
       "        <td id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow1_col1\" class=\"data row1 col1\" >-0.053</td> \n",
       "        <td id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow1_col2\" class=\"data row1 col2\" >Amy</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68clevel0_row2\" class=\"row_heading level0 row2\" >Jan</th> \n",
       "        <td id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow2_col0\" class=\"data row2 col0\" >-0.036</td> \n",
       "        <td id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow2_col1\" class=\"data row2 col1\" >-0.027</td> \n",
       "        <td id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow2_col2\" class=\"data row2 col2\" >Amy</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68clevel0_row3\" class=\"row_heading level0 row3\" >Mar</th> \n",
       "        <td id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow3_col0\" class=\"data row3 col0\" >-0.017</td> \n",
       "        <td id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow3_col1\" class=\"data row3 col1\" >-0.026</td> \n",
       "        <td id=\"T_0da2f4d2_b3cd_11e7_b9d4_b8e85647e68crow3_col2\" class=\"data row3 col2\" >Bob</td> \n",
       "    </tr></tbody> \n",
       "</table> "
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1165f5c88>"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "winner['Winner'] = np.where(winner['Amy'] < winner['Bob'], 'Amy', 'Bob')\n",
    "winner.style.highlight_min(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Amy    3\n",
       "Bob    1\n",
       "Name: Winner, dtype: int64"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "winner.Winner.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## There's more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Jan', 'Feb', 'Mar', 'Apr'], dtype=object)"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "week4a = week4.copy()\n",
    "month_chron = week4a['Month'].unique() # or month.drop_duplicates\n",
    "month_chron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Name</th>\n",
       "      <th>Amy</th>\n",
       "      <th>Bob</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Month</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Jan</th>\n",
       "      <td>-0.036</td>\n",
       "      <td>-0.027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Feb</th>\n",
       "      <td>-0.089</td>\n",
       "      <td>-0.053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mar</th>\n",
       "      <td>-0.017</td>\n",
       "      <td>-0.026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Apr</th>\n",
       "      <td>-0.053</td>\n",
       "      <td>-0.042</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Name     Amy    Bob\n",
       "Month              \n",
       "Jan   -0.036 -0.027\n",
       "Feb   -0.089 -0.053\n",
       "Mar   -0.017 -0.026\n",
       "Apr   -0.053 -0.042"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "week4a['Month'] = pd.Categorical(week4a['Month'], \n",
    "                                 categories=month_chron,\n",
    "                                 ordered=True)\n",
    "week4a.pivot(index='Month', columns='Name', values='Perc Weight Loss')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculating weighted mean SAT scores per state with apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7535, 27)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college = pd.read_csv('data/college.csv')\n",
    "subset = ['UGDS', 'SATMTMID', 'SATVRMID']\n",
    "college2 = college.dropna(subset=subset)\n",
    "college.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1184, 27)"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def weighted_math_average(df):\n",
    "    weighted_math = df['UGDS'] * df['SATMTMID']\n",
    "    return int(weighted_math.sum() / df['UGDS'].sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "STABBR\n",
       "AK    503\n",
       "AL    536\n",
       "AR    529\n",
       "AZ    569\n",
       "CA    564\n",
       "dtype: int64"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college2.groupby('STABBR').apply(weighted_math_average).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>INSTNM</th>\n",
       "      <th>CITY</th>\n",
       "      <th>HBCU</th>\n",
       "      <th>MENONLY</th>\n",
       "      <th>WOMENONLY</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>SATMTMID</th>\n",
       "      <th>DISTANCEONLY</th>\n",
       "      <th>UGDS</th>\n",
       "      <th>...</th>\n",
       "      <th>UGDS_2MOR</th>\n",
       "      <th>UGDS_NRA</th>\n",
       "      <th>UGDS_UNKN</th>\n",
       "      <th>PPTUG_EF</th>\n",
       "      <th>CURROPER</th>\n",
       "      <th>PCTPELL</th>\n",
       "      <th>PCTFLOAN</th>\n",
       "      <th>UG25ABV</th>\n",
       "      <th>MD_EARN_WNE_P10</th>\n",
       "      <th>GRAD_DEBT_MDN_SUPP</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>STABBR</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AK</th>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>...</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "      <td>503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AL</th>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>...</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "      <td>536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AR</th>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>...</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "      <td>529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AZ</th>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>...</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>...</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "      <td>564</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        INSTNM  CITY  HBCU  MENONLY  WOMENONLY  RELAFFIL  SATVRMID  SATMTMID  \\\n",
       "STABBR                                                                         \n",
       "AK         503   503   503      503        503       503       503       503   \n",
       "AL         536   536   536      536        536       536       536       536   \n",
       "AR         529   529   529      529        529       529       529       529   \n",
       "AZ         569   569   569      569        569       569       569       569   \n",
       "CA         564   564   564      564        564       564       564       564   \n",
       "\n",
       "        DISTANCEONLY  UGDS         ...          UGDS_2MOR  UGDS_NRA  \\\n",
       "STABBR                             ...                                \n",
       "AK               503   503         ...                503       503   \n",
       "AL               536   536         ...                536       536   \n",
       "AR               529   529         ...                529       529   \n",
       "AZ               569   569         ...                569       569   \n",
       "CA               564   564         ...                564       564   \n",
       "\n",
       "        UGDS_UNKN  PPTUG_EF  CURROPER  PCTPELL  PCTFLOAN  UG25ABV  \\\n",
       "STABBR                                                              \n",
       "AK            503       503       503      503       503      503   \n",
       "AL            536       536       536      536       536      536   \n",
       "AR            529       529       529      529       529      529   \n",
       "AZ            569       569       569      569       569      569   \n",
       "CA            564       564       564      564       564      564   \n",
       "\n",
       "        MD_EARN_WNE_P10  GRAD_DEBT_MDN_SUPP  \n",
       "STABBR                                       \n",
       "AK                  503                 503  \n",
       "AL                  536                 536  \n",
       "AR                  529                 529  \n",
       "AZ                  569                 569  \n",
       "CA                  564                 564  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college2.groupby('STABBR').agg(weighted_math_average).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'UGDS'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5126)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.Int64HashTable.get_item (pandas/_libs/hashtable.c:14010)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: an integer is required",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36magg_series\u001b[0;34m(self, obj, func)\u001b[0m\n\u001b[1;32m   2177\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2178\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_aggregate_series_fast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2179\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m_aggregate_series_fast\u001b[0;34m(self, obj, func)\u001b[0m\n\u001b[1;32m   2197\u001b[0m                                     dummy)\n\u001b[0;32m-> 2198\u001b[0;31m         \u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcounts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrouper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_result\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2199\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcounts\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/src/reduce.pyx\u001b[0m in \u001b[0;36mpandas._libs.lib.SeriesGrouper.get_result (pandas/_libs/lib.c:39105)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/src/reduce.pyx\u001b[0m in \u001b[0;36mpandas._libs.lib.SeriesGrouper.get_result (pandas/_libs/lib.c:38888)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m    841\u001b[0m         \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_builtin_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 842\u001b[0;31m         \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    843\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-76-01eb90aa258d>\u001b[0m in \u001b[0;36mweighted_math_average\u001b[0;34m(df)\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mweighted_math_average\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0mweighted_math\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'UGDS'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'SATMTMID'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mweighted_math\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'UGDS'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/series.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m    600\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 601\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_value\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    602\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_value\u001b[0;34m(self, series, key)\u001b[0m\n\u001b[1;32m   2476\u001b[0m             return self._engine.get_value(s, k,\n\u001b[0;32m-> 2477\u001b[0;31m                                           tz=getattr(series.dtype, 'tz', None))\n\u001b[0m\u001b[1;32m   2478\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_value (pandas/_libs/index.c:4404)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_value (pandas/_libs/index.c:4087)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5210)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'UGDS'",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5126)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.Int64HashTable.get_item (pandas/_libs/hashtable.c:14010)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: an integer is required",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36maggregate\u001b[0;34m(self, func_or_funcs, *args, **kwargs)\u001b[0m\n\u001b[1;32m   2882\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2883\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_python_agg_general\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc_or_funcs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2884\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m_python_agg_general\u001b[0;34m(self, func, *args, **kwargs)\u001b[0m\n\u001b[1;32m    847\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 848\u001b[0;31m                 \u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcounts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrouper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0magg_series\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    849\u001b[0m                 \u001b[0moutput\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_try_cast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumeric_only\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36magg_series\u001b[0;34m(self, obj, func)\u001b[0m\n\u001b[1;32m   2179\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2180\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_aggregate_series_pure_python\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2181\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m_aggregate_series_pure_python\u001b[0;34m(self, obj, func)\u001b[0m\n\u001b[1;32m   2210\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mlabel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgroup\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msplitter\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2211\u001b[0;31m             \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2212\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m    841\u001b[0m         \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_builtin_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 842\u001b[0;31m         \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    843\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-76-01eb90aa258d>\u001b[0m in \u001b[0;36mweighted_math_average\u001b[0;34m(df)\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mweighted_math_average\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0mweighted_math\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'UGDS'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'SATMTMID'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mweighted_math\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'UGDS'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/series.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m    600\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 601\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_value\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    602\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_value\u001b[0;34m(self, series, key)\u001b[0m\n\u001b[1;32m   2476\u001b[0m             return self._engine.get_value(s, k,\n\u001b[0;32m-> 2477\u001b[0;31m                                           tz=getattr(series.dtype, 'tz', None))\n\u001b[0m\u001b[1;32m   2478\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_value (pandas/_libs/index.c:4404)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_value (pandas/_libs/index.c:4087)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5210)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'UGDS'",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5126)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.Int64HashTable.get_item (pandas/_libs/hashtable.c:14010)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: an integer is required",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-79-1351e4f306c7>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mcollege2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroupby\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'STABBR'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'SATMTMID'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0magg\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mweighted_math_average\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36maggregate\u001b[0;34m(self, func_or_funcs, *args, **kwargs)\u001b[0m\n\u001b[1;32m   2883\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_python_agg_general\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc_or_funcs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2884\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2885\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_aggregate_named\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc_or_funcs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2886\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2887\u001b[0m             \u001b[0mindex\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mIndex\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msorted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrouper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnames\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/groupby.py\u001b[0m in \u001b[0;36m_aggregate_named\u001b[0;34m(self, func, *args, **kwargs)\u001b[0m\n\u001b[1;32m   3013\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgroup\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3014\u001b[0m             \u001b[0mgroup\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3015\u001b[0;31m             \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3016\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mSeries\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mIndex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mndarray\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3017\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Must produce aggregated value'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-76-01eb90aa258d>\u001b[0m in \u001b[0;36mweighted_math_average\u001b[0;34m(df)\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mweighted_math_average\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0mweighted_math\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'UGDS'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'SATMTMID'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mweighted_math\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'UGDS'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/series.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m    599\u001b[0m         \u001b[0mkey\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply_if_callable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    600\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 601\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_value\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    602\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    603\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mis_scalar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_value\u001b[0;34m(self, series, key)\u001b[0m\n\u001b[1;32m   2475\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2476\u001b[0m             return self._engine.get_value(s, k,\n\u001b[0;32m-> 2477\u001b[0;31m                                           tz=getattr(series.dtype, 'tz', None))\n\u001b[0m\u001b[1;32m   2478\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2479\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minferred_type\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'integer'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'boolean'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_value (pandas/_libs/index.c:4404)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_value (pandas/_libs/index.c:4087)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc (pandas/_libs/index.c:5210)\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'UGDS'"
     ]
    }
   ],
   "source": [
    "college2.groupby('STABBR')['SATMTMID'].agg(weighted_math_average)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>weighted_math_avg</th>\n",
       "      <th>weighted_verbal_avg</th>\n",
       "      <th>math_avg</th>\n",
       "      <th>verbal_avg</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>STABBR</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AK</th>\n",
       "      <td>503</td>\n",
       "      <td>555</td>\n",
       "      <td>503</td>\n",
       "      <td>555</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AL</th>\n",
       "      <td>536</td>\n",
       "      <td>533</td>\n",
       "      <td>504</td>\n",
       "      <td>508</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AR</th>\n",
       "      <td>529</td>\n",
       "      <td>504</td>\n",
       "      <td>515</td>\n",
       "      <td>491</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AZ</th>\n",
       "      <td>569</td>\n",
       "      <td>557</td>\n",
       "      <td>536</td>\n",
       "      <td>538</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>564</td>\n",
       "      <td>539</td>\n",
       "      <td>562</td>\n",
       "      <td>549</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CO</th>\n",
       "      <td>553</td>\n",
       "      <td>547</td>\n",
       "      <td>540</td>\n",
       "      <td>537</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CT</th>\n",
       "      <td>545</td>\n",
       "      <td>533</td>\n",
       "      <td>522</td>\n",
       "      <td>517</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DC</th>\n",
       "      <td>621</td>\n",
       "      <td>623</td>\n",
       "      <td>588</td>\n",
       "      <td>589</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DE</th>\n",
       "      <td>569</td>\n",
       "      <td>553</td>\n",
       "      <td>495</td>\n",
       "      <td>486</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>565</td>\n",
       "      <td>565</td>\n",
       "      <td>521</td>\n",
       "      <td>529</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        weighted_math_avg  weighted_verbal_avg  math_avg  verbal_avg  count\n",
       "STABBR                                                                     \n",
       "AK                    503                  555       503         555      1\n",
       "AL                    536                  533       504         508     21\n",
       "AR                    529                  504       515         491     16\n",
       "AZ                    569                  557       536         538      6\n",
       "CA                    564                  539       562         549     72\n",
       "CO                    553                  547       540         537     14\n",
       "CT                    545                  533       522         517     14\n",
       "DC                    621                  623       588         589      6\n",
       "DE                    569                  553       495         486      3\n",
       "FL                    565                  565       521         529     38"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import OrderedDict\n",
    "def weighted_average(df):\n",
    "    data = OrderedDict()\n",
    "    weight_m = df['UGDS'] * df['SATMTMID']\n",
    "    weight_v = df['UGDS'] * df['SATVRMID']\n",
    "\n",
    "    data['weighted_math_avg'] = weight_m.sum() / df['UGDS'].sum()\n",
    "    data['weighted_verbal_avg'] = weight_v.sum() / df['UGDS'].sum()\n",
    "    data['math_avg'] = df['SATMTMID'].mean()\n",
    "    data['verbal_avg'] = df['SATVRMID'].mean()\n",
    "    data['count'] = len(df)\n",
    "    return pd.Series(data, dtype='int')\n",
    "\n",
    "college2.groupby('STABBR').apply(weighted_average).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>weighted_math_avg</th>\n",
       "      <th>weighted_verbal_avg</th>\n",
       "      <th>math_avg</th>\n",
       "      <th>verbal_avg</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>STABBR</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AK</th>\n",
       "      <td>503</td>\n",
       "      <td>555</td>\n",
       "      <td>503</td>\n",
       "      <td>555</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AL</th>\n",
       "      <td>536</td>\n",
       "      <td>533</td>\n",
       "      <td>504</td>\n",
       "      <td>508</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AR</th>\n",
       "      <td>529</td>\n",
       "      <td>504</td>\n",
       "      <td>515</td>\n",
       "      <td>491</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AZ</th>\n",
       "      <td>569</td>\n",
       "      <td>557</td>\n",
       "      <td>536</td>\n",
       "      <td>538</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>564</td>\n",
       "      <td>539</td>\n",
       "      <td>562</td>\n",
       "      <td>549</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CO</th>\n",
       "      <td>553</td>\n",
       "      <td>547</td>\n",
       "      <td>540</td>\n",
       "      <td>537</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CT</th>\n",
       "      <td>545</td>\n",
       "      <td>533</td>\n",
       "      <td>522</td>\n",
       "      <td>517</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DC</th>\n",
       "      <td>621</td>\n",
       "      <td>623</td>\n",
       "      <td>588</td>\n",
       "      <td>589</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DE</th>\n",
       "      <td>569</td>\n",
       "      <td>553</td>\n",
       "      <td>495</td>\n",
       "      <td>486</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>565</td>\n",
       "      <td>565</td>\n",
       "      <td>521</td>\n",
       "      <td>529</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        weighted_math_avg  weighted_verbal_avg  math_avg  verbal_avg  count\n",
       "STABBR                                                                     \n",
       "AK                    503                  555       503         555      1\n",
       "AL                    536                  533       504         508     21\n",
       "AR                    529                  504       515         491     16\n",
       "AZ                    569                  557       536         538      6\n",
       "CA                    564                  539       562         549     72\n",
       "CO                    553                  547       540         537     14\n",
       "CT                    545                  533       522         517     14\n",
       "DC                    621                  623       588         589      6\n",
       "DE                    569                  553       495         486      3\n",
       "FL                    565                  565       521         529     38"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import OrderedDict\n",
    "def weighted_average(df):\n",
    "    data = OrderedDict()\n",
    "    weight_m = df['UGDS'] * df['SATMTMID']\n",
    "    weight_v = df['UGDS'] * df['SATVRMID']\n",
    "\n",
    "    wm_avg = weight_m.sum() / df['UGDS'].sum()\n",
    "    wv_avg = weight_v.sum() / df['UGDS'].sum()\n",
    "\n",
    "    data['weighted_math_avg'] = wm_avg\n",
    "    data['weighted_verbal_avg'] = wv_avg\n",
    "    data['math_avg'] = df['SATMTMID'].mean()\n",
    "    data['verbal_avg'] = df['SATVRMID'].mean()\n",
    "    data['count'] = len(df)\n",
    "    return pd.Series(data, dtype='int')\n",
    "\n",
    "college2.groupby('STABBR').apply(weighted_average).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## There's more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>SATMTMID</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>STABBR</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">AL</th>\n",
       "      <th>Arithmetic</th>\n",
       "      <td>504</td>\n",
       "      <td>508</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Weighted</th>\n",
       "      <td>536</td>\n",
       "      <td>533</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Geometric</th>\n",
       "      <td>500</td>\n",
       "      <td>505</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Harmonic</th>\n",
       "      <td>497</td>\n",
       "      <td>502</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">AR</th>\n",
       "      <th>Arithmetic</th>\n",
       "      <td>515</td>\n",
       "      <td>491</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Weighted</th>\n",
       "      <td>529</td>\n",
       "      <td>504</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Geometric</th>\n",
       "      <td>514</td>\n",
       "      <td>489</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Harmonic</th>\n",
       "      <td>513</td>\n",
       "      <td>487</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">AZ</th>\n",
       "      <th>Arithmetic</th>\n",
       "      <td>536</td>\n",
       "      <td>538</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Weighted</th>\n",
       "      <td>569</td>\n",
       "      <td>557</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   SATMTMID  SATVRMID  count\n",
       "STABBR                                      \n",
       "AL     Arithmetic       504       508     21\n",
       "       Weighted         536       533     21\n",
       "       Geometric        500       505     21\n",
       "       Harmonic         497       502     21\n",
       "AR     Arithmetic       515       491     16\n",
       "       Weighted         529       504     16\n",
       "       Geometric        514       489     16\n",
       "       Harmonic         513       487     16\n",
       "AZ     Arithmetic       536       538      6\n",
       "       Weighted         569       557      6"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.stats import gmean, hmean\n",
    "def calculate_means(df):\n",
    "    df_means = pd.DataFrame(index=['Arithmetic', 'Weighted', 'Geometric', 'Harmonic'])\n",
    "    cols = ['SATMTMID', 'SATVRMID']\n",
    "    for col in cols:\n",
    "        arithmetic = df[col].mean()\n",
    "        weighted = np.average(df[col], weights=df['UGDS'])\n",
    "        geometric = gmean(df[col])\n",
    "        harmonic = hmean(df[col])\n",
    "        df_means[col] = [arithmetic, weighted, geometric, harmonic]\n",
    "        \n",
    "    df_means['count'] = len(df)\n",
    "    return df_means.astype(int)\n",
    "\n",
    "college2.groupby('STABBR').filter(lambda x: len(x) != 1).groupby('STABBR').apply(calculate_means).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grouping by continuous variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MONTH</th>\n",
       "      <th>DAY</th>\n",
       "      <th>WEEKDAY</th>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>ORG_AIR</th>\n",
       "      <th>DEST_AIR</th>\n",
       "      <th>SCHED_DEP</th>\n",
       "      <th>DEP_DELAY</th>\n",
       "      <th>AIR_TIME</th>\n",
       "      <th>DIST</th>\n",
       "      <th>SCHED_ARR</th>\n",
       "      <th>ARR_DELAY</th>\n",
       "      <th>DIVERTED</th>\n",
       "      <th>CANCELLED</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>SLC</td>\n",
       "      <td>1625</td>\n",
       "      <td>58.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>590</td>\n",
       "      <td>1905</td>\n",
       "      <td>65.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>UA</td>\n",
       "      <td>DEN</td>\n",
       "      <td>IAD</td>\n",
       "      <td>823</td>\n",
       "      <td>7.0</td>\n",
       "      <td>154.0</td>\n",
       "      <td>1452</td>\n",
       "      <td>1333</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>MQ</td>\n",
       "      <td>DFW</td>\n",
       "      <td>VPS</td>\n",
       "      <td>1305</td>\n",
       "      <td>36.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>641</td>\n",
       "      <td>1453</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>AA</td>\n",
       "      <td>DFW</td>\n",
       "      <td>DCA</td>\n",
       "      <td>1555</td>\n",
       "      <td>7.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>1192</td>\n",
       "      <td>1935</td>\n",
       "      <td>-7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>MCI</td>\n",
       "      <td>1720</td>\n",
       "      <td>48.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>1363</td>\n",
       "      <td>2225</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MONTH  DAY  WEEKDAY AIRLINE ORG_AIR DEST_AIR  SCHED_DEP  DEP_DELAY  \\\n",
       "0      1    1        4      WN     LAX      SLC       1625       58.0   \n",
       "1      1    1        4      UA     DEN      IAD        823        7.0   \n",
       "2      1    1        4      MQ     DFW      VPS       1305       36.0   \n",
       "3      1    1        4      AA     DFW      DCA       1555        7.0   \n",
       "4      1    1        4      WN     LAX      MCI       1720       48.0   \n",
       "\n",
       "   AIR_TIME  DIST  SCHED_ARR  ARR_DELAY  DIVERTED  CANCELLED  \n",
       "0      94.0   590       1905       65.0         0          0  \n",
       "1     154.0  1452       1333      -13.0         0          0  \n",
       "2      85.0   641       1453       35.0         0          0  \n",
       "3     126.0  1192       1935       -7.0         0          0  \n",
       "4     166.0  1363       2225       39.0         0          0  "
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights = pd.read_csv('data/flights.csv')\n",
    "flights.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.DIST.hasnans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(58492, 14)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.dropna(subset=['DIST']).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     (500.0, 1000.0]\n",
       "1    (1000.0, 2000.0]\n",
       "2     (500.0, 1000.0]\n",
       "3    (1000.0, 2000.0]\n",
       "4    (1000.0, 2000.0]\n",
       "Name: DIST, dtype: category\n",
       "Categories (5, interval[float64]): [(-inf, 200.0] < (200.0, 500.0] < (500.0, 1000.0] < (1000.0, 2000.0] < (2000.0, inf]]"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bins = [-np.inf, 200, 500, 1000, 2000, np.inf]\n",
    "cuts = pd.cut(flights['DIST'], bins=bins)\n",
    "cuts.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(500.0, 1000.0]     20659\n",
       "(200.0, 500.0]      15874\n",
       "(1000.0, 2000.0]    14186\n",
       "(2000.0, inf]        4054\n",
       "(-inf, 200.0]        3719\n",
       "Name: DIST, dtype: int64"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cuts.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DIST            AIRLINE\n",
       "(-inf, 200.0]   OO         0.326\n",
       "                EV         0.289\n",
       "                MQ         0.211\n",
       "                DL         0.086\n",
       "                AA         0.052\n",
       "                UA         0.027\n",
       "                WN         0.009\n",
       "(200.0, 500.0]  WN         0.194\n",
       "                DL         0.189\n",
       "                OO         0.159\n",
       "                EV         0.156\n",
       "                MQ         0.100\n",
       "                AA         0.071\n",
       "                UA         0.062\n",
       "                VX         0.028\n",
       "Name: AIRLINE, dtype: float64"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.groupby(cuts)['AIRLINE'].value_counts(normalize=True).round(3).head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How it works..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DIST              AIRLINE\n",
       "(-inf, 200.0]     OO         0.325625\n",
       "                  EV         0.289325\n",
       "                  MQ         0.210809\n",
       "                  DL         0.086045\n",
       "                  AA         0.052165\n",
       "                  UA         0.027427\n",
       "                  WN         0.008604\n",
       "(200.0, 500.0]    WN         0.193902\n",
       "                  DL         0.188736\n",
       "                  OO         0.158687\n",
       "                  EV         0.156293\n",
       "                  MQ         0.100164\n",
       "                  AA         0.071375\n",
       "                  UA         0.062051\n",
       "                  VX         0.028222\n",
       "                  US         0.016001\n",
       "                  NK         0.011843\n",
       "                  B6         0.006867\n",
       "                  F9         0.004914\n",
       "                  AS         0.000945\n",
       "(500.0, 1000.0]   DL         0.205625\n",
       "                  AA         0.143908\n",
       "                  WN         0.138196\n",
       "                  UA         0.131129\n",
       "                  OO         0.106443\n",
       "                  EV         0.100683\n",
       "                  MQ         0.051213\n",
       "                  F9         0.038192\n",
       "                  NK         0.029527\n",
       "                  US         0.025316\n",
       "                  AS         0.023234\n",
       "                  VX         0.003582\n",
       "                  B6         0.002953\n",
       "(1000.0, 2000.0]  AA         0.263781\n",
       "                  UA         0.199070\n",
       "                  DL         0.165092\n",
       "                  WN         0.159664\n",
       "                  OO         0.046454\n",
       "                  NK         0.045115\n",
       "                  US         0.040462\n",
       "                  F9         0.030664\n",
       "                  AS         0.015931\n",
       "                  EV         0.015579\n",
       "                  VX         0.012125\n",
       "                  B6         0.003313\n",
       "                  MQ         0.002749\n",
       "(2000.0, inf]     UA         0.289097\n",
       "                  AA         0.211643\n",
       "                  DL         0.171436\n",
       "                  B6         0.080414\n",
       "                  VX         0.073754\n",
       "                  US         0.065121\n",
       "                  WN         0.046374\n",
       "                  HA         0.027627\n",
       "                  NK         0.019240\n",
       "                  AS         0.011593\n",
       "                  F9         0.003700\n",
       "Name: AIRLINE, dtype: float64"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.groupby(cuts)['AIRLINE'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## There's more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DIST                  \n",
       "(-inf, 200.0]     0.25    0.43\n",
       "                  0.50    0.50\n",
       "                  0.75    0.57\n",
       "(200.0, 500.0]    0.25    0.77\n",
       "                  0.50    0.92\n",
       "                  0.75    1.05\n",
       "(500.0, 1000.0]   0.25    1.43\n",
       "                  0.50    1.65\n",
       "                  0.75    1.92\n",
       "(1000.0, 2000.0]  0.25    2.50\n",
       "                  0.50    2.93\n",
       "                  0.75    3.40\n",
       "(2000.0, inf]     0.25    4.30\n",
       "                  0.50    4.70\n",
       "                  0.75    5.03\n",
       "Name: AIR_TIME, dtype: float64"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.groupby(cuts)['AIR_TIME'].quantile(q=[.25, .5, .75]).div(60).round(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col9 {\n",
       "            background-color:  yellow;\n",
       "        }    #T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col13 {\n",
       "            background-color:  yellow;\n",
       "        }    #T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col3 {\n",
       "            background-color:  yellow;\n",
       "        }    #T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col0 {\n",
       "            background-color:  yellow;\n",
       "        }    #T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col10 {\n",
       "            background-color:  yellow;\n",
       "        }</style>  \n",
       "<table id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68c\" > \n",
       "<thead>    <tr> \n",
       "        <th class=\"index_name level0\" >AIRLINE</th> \n",
       "        <th class=\"col_heading level0 col0\" >AA</th> \n",
       "        <th class=\"col_heading level0 col1\" >AS</th> \n",
       "        <th class=\"col_heading level0 col2\" >B6</th> \n",
       "        <th class=\"col_heading level0 col3\" >DL</th> \n",
       "        <th class=\"col_heading level0 col4\" >EV</th> \n",
       "        <th class=\"col_heading level0 col5\" >F9</th> \n",
       "        <th class=\"col_heading level0 col6\" >HA</th> \n",
       "        <th class=\"col_heading level0 col7\" >MQ</th> \n",
       "        <th class=\"col_heading level0 col8\" >NK</th> \n",
       "        <th class=\"col_heading level0 col9\" >OO</th> \n",
       "        <th class=\"col_heading level0 col10\" >UA</th> \n",
       "        <th class=\"col_heading level0 col11\" >US</th> \n",
       "        <th class=\"col_heading level0 col12\" >VX</th> \n",
       "        <th class=\"col_heading level0 col13\" >WN</th> \n",
       "    </tr>    <tr> \n",
       "        <th class=\"index_name level0\" >DIST</th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "    </tr></thead> \n",
       "<tbody>    <tr> \n",
       "        <th id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68clevel0_row0\" class=\"row_heading level0 row0\" >Under an Hour</th> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col0\" class=\"data row0 col0\" >0.052</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col1\" class=\"data row0 col1\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col2\" class=\"data row0 col2\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col3\" class=\"data row0 col3\" >0.086</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col4\" class=\"data row0 col4\" >0.289</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col5\" class=\"data row0 col5\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col6\" class=\"data row0 col6\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col7\" class=\"data row0 col7\" >0.211</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col8\" class=\"data row0 col8\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col9\" class=\"data row0 col9\" >0.326</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col10\" class=\"data row0 col10\" >0.027</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col11\" class=\"data row0 col11\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col12\" class=\"data row0 col12\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow0_col13\" class=\"data row0 col13\" >0.009</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68clevel0_row1\" class=\"row_heading level0 row1\" >1 Hour</th> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col0\" class=\"data row1 col0\" >0.071</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col1\" class=\"data row1 col1\" >0.001</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col2\" class=\"data row1 col2\" >0.007</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col3\" class=\"data row1 col3\" >0.189</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col4\" class=\"data row1 col4\" >0.156</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col5\" class=\"data row1 col5\" >0.005</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col6\" class=\"data row1 col6\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col7\" class=\"data row1 col7\" >0.1</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col8\" class=\"data row1 col8\" >0.012</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col9\" class=\"data row1 col9\" >0.159</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col10\" class=\"data row1 col10\" >0.062</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col11\" class=\"data row1 col11\" >0.016</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col12\" class=\"data row1 col12\" >0.028</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow1_col13\" class=\"data row1 col13\" >0.194</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68clevel0_row2\" class=\"row_heading level0 row2\" >1-2 Hours</th> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col0\" class=\"data row2 col0\" >0.144</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col1\" class=\"data row2 col1\" >0.023</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col2\" class=\"data row2 col2\" >0.003</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col3\" class=\"data row2 col3\" >0.206</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col4\" class=\"data row2 col4\" >0.101</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col5\" class=\"data row2 col5\" >0.038</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col6\" class=\"data row2 col6\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col7\" class=\"data row2 col7\" >0.051</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col8\" class=\"data row2 col8\" >0.03</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col9\" class=\"data row2 col9\" >0.106</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col10\" class=\"data row2 col10\" >0.131</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col11\" class=\"data row2 col11\" >0.025</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col12\" class=\"data row2 col12\" >0.004</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow2_col13\" class=\"data row2 col13\" >0.138</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68clevel0_row3\" class=\"row_heading level0 row3\" >2-4 Hours</th> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col0\" class=\"data row3 col0\" >0.264</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col1\" class=\"data row3 col1\" >0.016</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col2\" class=\"data row3 col2\" >0.003</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col3\" class=\"data row3 col3\" >0.165</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col4\" class=\"data row3 col4\" >0.016</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col5\" class=\"data row3 col5\" >0.031</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col6\" class=\"data row3 col6\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col7\" class=\"data row3 col7\" >0.003</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col8\" class=\"data row3 col8\" >0.045</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col9\" class=\"data row3 col9\" >0.046</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col10\" class=\"data row3 col10\" >0.199</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col11\" class=\"data row3 col11\" >0.04</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col12\" class=\"data row3 col12\" >0.012</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow3_col13\" class=\"data row3 col13\" >0.16</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68clevel0_row4\" class=\"row_heading level0 row4\" >4+ Hours</th> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col0\" class=\"data row4 col0\" >0.212</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col1\" class=\"data row4 col1\" >0.012</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col2\" class=\"data row4 col2\" >0.08</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col3\" class=\"data row4 col3\" >0.171</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col4\" class=\"data row4 col4\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col5\" class=\"data row4 col5\" >0.004</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col6\" class=\"data row4 col6\" >0.028</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col7\" class=\"data row4 col7\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col8\" class=\"data row4 col8\" >0.019</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col9\" class=\"data row4 col9\" >nan</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col10\" class=\"data row4 col10\" >0.289</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col11\" class=\"data row4 col11\" >0.065</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col12\" class=\"data row4 col12\" >0.074</td> \n",
       "        <td id=\"T_23d9b1fa_b3cd_11e7_9216_b8e85647e68crow4_col13\" class=\"data row4 col13\" >0.046</td> \n",
       "    </tr></tbody> \n",
       "</table> "
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x11aaea898>"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels=['Under an Hour', '1 Hour', '1-2 Hours', '2-4 Hours', '4+ Hours']\n",
    "cuts2 = pd.cut(flights['DIST'], bins=bins, labels=labels)\n",
    "flights.groupby(cuts2)['AIRLINE'].value_counts(normalize=True).round(3).unstack().style.highlight_max(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Counting the total number of flights between cities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MONTH</th>\n",
       "      <th>DAY</th>\n",
       "      <th>WEEKDAY</th>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>ORG_AIR</th>\n",
       "      <th>DEST_AIR</th>\n",
       "      <th>SCHED_DEP</th>\n",
       "      <th>DEP_DELAY</th>\n",
       "      <th>AIR_TIME</th>\n",
       "      <th>DIST</th>\n",
       "      <th>SCHED_ARR</th>\n",
       "      <th>ARR_DELAY</th>\n",
       "      <th>DIVERTED</th>\n",
       "      <th>CANCELLED</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>SLC</td>\n",
       "      <td>1625</td>\n",
       "      <td>58.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>590</td>\n",
       "      <td>1905</td>\n",
       "      <td>65.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>UA</td>\n",
       "      <td>DEN</td>\n",
       "      <td>IAD</td>\n",
       "      <td>823</td>\n",
       "      <td>7.0</td>\n",
       "      <td>154.0</td>\n",
       "      <td>1452</td>\n",
       "      <td>1333</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>MQ</td>\n",
       "      <td>DFW</td>\n",
       "      <td>VPS</td>\n",
       "      <td>1305</td>\n",
       "      <td>36.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>641</td>\n",
       "      <td>1453</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>AA</td>\n",
       "      <td>DFW</td>\n",
       "      <td>DCA</td>\n",
       "      <td>1555</td>\n",
       "      <td>7.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>1192</td>\n",
       "      <td>1935</td>\n",
       "      <td>-7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>MCI</td>\n",
       "      <td>1720</td>\n",
       "      <td>48.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>1363</td>\n",
       "      <td>2225</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MONTH  DAY  WEEKDAY AIRLINE ORG_AIR DEST_AIR  SCHED_DEP  DEP_DELAY  \\\n",
       "0      1    1        4      WN     LAX      SLC       1625       58.0   \n",
       "1      1    1        4      UA     DEN      IAD        823        7.0   \n",
       "2      1    1        4      MQ     DFW      VPS       1305       36.0   \n",
       "3      1    1        4      AA     DFW      DCA       1555        7.0   \n",
       "4      1    1        4      WN     LAX      MCI       1720       48.0   \n",
       "\n",
       "   AIR_TIME  DIST  SCHED_ARR  ARR_DELAY  DIVERTED  CANCELLED  \n",
       "0      94.0   590       1905       65.0         0          0  \n",
       "1     154.0  1452       1333      -13.0         0          0  \n",
       "2      85.0   641       1453       35.0         0          0  \n",
       "3     126.0  1192       1935       -7.0         0          0  \n",
       "4     166.0  1363       2225       39.0         0          0  "
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights = pd.read_csv('data/flights.csv')\n",
    "flights.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ORG_AIR  DEST_AIR\n",
       "ATL      ABE         31\n",
       "         ABQ         16\n",
       "         ABY         19\n",
       "         ACY          6\n",
       "         AEX         40\n",
       "dtype: int64"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights_ct = flights.groupby(['ORG_AIR', 'DEST_AIR']).size()\n",
    "flights_ct.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ORG_AIR  DEST_AIR\n",
       "ATL      IAH         121\n",
       "IAH      ATL         148\n",
       "dtype: int64"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights_ct.loc[[('ATL', 'IAH'), ('IAH', 'ATL')]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ORG_AIR</th>\n",
       "      <th>DEST_AIR</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LAX</td>\n",
       "      <td>SLC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>DEN</td>\n",
       "      <td>IAD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DFW</td>\n",
       "      <td>VPS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>DCA</td>\n",
       "      <td>DFW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>LAX</td>\n",
       "      <td>MCI</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  ORG_AIR DEST_AIR\n",
       "0     LAX      SLC\n",
       "1     DEN      IAD\n",
       "2     DFW      VPS\n",
       "3     DCA      DFW\n",
       "4     LAX      MCI"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights_sort = flights[['ORG_AIR', 'DEST_AIR']].apply(sorted, axis=1)\n",
    "flights_sort.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIR1  AIR2\n",
       "ABE   ATL     31\n",
       "      ORD     24\n",
       "ABI   DFW     74\n",
       "ABQ   ATL     16\n",
       "      DEN     46\n",
       "dtype: int64"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rename_dict = {'ORG_AIR':'AIR1','DEST_AIR':'AIR2'}\n",
    "flights_sort = flights_sort.rename(columns=rename_dict)\n",
    "flights_ct2 = flights_sort.groupby(['AIR1', 'AIR2']).size()\n",
    "flights_ct2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "269"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights_ct2.loc[('ATL', 'IAH')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexingError",
     "evalue": "Too many indexers",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexingError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-98-56147a7d0bb5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mflights_ct2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'IAH'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'ATL'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/indexing.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m   1323\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mKeyError\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mIndexError\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1324\u001b[0m                 \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1325\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_getitem_tuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1326\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1327\u001b[0m             \u001b[0mkey\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply_if_callable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/indexing.py\u001b[0m in \u001b[0;36m_getitem_tuple\u001b[0;34m(self, tup)\u001b[0m\n\u001b[1;32m    839\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    840\u001b[0m         \u001b[0;31m# no multi-index, so validate all of the indexers\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 841\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_has_valid_tuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtup\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    842\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    843\u001b[0m         \u001b[0;31m# ugly hack for GH #836\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Ted/anaconda/lib/python3.6/site-packages/pandas/core/indexing.py\u001b[0m in \u001b[0;36m_has_valid_tuple\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m    186\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    187\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mi\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mndim\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 188\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mIndexingError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Too many indexers'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    189\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_has_valid_type\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    190\u001b[0m                 raise ValueError(\"Location based indexing can only have [%s] \"\n",
      "\u001b[0;31mIndexingError\u001b[0m: Too many indexers"
     ]
    }
   ],
   "source": [
    "flights_ct2.loc[('IAH', 'ATL')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## There's more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['LAX', 'SLC'],\n",
       "       ['DEN', 'IAD'],\n",
       "       ['DFW', 'VPS'],\n",
       "       ['DCA', 'DFW'],\n",
       "       ['LAX', 'MCI'],\n",
       "       ['IAH', 'SAN'],\n",
       "       ['DFW', 'MSY'],\n",
       "       ['PHX', 'SFO'],\n",
       "       ['ORD', 'STL'],\n",
       "       ['IAH', 'SJC']], dtype=object)"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_sorted = np.sort(flights[['ORG_AIR', 'DEST_AIR']])\n",
    "data_sorted[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights_sort2 = pd.DataFrame(data_sorted, columns=['AIR1', 'AIR2'])\n",
    "fs_orig = flights_sort.rename(columns={'ORG_AIR':'AIR1', 'DEST_AIR':'AIR2'})\n",
    "flights_sort2.equals(fs_orig)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7.82 s ± 189 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit flights_sort = flights[['ORG_AIR', 'DEST_AIR']].apply(sorted, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.9 ms ± 325 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "data_sorted = np.sort(flights[['ORG_AIR', 'DEST_AIR']])\n",
    "flights_sort2 = pd.DataFrame(data_sorted, columns=['AIR1', 'AIR2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding the longest streak of on-time flights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    1\n",
       "2    1\n",
       "3    0\n",
       "4    1\n",
       "5    1\n",
       "6    1\n",
       "7    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series([1, 1, 1, 0, 1, 1, 1, 0])\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    2\n",
       "2    3\n",
       "3    3\n",
       "4    4\n",
       "5    5\n",
       "6    6\n",
       "7    6\n",
       "dtype: int64"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s1 = s.cumsum()\n",
    "s1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    NaN\n",
       "1    1.0\n",
       "2    1.0\n",
       "3   -3.0\n",
       "4    4.0\n",
       "5    1.0\n",
       "6    1.0\n",
       "7   -6.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.mul(s1).diff()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    NaN\n",
       "1    NaN\n",
       "2    NaN\n",
       "3   -3.0\n",
       "4    NaN\n",
       "5    NaN\n",
       "6    NaN\n",
       "7   -6.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.mul(s1).diff().where(lambda x: x < 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1.0\n",
       "1    2.0\n",
       "2    3.0\n",
       "3    0.0\n",
       "4    1.0\n",
       "5    2.0\n",
       "6    3.0\n",
       "7    0.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.mul(s1).diff().where(lambda x: x < 0).ffill().add(s1, fill_value=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>ORG_AIR</th>\n",
       "      <th>ON_TIME</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>UA</td>\n",
       "      <td>DEN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>MQ</td>\n",
       "      <td>DFW</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AA</td>\n",
       "      <td>DFW</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>UA</td>\n",
       "      <td>IAH</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>AA</td>\n",
       "      <td>DFW</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>F9</td>\n",
       "      <td>SFO</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>AA</td>\n",
       "      <td>ORD</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>UA</td>\n",
       "      <td>IAH</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  AIRLINE ORG_AIR  ON_TIME\n",
       "0      WN     LAX        0\n",
       "1      UA     DEN        1\n",
       "2      MQ     DFW        0\n",
       "3      AA     DFW        1\n",
       "4      WN     LAX        0\n",
       "5      UA     IAH        1\n",
       "6      AA     DFW        0\n",
       "7      F9     SFO        1\n",
       "8      AA     ORD        1\n",
       "9      UA     IAH        1"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights = pd.read_csv('data/flights.csv')\n",
    "flights['ON_TIME'] = flights['ARR_DELAY'].lt(15).astype(int)\n",
    "flights[['AIRLINE', 'ORG_AIR', 'ON_TIME']].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def max_streak(s):\n",
    "    s1 = s.cumsum()\n",
    "    return s.mul(s1).diff().where(lambda x: x < 0) \\\n",
    "            .ffill().add(s1, fill_value=0).max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>size</th>\n",
       "      <th>max_streak</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>ORG_AIR</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">AA</th>\n",
       "      <th>ATL</th>\n",
       "      <td>0.82</td>\n",
       "      <td>233</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DEN</th>\n",
       "      <td>0.74</td>\n",
       "      <td>219</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DFW</th>\n",
       "      <td>0.78</td>\n",
       "      <td>4006</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IAH</th>\n",
       "      <td>0.80</td>\n",
       "      <td>196</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LAS</th>\n",
       "      <td>0.79</td>\n",
       "      <td>374</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 mean  size  max_streak\n",
       "AIRLINE ORG_AIR                        \n",
       "AA      ATL      0.82   233          15\n",
       "        DEN      0.74   219          17\n",
       "        DFW      0.78  4006          64\n",
       "        IAH      0.80   196          24\n",
       "        LAS      0.79   374          29"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.sort_values(['MONTH', 'DAY', 'SCHED_DEP']) \\\n",
    "       .groupby(['AIRLINE', 'ORG_AIR'])['ON_TIME'] \\\n",
    "       .agg(['mean', 'size', max_streak]).round(2).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## There's more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def max_delay_streak(df):\n",
    "    df = df.reset_index(drop=True)\n",
    "    s = 1 - df['ON_TIME']\n",
    "    s1 = s.cumsum()\n",
    "    streak = s.mul(s1).diff().where(lambda x: x < 0) \\\n",
    "              .ffill().add(s1, fill_value=0)\n",
    "    last_idx = streak.idxmax()\n",
    "    first_idx = last_idx - streak.max() + 1\n",
    "    df_return = df.loc[[first_idx, last_idx], ['MONTH', 'DAY']]\n",
    "    df_return['streak'] = streak.max()\n",
    "    df_return.index = ['first', 'last']\n",
    "    df_return.index.name='streak_row'\n",
    "    return df_return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>MONTH</th>\n",
       "      <th>DAY</th>\n",
       "      <th>streak</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>ORG_AIR</th>\n",
       "      <th>streak_row</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">AA</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">DFW</th>\n",
       "      <th>first</th>\n",
       "      <td>2.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>38.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>last</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>38.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">MQ</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">ORD</th>\n",
       "      <th>first</th>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>28.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>last</th>\n",
       "      <td>1.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>28.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">DFW</th>\n",
       "      <th>first</th>\n",
       "      <td>2.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>last</th>\n",
       "      <td>2.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">NK</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">ORD</th>\n",
       "      <th>first</th>\n",
       "      <td>6.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>last</th>\n",
       "      <td>6.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">DL</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">ATL</th>\n",
       "      <th>first</th>\n",
       "      <td>12.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>last</th>\n",
       "      <td>12.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            MONTH   DAY  streak\n",
       "AIRLINE ORG_AIR streak_row                     \n",
       "AA      DFW     first         2.0  26.0    38.0\n",
       "                last          3.0   1.0    38.0\n",
       "MQ      ORD     first         1.0   6.0    28.0\n",
       "                last          1.0  12.0    28.0\n",
       "        DFW     first         2.0  21.0    25.0\n",
       "                last          2.0  26.0    25.0\n",
       "NK      ORD     first         6.0   7.0    15.0\n",
       "                last          6.0  18.0    15.0\n",
       "DL      ATL     first        12.0  23.0    14.0\n",
       "                last         12.0  24.0    14.0"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.sort_values(['MONTH', 'DAY', 'SCHED_DEP']) \\\n",
    "       .groupby(['AIRLINE', 'ORG_AIR']) \\\n",
    "       .apply(max_delay_streak) \\\n",
    "       .sort_values(['streak','MONTH','DAY'], ascending=[False, True, True]).head(10)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}