ls88_c04_subchar.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.5.6'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# HIDDEN\n",
    "# This useful nonsense should just go at the top of your notebook.\n",
    "from datascience import *\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plots\n",
    "import numpy as np\n",
    "plots.style.use('fivethirtyeight')\n",
    "# datascience version number of last run of this notebook\n",
    "version.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Class 4: Microdata, subcharacteristics, and crazy cat people</h1>\n",
    "\n",
    "Many of you will want to <i>compare subgroups</i> within a dataset as part of your term paper research. \n",
    "\n",
    "For example, maybe you have a dataset of people that includes measures of their health and their smoking. Smoking could be a dichotomous yes/no or 0/1 measure, in which case you'd like to compare smokers to nonsmokers. Or it might be a tricohotomous measure 1/2/3 measure, where we know *current* smokers, *former* smokers, and *never* smokers. There are several interesting comparisons one could draw between those three groups. And measuring them separately is important. Let's use some Python to separate data tables into sub-tables. \n",
    "\n",
    "<h2>Data sources</h2>\n",
    "\n",
    "Let's examine data from the U.S. <a href=\"http://hrsonline.isr.umich.edu\">Health and Retirement Study</a> (HRS), a panel survey of thousands of Americans aged 50 and over and their spouses that is conducted biennially (every 2 years) starting in 1992.\n",
    "\n",
    "In its 11th wave in 2012, a subsample of about 1,700 individuals were asked about <b>pet ownership</b>. Let's examine pet ownership and health in these data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<u><h3>File I/O</h3></u>\n",
    "\n",
    "Python can import an external file into a table using a this function and a call to a valid URL like this one, where the target file probably needs to be comma-delimited (CSV):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "    <thead>\n",
       "        <tr>\n",
       "            <th>hhidpn</th> <th>age</th> <th>sex</th> <th>edyrs</th> <th>health</th> <th>anypets</th> <th>numdogs</th> <th>numcats</th>\n",
       "        </tr>\n",
       "    </thead>\n",
       "    <tbody>\n",
       "        <tr>\n",
       "            <td>10059030</td> <td>84  </td> <td>1   </td> <td>17   </td> <td>2     </td> <td>0      </td> <td>0      </td> <td>0      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>10210020</td> <td>73  </td> <td>1   </td> <td>6    </td> <td>5     </td> <td>0      </td> <td>0      </td> <td>0      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>10372010</td> <td>76  </td> <td>2   </td> <td>10   </td> <td>3     </td> <td>0      </td> <td>0      </td> <td>0      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>10395020</td> <td>74  </td> <td>2   </td> <td>15   </td> <td>5     </td> <td>1      </td> <td>1      </td> <td>0      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>10458030</td> <td>68  </td> <td>2   </td> <td>16   </td> <td>4     </td> <td>1      </td> <td>0      </td> <td>1      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>10475010</td> <td>81  </td> <td>2   </td> <td>17   </td> <td>3     </td> <td>0      </td> <td>0      </td> <td>0      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>10648010</td> <td>71  </td> <td>1   </td> <td>12   </td> <td>4     </td> <td>0      </td> <td>0      </td> <td>0      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>10773030</td> <td>83  </td> <td>1   </td> <td>17   </td> <td>2     </td> <td>0      </td> <td>0      </td> <td>0      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>10818040</td> <td>63  </td> <td>2   </td> <td>13   </td> <td>2     </td> <td>0      </td> <td>0      </td> <td>0      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>11141010</td> <td>78  </td> <td>2   </td> <td>16   </td> <td>2     </td> <td>0      </td> <td>0      </td> <td>0      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "</table>\n",
       "<p>... (1705 rows omitted)</p"
      ],
      "text/plain": [
       "hhidpn   | age  | sex  | edyrs | health | anypets | numdogs | numcats\n",
       "10059030 | 84   | 1    | 17    | 2      | 0       | 0       | 0\n",
       "10210020 | 73   | 1    | 6     | 5      | 0       | 0       | 0\n",
       "10372010 | 76   | 2    | 10    | 3      | 0       | 0       | 0\n",
       "10395020 | 74   | 2    | 15    | 5      | 1       | 1       | 0\n",
       "10458030 | 68   | 2    | 16    | 4      | 1       | 0       | 1\n",
       "10475010 | 81   | 2    | 17    | 3      | 0       | 0       | 0\n",
       "10648010 | 71   | 1    | 12    | 4      | 0       | 0       | 0\n",
       "10773030 | 83   | 1    | 17    | 2      | 0       | 0       | 0\n",
       "10818040 | 63   | 2    | 13    | 2      | 0       | 0       | 0\n",
       "11141010 | 78   | 2    | 16    | 2      | 0       | 0       | 0\n",
       "... (1705 rows omitted)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HRSpets = Table.read_table(\"http://demog.berkeley.edu/~redwards/Courses/LS88/c04_hrspets.csv\")\n",
    "HRSpets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://vignette1.wikia.nocookie.net/simpsons/images/b/b5/230px-Eleanor_Abernathy.png/revision/latest?cb=20140817113422\" align=right valign=top width=160> \n",
    "Let's debunk or rebunk some ridiculous theories, and then let's actually look at health and how it varies.\n",
    "\n",
    "First, some fun. \n",
    "\n",
    "<h3>The Theory of the Crazy Cat Lady</h3>\n",
    "\n",
    "Everybody's favorite TV show of the 1990s, *The Simpsons*, included a memorable character apparently named <a href=\"http://simpsons.wikia.com/wiki/Eleanor_Abernathy\">Eleanor Abernathy MD JD, whom most of us know as the Crazy Cat Lady</a>. Go figure.\n",
    "\n",
    "\n",
    "Are the cat owners in the HRS more likely to be female?\n",
    "\n",
    "To find out, you can create a sub-table that includes only the cat owners (numcats > 0) using this code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "    <thead>\n",
       "        <tr>\n",
       "            <th>hhidpn</th> <th>age</th> <th>sex</th> <th>edyrs</th> <th>health</th> <th>anypets</th> <th>numdogs</th> <th>numcats</th>\n",
       "        </tr>\n",
       "    </thead>\n",
       "    <tbody>\n",
       "        <tr>\n",
       "            <td>10458030</td> <td>68  </td> <td>2   </td> <td>16   </td> <td>4     </td> <td>1      </td> <td>0      </td> <td>1      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>11332010</td> <td>80  </td> <td>1   </td> <td>12   </td> <td>3     </td> <td>1      </td> <td>1      </td> <td>2      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>11612010</td> <td>72  </td> <td>1   </td> <td>10   </td> <td>3     </td> <td>1      </td> <td>0      </td> <td>2      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>11876010</td> <td>76  </td> <td>2   </td> <td>12   </td> <td>4     </td> <td>1      </td> <td>0      </td> <td>1      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>12315040</td> <td>70  </td> <td>2   </td> <td>12   </td> <td>3     </td> <td>1      </td> <td>0      </td> <td>1      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>12738020</td> <td>64  </td> <td>2   </td> <td>3    </td> <td>5     </td> <td>1      </td> <td>1      </td> <td>1      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>13140010</td> <td>80  </td> <td>2   </td> <td>12   </td> <td>2     </td> <td>1      </td> <td>1      </td> <td>1      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>14427010</td> <td>71  </td> <td>1   </td> <td>12   </td> <td>3     </td> <td>1      </td> <td>2      </td> <td>1      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>14587010</td> <td>71  </td> <td>2   </td> <td>14   </td> <td>1     </td> <td>1      </td> <td>1      </td> <td>1      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>15516020</td> <td>71  </td> <td>2   </td> <td>12   </td> <td>1     </td> <td>1      </td> <td>0      </td> <td>3      </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "</table>\n",
       "<p>... (337 rows omitted)</p"
      ],
      "text/plain": [
       "hhidpn   | age  | sex  | edyrs | health | anypets | numdogs | numcats\n",
       "10458030 | 68   | 2    | 16    | 4      | 1       | 0       | 1\n",
       "11332010 | 80   | 1    | 12    | 3      | 1       | 1       | 2\n",
       "11612010 | 72   | 1    | 10    | 3      | 1       | 0       | 2\n",
       "11876010 | 76   | 2    | 12    | 4      | 1       | 0       | 1\n",
       "12315040 | 70   | 2    | 12    | 3      | 1       | 0       | 1\n",
       "12738020 | 64   | 2    | 3     | 5      | 1       | 1       | 1\n",
       "13140010 | 80   | 2    | 12    | 2      | 1       | 1       | 1\n",
       "14427010 | 71   | 1    | 12    | 3      | 1       | 2       | 1\n",
       "14587010 | 71   | 2    | 14    | 1      | 1       | 1       | 1\n",
       "15516020 | 71   | 2    | 12    | 1      | 1       | 0       | 3\n",
       "... (337 rows omitted)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Catowners = HRSpets.where(HRSpets['numcats'] > 0)\n",
    "Catowners"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's measure the average sex of the cat owners. To select just a column of the table, call the table name followed by square brackets around the single-quoted column that you want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "catowners_sex = Catowners['sex']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To measure the average or mean of that column, attach `.mean()` to the end of the column name. You can combine this with the column-selecting command one step ago, too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.6829971181556196"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Catowners['sex'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a quick aside, note that there are all kinds of nifty functions in numpy like `.mean()`, and another one of probable interest is the standard deviation, `.sd()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.46530855864333526"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Catowners['sex'].std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Anyway, back to means:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.6829971181556196"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "catowners_sex.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks to me like Python *might* also like to perform `.mean()` on an entire matrix, column-wise appears to be the default (along with a fun error message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.4/site-packages/datascience/tables.py:185: FutureWarning: Implicit column method lookup is deprecated.\n",
      "  warnings.warn(\"Implicit column method lookup is deprecated.\", FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "    <thead>\n",
       "        <tr>\n",
       "            <th>hhidpn</th> <th>age</th> <th>sex</th> <th>edyrs</th> <th>health</th> <th>anypets</th> <th>numdogs</th> <th>numcats</th>\n",
       "        </tr>\n",
       "    </thead>\n",
       "    <tbody>\n",
       "        <tr>\n",
       "            <td>3.06675e+08</td> <td>64.3487</td> <td>1.683</td> <td>13.1037</td> <td>2.71182</td> <td>1      </td> <td>0.850144</td> <td>2.04611</td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "hhidpn      | age     | sex   | edyrs   | health  | anypets | numdogs  | numcats\n",
       "3.06675e+08 | 64.3487 | 1.683 | 13.1037 | 2.71182 | 1       | 0.850144 | 2.04611"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Catowners.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is this number 1.683 strange or what?\n",
    "\n",
    "It's actually not so bad.  Males are 1's and females are 2's. Maybe it's already intuitive to you, but 1.683 as an average means that 68.3% of this group is female. Why? Imagine we took `sex` and subtracted 1 from its value across everyone. Then effectively the variable is 0 for males and 1 for females, and its average, which must equal $1.683 - 1 = 0.683$, is more clearly the percent female in the subsample.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<font color=blue>Q1: Now ascertain whether crazy cat owners are in fact more likely to be female.</font>\n",
    "\n",
    "You could check this in a variety of ways.  Two that come to mind are that you'll want to check the average sex of cat owners compared to (A) the average sex of everyone in the sample, or compared to (B) the average sex of people in the sample who aren't cat owners.  (There are other comparisons I can think of as well.) Calculate both A and B!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "average_sex_of_everyone = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "Noncatowners = HRSpets.where(HRSpets['numcats'] == 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "average_sex_of_noncatowners = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=blue>Q2: What else is interesting about Eleanor Abernathy? Look at those uppercase letters that come after her name. Can you test the implicit hypothesis about an average characteristic of cat owners other than sex? *(There are multiple ways to do this, but one general idea)*</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=blue>Q3: Now let's tackle a question about health. What do we see when we compare the self-reported health status of cat owners to other people?  What about dog owners?  Do as much as you can that's interesting. *Remember the coding* of `health`, which is that 1 is excellent and 5 is poor.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.711815561959654"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Catowners['health'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}