Notebook7_Pandas_TidyData.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-3311120346f220fa",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "# Tidy data and the Pandas module\n",
    "\n",
    "This notebook accompanies Topic 7, which is about \"tidying data,\" or cleaning up tabular data for analysis purposes. It also introduces one of the most important Python modules for data analysis: Pandas! (not the bear)\n",
    "\n",
    "> _Note_: All parts are included in this single notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-25b42e1a487d70f2",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "# Part 0: Getting the data\n",
    "\n",
    "Before beginning, you'll need to download several files containing the data for the exercises below.\n",
    "\n",
    "**Exercise 0** (ungraded). Run the code cell below to download the data. (This code will check if each dataset has already been downloaded and, if so, will avoid re-downloading it.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-93e719ce80cc83ac",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'table1.csv' is ready!\n",
      "'who3_soln.csv' is ready!\n",
      "'who2_soln.csv' is ready!\n",
      "'table4b.csv' is ready!\n",
      "'iris.csv' is ready!\n",
      "'who.csv' is ready!\n",
      "'table2.csv' is ready!\n",
      "'table4a.csv' is ready!\n",
      "'table3.csv' is ready!\n",
      "\n",
      "(All data appears to be ready.)\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import os\n",
    "import hashlib\n",
    "import io\n",
    "\n",
    "def download(file, url_suffix=None, checksum=None):\n",
    "    if url_suffix is None:\n",
    "        url_suffix = file\n",
    "        \n",
    "    if not os.path.exists(file):\n",
    "        if os.path.exists('.voc'):\n",
    "            url = 'https://cse6040.gatech.edu/datasets/{}'.format(url_suffix)\n",
    "        else:\n",
    "            url = 'https://github.com/cse6040/labs-fa17/raw/master/datasets/{}'.format(url_suffix)\n",
    "        print(\"Downloading: {} ...\".format(url))\n",
    "        r = requests.get(url)\n",
    "        with open(file, 'w', encoding=r.encoding) as f:\n",
    "            f.write(r.text)\n",
    "            \n",
    "    if checksum is not None:\n",
    "        with io.open(file, 'r', encoding='utf-8', errors='replace') as f:\n",
    "            body = f.read()\n",
    "            body_checksum = hashlib.md5(body.encode('utf-8')).hexdigest()\n",
    "            assert body_checksum == checksum, \\\n",
    "                \"Downloaded file '{}' has incorrect checksum: '{}' instead of '{}'\".format(file, body_checksum, checksum)\n",
    "    \n",
    "    print(\"'{}' is ready!\".format(file))\n",
    "    \n",
    "datasets = {'iris.csv': 'd1175c032e1042bec7f974c91e4a65ae',\n",
    "            'table1.csv': '556ffe73363752488d6b41462f5ff3c9',\n",
    "            'table2.csv': '16e04efbc7122e515f7a81a3361e6b87',\n",
    "            'table3.csv': '531d13889f191d6c07c27c3c7ea035ff',\n",
    "            'table4a.csv': '3c0bbecb40c6958df33a1f9aa5629a80',\n",
    "            'table4b.csv': '8484bcdf07b50a7e0932099daa72a93d',\n",
    "            'who.csv': '59fed6bbce66349bf00244b550a93544',\n",
    "            'who2_soln.csv': 'f6d4875feea9d6fca82ae7f87f760f44',\n",
    "            'who3_soln.csv': 'fba14f1e088d871e4407f5f737cfbc06'}\n",
    "\n",
    "for filename, checksum in datasets.items():\n",
    "    download(filename, url_suffix='tidy/{}'.format(filename), checksum=checksum)\n",
    "    \n",
    "print(\"\\n(All data appears to be ready.)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-6bc2a8ff1c3d1927",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "# Part 1: Tidy data \n",
    "\n",
    "The overall topic for this lab is what we'll refer to as representing data _relationally_. The topic of this part is a specific type of relational representation sometimes referred to as the _tidy_ (as opposed to _untidy_ or _messy_) form. The concept of tidy data was developed by [Hadley Wickham](http://hadley.nz/), a statistician and R programming maestro. Much of this lab is based on his tutorial materials (see below).\n",
    "\n",
    "If you know [SQL](https://en.wikipedia.org/wiki/SQL), then you are already familiar with relational data representations. However, we might discuss it a little differently from the way you may have encountered the subject previously. The main reason is our overall goal in the class: to build data _analysis_ pipelines. If our end goal is analysis, then we often want to extract or prepare data in a way that makes analysis easier.\n",
    "\n",
    "You may find it helpful to also refer to the original materials on which this lab is based:\n",
    "\n",
    "* Wickham's R tutorial on making data tidy: http://r4ds.had.co.nz/tidy-data.html\n",
    "* The slides from a talk by Wickham on the concept: http://vita.had.co.nz/papers/tidy-data-pres.pdf\n",
    "* Wickham's more theoretical paper of \"tidy\" vs. \"untidy\" data: http://www.jstatsoft.org/v59/i10/paper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-8ef5c6dbcd8a7067",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "------------------------------------------------------------\n",
    "\n",
    "## What is tidy data?\n",
    "\n",
    "To build your intuition, consider the following data set collected from a survey or study."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-fea9df9be555d16d",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "**Representation 1.** [Two-way contigency table](https://en.wikipedia.org/wiki/Contingency_table).\n",
    "\n",
    "|            | Pregnant | Not pregnant |\n",
    "|-----------:|:--------:|:------------:|\n",
    "| **Male**   |     0    |      5       |\n",
    "| **Female** |     1    |      4       |\n",
    "\n",
    "**Representation 2.** Observation list or \"data frame.\"\n",
    "\n",
    "| Gender  | Pregnant | Count |\n",
    "|:-------:|:--------:|:-----:|\n",
    "| Male    | Yes      | 0     |\n",
    "| Male    | No       | 5     |\n",
    "| Female  | Yes      | 1     |\n",
    "| Female  | No       | 4     |\n",
    "\n",
    "These are two entirely equivalent ways of representing the same data. However, each may be suited to a particular task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-5765b4daff7feec6",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "For instance, Representation 1 is a typical input format for statistical routines that implement Pearson's $\\chi^2$-test, which can check for independence between factors. (Are gender and pregnancy status independent?) By contrast, Representation 2 might be better suited to regression. (Can you predict relative counts from gender and pregnancy status?)\n",
    "\n",
    "While [Representation 1 has its uses](http://simplystatistics.org/2016/02/17/non-tidy-data/), Wickham argues that Representation 2 is often the cleaner and more general way to supply data to a wide variety of statistical analysis and visualization tasks. He refers to Representation 2 as _tidy_ and Representation 1 as _untidy_ or _messy_.\n",
    "\n",
    "> The term \"messy\" is, as Wickham states, not intended to be perjorative since \"messy\" representations may be exactly the right ones for particular analysis tasks, as noted above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-9b90486c7f1c7d12",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "**Definition: Tidy datasets.** More specifically, Wickham defines a tidy data set as one that can be organized into a 2-D table such that\n",
    "\n",
    "1. each column represents a _variable_;\n",
    "2. each row represents an _observation_;\n",
    "3. each entry of the table represents a single _value_, which may come from either categorical (discrete) or continuous spaces."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-463c10417834caae",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "Here is a visual schematic of this definition, taken from [another source](http://r4ds.had.co.nz/images/tidy-1.png):\n",
    "\n",
    "![Wickham's illustration of the definition of tidy](http://r4ds.had.co.nz/images/tidy-1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-7e049cf111aa1eb1",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "This definition appeals to a statistician's intuitive idea of data he or she wishes to analyze. It is also consistent with tasks that seek to establish a functional relationship between some response (output) variable from one or more independent variables.\n",
    "\n",
    "> A computer scientist with a machine learning outlook might refer to columns as _features_ and rows as _data points_, especially when all values are numerical (ordinal or continuous)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-31de85338fed507d",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "**Definition: Tibbles.** Here's one more bit of terminology: if a table is tidy, we will call it a _tidy table_, or _tibble_, for short."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-17babd079ae9f225",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Part 2: Tidy Basics and Pandas\n",
    "\n",
    "In Python, the [Pandas](http://pandas.pydata.org/) module is a convenient way to store tibbles. If you know [R](http://r-project.org), you will see that the design and API of Pandas's data frames derives from [R's data frames](https://stat.ethz.ch/R-manual/R-devel/library/base/html/data.frame.html).\n",
    "\n",
    "In this part of this notebook, let's look at how Pandas works and can help us store Tidy data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-33183cc7b66977e4",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "Consider the famous [Iris data set](https://en.wikipedia.org/wiki/Iris_flower_data_set). It consists of 50 samples from each of three species of Iris (_Iris setosa_, _Iris virginica_, and _Iris versicolor_). Four features were measured from each sample: the lengths and the widths of the [sepals](https://en.wikipedia.org/wiki/Sepal) and [petals](https://en.wikipedia.org/wiki/Petal).\n",
    "\n",
    "The following code uses Pandas to read and represent this data in a Pandas data frame object, stored in a variable named `irises`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Iris data set: 150 rows x 5 columns. ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal length</th>\n",
       "      <th>sepal width</th>\n",
       "      <th>petal length</th>\n",
       "      <th>petal width</th>\n",
       "      <th>species</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>Iris-setosa</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sepal length  sepal width  petal length  petal width      species\n",
       "0           5.1          3.5           1.4          0.2  Iris-setosa\n",
       "1           4.9          3.0           1.4          0.2  Iris-setosa\n",
       "2           4.7          3.2           1.3          0.2  Iris-setosa\n",
       "3           4.6          3.1           1.5          0.2  Iris-setosa\n",
       "4           5.0          3.6           1.4          0.2  Iris-setosa"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Some modules you'll need in this part\n",
    "import pandas as pd\n",
    "from io import StringIO\n",
    "from IPython.display import display\n",
    "\n",
    "# Ignore this line. It will be used later.\n",
    "SAVE_APPLY = getattr(pd.DataFrame, 'apply')\n",
    "\n",
    "irises = pd.read_csv('iris.csv')\n",
    "print(\"=== Iris data set: {} rows x {} columns. ===\".format(irises.shape[0], irises.shape[1]))\n",
    "display (irises.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-2d1633b0021239d4",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "In a Pandas data frame, every column has a name (stored as a string) and all values within the column must have the same primitive type. This fact makes columns different from, for instance, lists.\n",
    "\n",
    "In addition, every row has a special column, called the data frame's _index_. (Try printing `irises.index`.) Any particular index value serves as a name for its row; these index values are usually integers but can be more complex types, like tuples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RangeIndex(start=0, stop=150, step=1)\n"
     ]
    }
   ],
   "source": [
    "print(irises.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-be0336070f5da5de",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "Separate from the index values (row names), you can also refer to rows by their integer offset from the top, where the first row has an offset of 0 and the last row has an offset of `n-1` if the data frame has `n` rows. You'll see that in action in Exercise 1, below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1** (ungraded). Run the following commands to understand what each one does. If it's not obvious, try reading the [Pandas documentation](http://pandas.pydata.org/) or going online to get more information.\n",
    "\n",
    "```python\n",
    "irises.describe()\n",
    "irises['sepal length'].head()\n",
    "irises[[\"sepal length\", \"petal width\"]].head()\n",
    "irises.iloc[5:10]\n",
    "irises[irises[\"sepal length\"] > 5.0]\n",
    "irises[\"sepal length\"].max()\n",
    "irises['species'].unique()\n",
    "irises.sort_values(by=\"sepal length\", ascending=False).head(1)\n",
    "irises.sort_values(by=\"sepal length\", ascending=False).iloc[5:10]\n",
    "irises.sort_values(by=\"sepal length\", ascending=False).loc[5:10]\n",
    "irises['x'] = 3.14\n",
    "irises.rename(columns={'species': 'type'})\n",
    "del irises['x']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-9896c634824d0d5c",
     "locked": false,
     "schema_version": 1,
     "solution": true
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#\n",
    "# YOUR CODE HERE\n",
    "#\n",
    "# irises.describe()\n",
    "# irises['sepal length'].head()\n",
    "# irises[[\"sepal length\", \"petal width\"]].head()\n",
    "# irises.iloc[5:10]\n",
    "# irises[irises[\"sepal length\"] > 5.0]\n",
    "# irises[\"sepal length\"].max()\n",
    "# irises['species'].unique()\n",
    "# irises.sort_values(by=\"sepal length\", ascending=False).head(5)\n",
    "# irises.sort_values(by=\"sepal length\", ascending=False).iloc[:5]\n",
    "# irises.sort_values(by=\"sepal length\", ascending=False).loc[:5]\n",
    "# irises['x'] = 3.14\n",
    "# irises.rename(columns={'species': 'type'})\n",
    "# del irises['x']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-5d6fe7a059bf0059",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Merging data frames: join operations\n",
    "\n",
    "Another useful operation on data frames is [merging](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html).\n",
    "\n",
    "For instance, consider the following two tables, `A` and `B`:\n",
    "\n",
    "| country     | year | cases  |\n",
    "|:------------|-----:|-------:|\n",
    "| Afghanistan | 1999 |    745 |\n",
    "| Brazil      | 1999 |  37737 |\n",
    "| China       | 1999 | 212258 |\n",
    "| Afghanistan | 2000 |   2666 |\n",
    "| Brazil      | 2000 |  80488 |\n",
    "| China       | 2000 | 213766 |\n",
    "\n",
    "| country     | year | population |\n",
    "|:------------|-----:|-----------:|\n",
    "| Afghanistan | 1999 |   19987071 |\n",
    "| Brazil      | 1999 |  172006362 |\n",
    "| China       | 1999 | 1272915272 |\n",
    "| Afghanistan | 2000 |   20595360 |\n",
    "| Brazil      | 2000 |  174504898 |\n",
    "| China       | 2000 | 1280428583 |\n",
    "\n",
    "Suppose we wish to combine these into a single table, `C`:\n",
    "\n",
    "| country     | year | cases  | population |\n",
    "|:------------|-----:|-------:|-----------:|\n",
    "| Afghanistan | 1999 |    745 |   19987071 |\n",
    "| Brazil      | 1999 |  37737 |  172006362 |\n",
    "| China       | 1999 | 212258 | 1272915272 |\n",
    "| Afghanistan | 2000 |   2666 |   20595360 |\n",
    "| Brazil      | 2000 |  80488 |  174504898 |\n",
    "| China       | 2000 | 213766 | 1280428583 |\n",
    "\n",
    "In Pandas, you can perform this merge using the [`.merge()` function](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html):\n",
    "\n",
    "```python\n",
    "C = A.merge (B, on=['country', 'year'])\n",
    "```\n",
    "\n",
    "In this call, the `on=` parameter specifies the list of column names to use to align or \"match\" the two tables, `A` and `B`. By default, `merge()` will only include rows from `A` and `B` where all keys match between the two tables.\n",
    "\n",
    "The following code cell demonstrates this functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-2c24a10b395d5c9e",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== A ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>cases</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>745</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>37737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>212258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>2666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>80488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>213766</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year   cases\n",
       "0  Afghanistan  1999     745\n",
       "1       Brazil  1999   37737\n",
       "2        China  1999  212258\n",
       "3  Afghanistan  2000    2666\n",
       "4       Brazil  2000   80488\n",
       "5        China  2000  213766"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "A_csv = \"\"\"country,year,cases\n",
    "Afghanistan,1999,745\n",
    "Brazil,1999,37737\n",
    "China,1999,212258\n",
    "Afghanistan,2000,2666\n",
    "Brazil,2000,80488\n",
    "China,2000,213766\"\"\"\n",
    "\n",
    "with StringIO(A_csv) as fp:\n",
    "    A = pd.read_csv(fp)\n",
    "print(\"=== A ===\")\n",
    "display(A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-42d7281e768dad73",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== B ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year  population\n",
       "0  Afghanistan  1999    19987071\n",
       "1       Brazil  1999   172006362\n",
       "2        China  1999  1272915272\n",
       "3  Afghanistan  2000    20595360\n",
       "4       Brazil  2000   174504898\n",
       "5        China  2000  1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "B_csv = \"\"\"country,year,population\n",
    "Afghanistan,1999,19987071\n",
    "Brazil,1999,172006362\n",
    "China,1999,1272915272\n",
    "Afghanistan,2000,20595360\n",
    "Brazil,2000,174504898\n",
    "China,2000,1280428583\"\"\"\n",
    "\n",
    "with StringIO(B_csv) as fp:\n",
    "    B = pd.read_csv(fp)\n",
    "print(\"\\n=== B ===\")\n",
    "display(B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-efe43ea0c175ea70",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== C = merge(A, B) ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>cases</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>745</td>\n",
       "      <td>19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>37737</td>\n",
       "      <td>172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>212258</td>\n",
       "      <td>1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>2666</td>\n",
       "      <td>20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>80488</td>\n",
       "      <td>174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>213766</td>\n",
       "      <td>1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year   cases  population\n",
       "0  Afghanistan  1999     745    19987071\n",
       "1       Brazil  1999   37737   172006362\n",
       "2        China  1999  212258  1272915272\n",
       "3  Afghanistan  2000    2666    20595360\n",
       "4       Brazil  2000   80488   174504898\n",
       "5        China  2000  213766  1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "C = A.merge(B, on=['country', 'year'])\n",
    "print(\"\\n=== C = merge(A, B) ===\")\n",
    "display(C)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-f1c3833084dd1a1a",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "**Joins.** This default behavior of keeping only rows that match both input frames is an example of what relational database systems call an _inner-join_ operation. But there are several other types of joins.\n",
    "\n",
    "- _Inner-join (`A`, `B`)_ (default): Keep only rows of `A` and `B` where the on-keys match in both.\n",
    "- _Outer-join (`A`, `B`)_: Keep all rows of both frames, but merge rows when the on-keys match. For non-matches, fill in missing values with not-a-number (`NaN`) values.\n",
    "- _Left-join (`A`, `B`)_: Keep all rows of `A`. Only merge rows of `B` whose on-keys match `A`.\n",
    "- _Right-join (`A`, `B`)_: Keep all rows of `B`. Only merge rows of `A` whose on-keys match `B`.\n",
    "\n",
    "You can use `merge`'s `how=...` parameter, which takes the (string) values, `'inner`', `'outer'`, `'left'`, and `'right'`. Here is an example of an outer join."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-2443839be00e101d",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Apply functions to data frames\n",
    "\n",
    "Another useful primitive is `apply()`, which can apply a function to a data frame or to a series (column of the data frame).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-61240b0ce6cf7a90",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== D ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bug</td>\n",
       "      <td>1</td>\n",
       "      <td>d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>rug</td>\n",
       "      <td>2</td>\n",
       "      <td>d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>lug</td>\n",
       "      <td>3</td>\n",
       "      <td>d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mug</td>\n",
       "      <td>4</td>\n",
       "      <td>d</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     x  y  z\n",
       "0  bug  1  d\n",
       "1  rug  2  d\n",
       "2  lug  3  d\n",
       "3  mug  4  d"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== E ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>w</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>hug</td>\n",
       "      <td>-1</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>smug</td>\n",
       "      <td>-2</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>rug</td>\n",
       "      <td>-3</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>tug</td>\n",
       "      <td>-4</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>bug</td>\n",
       "      <td>1</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      x  y  w\n",
       "0   hug -1  e\n",
       "1  smug -2  e\n",
       "2   rug -3  e\n",
       "3   tug -4  e\n",
       "4   bug  1  e"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== Outer-join (D, E) ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>w</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bug</td>\n",
       "      <td>1</td>\n",
       "      <td>d</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>rug</td>\n",
       "      <td>2</td>\n",
       "      <td>d</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>lug</td>\n",
       "      <td>3</td>\n",
       "      <td>d</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mug</td>\n",
       "      <td>4</td>\n",
       "      <td>d</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>hug</td>\n",
       "      <td>-1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>smug</td>\n",
       "      <td>-2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>rug</td>\n",
       "      <td>-3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>tug</td>\n",
       "      <td>-4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      x  y    z    w\n",
       "0   bug  1    d    e\n",
       "1   rug  2    d  NaN\n",
       "2   lug  3    d  NaN\n",
       "3   mug  4    d  NaN\n",
       "4   hug -1  NaN    e\n",
       "5  smug -2  NaN    e\n",
       "6   rug -3  NaN    e\n",
       "7   tug -4  NaN    e"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== Left-join (D, E) ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>w</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bug</td>\n",
       "      <td>1</td>\n",
       "      <td>d</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>rug</td>\n",
       "      <td>2</td>\n",
       "      <td>d</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>lug</td>\n",
       "      <td>3</td>\n",
       "      <td>d</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mug</td>\n",
       "      <td>4</td>\n",
       "      <td>d</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     x  y  z    w\n",
       "0  bug  1  d    e\n",
       "1  rug  2  d  NaN\n",
       "2  lug  3  d  NaN\n",
       "3  mug  4  d  NaN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== Right-join (D, E) ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>w</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bug</td>\n",
       "      <td>1</td>\n",
       "      <td>d</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>hug</td>\n",
       "      <td>-1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>smug</td>\n",
       "      <td>-2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>rug</td>\n",
       "      <td>-3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>tug</td>\n",
       "      <td>-4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      x  y    z  w\n",
       "0   bug  1    d  e\n",
       "1   hug -1  NaN  e\n",
       "2  smug -2  NaN  e\n",
       "3   rug -3  NaN  e\n",
       "4   tug -4  NaN  e"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== Inner-join (D, E) ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>w</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bug</td>\n",
       "      <td>1</td>\n",
       "      <td>d</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     x  y  z  w\n",
       "0  bug  1  d  e"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with StringIO(\"\"\"x,y,z\n",
    "bug,1,d\n",
    "rug,2,d\n",
    "lug,3,d\n",
    "mug,4,d\"\"\") as fp:\n",
    "    D = pd.read_csv(fp)\n",
    "print(\"=== D ===\")\n",
    "display(D)\n",
    "\n",
    "with StringIO(\"\"\"x,y,w\n",
    "hug,-1,e\n",
    "smug,-2,e\n",
    "rug,-3,e\n",
    "tug,-4,e\n",
    "bug,1,e\"\"\") as fp:\n",
    "    E = pd.read_csv(fp)\n",
    "print(\"\\n=== E ===\")\n",
    "display(E)\n",
    "\n",
    "print(\"\\n=== Outer-join (D, E) ===\")\n",
    "display(D.merge(E, on=['x', 'y'], how='outer'))\n",
    "\n",
    "print(\"\\n=== Left-join (D, E) ===\")\n",
    "display(D.merge(E, on=['x', 'y'], how='left'))\n",
    "\n",
    "print(\"\\n=== Right-join (D, E) ===\")\n",
    "display(D.merge(E, on=['x', 'y'], how='right'))\n",
    "\n",
    "\n",
    "print(\"\\n=== Inner-join (D, E) ===\")\n",
    "display(D.merge(E, on=['x', 'y']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-95019037a3890f29",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>cases</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>745</td>\n",
       "      <td>19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>37737</td>\n",
       "      <td>172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>212258</td>\n",
       "      <td>1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>2666</td>\n",
       "      <td>20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>80488</td>\n",
       "      <td>174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>213766</td>\n",
       "      <td>1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year   cases  population\n",
       "0  Afghanistan  1999     745    19987071\n",
       "1       Brazil  1999   37737   172006362\n",
       "2        China  1999  212258  1272915272\n",
       "3  Afghanistan  2000    2666    20595360\n",
       "4       Brazil  2000   80488   174504898\n",
       "5        China  2000  213766  1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(C)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-dede5baf6e0c627e",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "\n",
    "For instance, suppose we wish to convert the year into an abbrievated two-digit form. The following code will do it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-4f3766e96924dd31",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>cases</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>'99</td>\n",
       "      <td>745</td>\n",
       "      <td>19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>'99</td>\n",
       "      <td>37737</td>\n",
       "      <td>172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>'99</td>\n",
       "      <td>212258</td>\n",
       "      <td>1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>'00</td>\n",
       "      <td>2666</td>\n",
       "      <td>20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>'00</td>\n",
       "      <td>80488</td>\n",
       "      <td>174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>'00</td>\n",
       "      <td>213766</td>\n",
       "      <td>1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country year   cases  population\n",
       "0  Afghanistan  '99     745    19987071\n",
       "1       Brazil  '99   37737   172006362\n",
       "2        China  '99  212258  1272915272\n",
       "3  Afghanistan  '00    2666    20595360\n",
       "4       Brazil  '00   80488   174504898\n",
       "5        China  '00  213766  1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "G = C.copy()\n",
    "G['year'] = G['year'].apply(lambda x: \"'{:02d}\".format(x % 100))\n",
    "display(G)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-9cc2d47e53ba075f",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "**Exercise 2** (2 points). Suppose you wish to compute the prevalence, which is the ratio of cases to the population.\n",
    "\n",
    "The simplest way to do it is as follows:\n",
    "\n",
    "```python\n",
    "    G['prevalence'] = G['cases'] / G['population']\n",
    "```\n",
    "\n",
    "However, for this exercise, try to figure out how to use `apply()` to do it instead. To figure that out, you'll need to consult the documentation for [`apply()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.apply.html) or go online to find some hints.\n",
    "\n",
    "Implement your solution in a function, `calc_prevalence(G)`, which given `G` returns a new copy `H` that has a column named `'prevalence'` holding the correctly computed prevalence values.\n",
    "\n",
    "> Although there is the easy solution above, the purpose of this exercise is to force you to learn more about how `apply()` works, so that you can \"apply\" it in more settings in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "prevalence",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>cases</th>\n",
       "      <th>population</th>\n",
       "      <th>prevalence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>745</td>\n",
       "      <td>19987071</td>\n",
       "      <td>0.000037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>37737</td>\n",
       "      <td>172006362</td>\n",
       "      <td>0.000219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>212258</td>\n",
       "      <td>1272915272</td>\n",
       "      <td>0.000167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>2666</td>\n",
       "      <td>20595360</td>\n",
       "      <td>0.000129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>80488</td>\n",
       "      <td>174504898</td>\n",
       "      <td>0.000461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>213766</td>\n",
       "      <td>1280428583</td>\n",
       "      <td>0.000167</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year   cases  population  prevalence\n",
       "0  Afghanistan  1999     745    19987071    0.000037\n",
       "1       Brazil  1999   37737   172006362    0.000219\n",
       "2        China  1999  212258  1272915272    0.000167\n",
       "3  Afghanistan  2000    2666    20595360    0.000129\n",
       "4       Brazil  2000   80488   174504898    0.000461\n",
       "5        China  2000  213766  1280428583    0.000167"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def calc_prevalence(G):\n",
    "    assert 'cases' in G.columns and 'population' in G.columns\n",
    "#\n",
    "# YOUR CODE HERE\n",
    "    H = G.copy()\n",
    "    H['prevalence'] = H.apply(lambda row: row['cases']/row['population'], axis=1)\n",
    "    return H\n",
    "    \n",
    "calc_prevalence(C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "prevalence_test",
     "locked": true,
     "points": 2,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>cases</th>\n",
       "      <th>population</th>\n",
       "      <th>prevalence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>'99</td>\n",
       "      <td>745</td>\n",
       "      <td>19987071</td>\n",
       "      <td>0.000037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>'99</td>\n",
       "      <td>37737</td>\n",
       "      <td>172006362</td>\n",
       "      <td>0.000219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>'99</td>\n",
       "      <td>212258</td>\n",
       "      <td>1272915272</td>\n",
       "      <td>0.000167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>'00</td>\n",
       "      <td>2666</td>\n",
       "      <td>20595360</td>\n",
       "      <td>0.000129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>'00</td>\n",
       "      <td>80488</td>\n",
       "      <td>174504898</td>\n",
       "      <td>0.000461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>'00</td>\n",
       "      <td>213766</td>\n",
       "      <td>1280428583</td>\n",
       "      <td>0.000167</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country year   cases  population  prevalence\n",
       "0  Afghanistan  '99     745    19987071    0.000037\n",
       "1       Brazil  '99   37737   172006362    0.000219\n",
       "2        China  '99  212258  1272915272    0.000167\n",
       "3  Afghanistan  '00    2666    20595360    0.000129\n",
       "4       Brazil  '00   80488   174504898    0.000461\n",
       "5        China  '00  213766  1280428583    0.000167"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prevalance values seem correct. But did you use `apply()?` Let's see...\n",
      "You used `apply()`. You may have even used it as intended.\n",
      "\n",
      "(Passed!)\n"
     ]
    }
   ],
   "source": [
    "# Test cell: `prevalence_test`\n",
    "\n",
    "H = calc_prevalence(G)\n",
    "display(H) # Displayed `H` should have a 'prevalence' column\n",
    "\n",
    "Easy_prevalence_method = G['cases'] / G['population']\n",
    "assert (H['prevalence'] == Easy_prevalence_method).all(), \"One or more prevalence values is incorrect.\"\n",
    "\n",
    "print(\"Prevalance values seem correct. But did you use `apply()?` Let's see...\")\n",
    "\n",
    "# Tests that you actually used `apply()` in your function:\n",
    "def apply_fail():\n",
    "    raise ValueError(\"Did you really use apply?\")\n",
    "setattr(pd.DataFrame, 'apply', apply_fail)\n",
    "try:\n",
    "    calc_prevalence(G)\n",
    "except (ValueError, TypeError):\n",
    "    setattr(pd.DataFrame, 'apply', SAVE_APPLY)\n",
    "    print(\"You used `apply()`. You may have even used it as intended.\")\n",
    "else:\n",
    "    setattr(pd.DataFrame, 'apply', SAVE_APPLY)\n",
    "    assert False, \"Are you sure you used `apply()`?\"\n",
    "    \n",
    "\n",
    "print(\"\\n(Passed!)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-c4c988db6332ba17",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Part 3 : Tibbles and Bits \n",
    "\n",
    "Now let's start creating and manipulating tibbles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-0e93196eb8c0f7a8",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd  # The suggested idiom\n",
    "from io import StringIO\n",
    "\n",
    "from IPython.display import display # For pretty-printing data frames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3** (3 points). Write a function, `canonicalize_tibble(X)`, that, given a tibble `X`, returns a new copy `Y` of `X` in _canonical order_. We say `Y` is in canonical order if it has the following properties.\n",
    "\n",
    "1. The variables appear in sorted order by name, ascending from left to right.\n",
    "2. The rows appear in lexicographically sorted order by variable, ascending from top to bottom.\n",
    "3. The row labels (`Y.index`) go from 0 to `n-1`, where `n` is the number of observations.\n",
    "\n",
    "For instance, here is a **non-canonical tibble** ...\n",
    "\n",
    "|   |  c  | a | b |\n",
    "|:-:|:---:|:-:|:-:|\n",
    "| 2 | hat | x | 1 |\n",
    "| 0 | rat | y | 4 |\n",
    "| 3 | cat | x | 2 |\n",
    "| 1 | bat | x | 2 |\n",
    "\n",
    "\n",
    "... and here is its **canonical counterpart.**\n",
    "\n",
    "|   | a | b |  c  |\n",
    "|:-:|:-:|:-:|:---:|\n",
    "| 0 | x | 1 | hat |\n",
    "| 1 | x | 2 | bat |\n",
    "| 2 | x | 2 | cat |\n",
    "| 3 | y | 4 | rat |\n",
    "\n",
    "A partial solution appears below, which ensures that Property 1 above holds. Complete the solution to ensure Properties 2 and 3 hold. Feel free to consult the [Pandas API](http://pandas.pydata.org/pandas-docs/stable/api.html).\n",
    "\n",
    "> **Hint**. For Property 3, you may find `reset_index()` handy: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.reset_index.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true,
    "nbgrader": {
     "grade": false,
     "grade_id": "canonicalize_tibble",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def canonicalize_tibble(X):\n",
    "    # Enforce Property 1:\n",
    "    var_names = sorted(X.columns)\n",
    "    Y = X[var_names].copy()\n",
    "\n",
    "#\n",
    "# YOUR CODE HERE\n",
    "    Y = Y.sort_values(by=var_names)   #['a', 'b', 'c'])\n",
    "    Y = Y.reset_index(drop = True)\n",
    "    return Y\n",
    "\n",
    "# canonicalize_tibble(canonical_in)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "canonicalize_tibble_test",
     "locked": true,
     "points": 3,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Input ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>c</th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>hat</td>\n",
       "      <td>x</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>rat</td>\n",
       "      <td>y</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cat</td>\n",
       "      <td>x</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>bat</td>\n",
       "      <td>x</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     c  a  b\n",
       "2  hat  x  1\n",
       "0  rat  y  4\n",
       "3  cat  x  2\n",
       "1  bat  x  2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== True solution ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>x</td>\n",
       "      <td>1</td>\n",
       "      <td>hat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>x</td>\n",
       "      <td>2</td>\n",
       "      <td>bat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>x</td>\n",
       "      <td>2</td>\n",
       "      <td>cat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>y</td>\n",
       "      <td>4</td>\n",
       "      <td>rat</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   a  b    c\n",
       "0  x  1  hat\n",
       "1  x  2  bat\n",
       "2  x  2  cat\n",
       "3  y  4  rat"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== Your computed solution ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>x</td>\n",
       "      <td>1</td>\n",
       "      <td>hat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>x</td>\n",
       "      <td>2</td>\n",
       "      <td>bat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>x</td>\n",
       "      <td>2</td>\n",
       "      <td>cat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>y</td>\n",
       "      <td>4</td>\n",
       "      <td>rat</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   a  b    c\n",
       "0  x  1  hat\n",
       "1  x  2  bat\n",
       "2  x  2  cat\n",
       "3  y  4  rat"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== Matches? (Should be all True) ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      a     b     c\n",
       "0  True  True  True\n",
       "1  True  True  True\n",
       "2  True  True  True\n",
       "3  True  True  True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "(Passed.)\n"
     ]
    }
   ],
   "source": [
    "# Test: `canonicalize_tibble_test`\n",
    "\n",
    "# Test input\n",
    "canonical_in_csv = \"\"\",c,a,b\n",
    "2,hat,x,1\n",
    "0,rat,y,4\n",
    "3,cat,x,2\n",
    "1,bat,x,2\"\"\"\n",
    "\n",
    "with StringIO(canonical_in_csv) as fp:\n",
    "    canonical_in = pd.read_csv(fp, index_col=0)\n",
    "print(\"=== Input ===\")\n",
    "display(canonical_in)\n",
    "print(\"\")\n",
    "    \n",
    "# Test output solution\n",
    "canonical_soln_csv = \"\"\",a,b,c\n",
    "0,x,1,hat\n",
    "1,x,2,bat\n",
    "2,x,2,cat\n",
    "3,y,4,rat\"\"\"\n",
    "\n",
    "with StringIO(canonical_soln_csv) as fp:\n",
    "    canonical_soln = pd.read_csv(fp, index_col=0)\n",
    "print(\"=== True solution ===\")\n",
    "display(canonical_soln)\n",
    "print(\"\")\n",
    "\n",
    "canonical_out = canonicalize_tibble(canonical_in)\n",
    "print(\"=== Your computed solution ===\")\n",
    "display(canonical_out)\n",
    "print(\"\")\n",
    "\n",
    "canonical_matches = (canonical_out == canonical_soln)\n",
    "print(\"=== Matches? (Should be all True) ===\")\n",
    "display(canonical_matches)\n",
    "assert canonical_matches.all().all()\n",
    "\n",
    "print (\"\\n(Passed.)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-b10e9af4a4d924b7",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "**Exercise 4** (1 point). Write a function, `tibbles_are_equivalent(A, B)` to determine if two tibbles, `A` and `B`, are equivalent. \"Equivalent\" means that `A` and `B` have identical variables and observations, up to permutations. If `A` and `B` are equivalent, then the function should return `True`. Otherwise, it should return `False`.\n",
    "\n",
    "The last condition, \"up to permutations,\" means that the variables and observations might not appear in the table in the same order. For example, the following two tibbles are equivalent:\n",
    "\n",
    "\n",
    "| a | b |  c  |\n",
    "|:-:|:-:|:---:|\n",
    "| x | 1 | hat |\n",
    "| y | 2 | cat |\n",
    "| z | 3 | bat |\n",
    "| w | 4 | rat |\n",
    "\n",
    "| b |  c  | a |\n",
    "|:-:|:---:|:-:|\n",
    "| 2 | cat | y |\n",
    "| 3 | bat | z |\n",
    "| 1 | hat | x |\n",
    "| 4 | rat | w |\n",
    "\n",
    "By contrast, the following table would not be equivalent to either of the above tibbles.\n",
    "\n",
    "| a | b |  c  |\n",
    "|:-:|:-:|:---:|\n",
    "| 2 | y | cat |\n",
    "| 3 | z | bat |\n",
    "| 1 | x | hat |\n",
    "| 4 | w | rat |\n",
    "\n",
    "> **Note**: Unlike Pandas data frames, tibbles conceptually do not have row labels. So you should ignore row labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true,
    "nbgrader": {
     "grade": false,
     "grade_id": "tibbles_are_equivalent",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def tibbles_are_equivalent(A, B):\n",
    "    \"\"\"Given two tidy tables ('tibbles'), returns True iff they are\n",
    "    equivalent.\n",
    "    \"\"\"\n",
    "#\n",
    "# YOUR CODE HERE\n",
    "    for col in A.columns:\n",
    "        if sorted(A[col]) == sorted(B[col]):\n",
    "            continue\n",
    "        else:\n",
    "            return False\n",
    "    return True\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "tibbles_are_equivalent_test",
     "locked": true,
     "points": 1,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Tibble A ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>x</td>\n",
       "      <td>1</td>\n",
       "      <td>hat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>y</td>\n",
       "      <td>2</td>\n",
       "      <td>cat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>z</td>\n",
       "      <td>3</td>\n",
       "      <td>bat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>w</td>\n",
       "      <td>4</td>\n",
       "      <td>rat</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   a  b    c\n",
       "0  x  1  hat\n",
       "1  y  2  cat\n",
       "2  z  3  bat\n",
       "3  w  4  rat"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Tibble B == A ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>a</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>hat</td>\n",
       "      <td>x</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>cat</td>\n",
       "      <td>y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>rat</td>\n",
       "      <td>w</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>bat</td>\n",
       "      <td>z</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   b    c  a\n",
       "0  1  hat  x\n",
       "1  2  cat  y\n",
       "3  4  rat  w\n",
       "2  3  bat  z"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Tibble C != A ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>a</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>x</td>\n",
       "      <td>1</td>\n",
       "      <td>hat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>y</td>\n",
       "      <td>2</td>\n",
       "      <td>cat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>z</td>\n",
       "      <td>3</td>\n",
       "      <td>bat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>w</td>\n",
       "      <td>4</td>\n",
       "      <td>rat</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   b  c    a\n",
       "0  x  1  hat\n",
       "1  y  2  cat\n",
       "2  z  3  bat\n",
       "3  w  4  rat"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "(Passed.)\n"
     ]
    }
   ],
   "source": [
    "# Test: `tibble_are_equivalent_test`\n",
    "\n",
    "A = pd.DataFrame(columns=['a', 'b', 'c'],\n",
    "                 data=list(zip (['x', 'y', 'z', 'w'],\n",
    "                                [1, 2, 3, 4],\n",
    "                                ['hat', 'cat', 'bat', 'rat'])))\n",
    "print(\"=== Tibble A ===\")\n",
    "display(A)\n",
    "\n",
    "# Permute rows and columns, preserving equivalence\n",
    "import random\n",
    "\n",
    "obs_ind_orig = list(range(A.shape[0]))\n",
    "var_names = list(A.columns)\n",
    "\n",
    "obs_ind = obs_ind_orig.copy()\n",
    "while obs_ind == obs_ind_orig:\n",
    "    random.shuffle(obs_ind)\n",
    "    \n",
    "while var_names == list(A.columns):\n",
    "    random.shuffle(var_names)\n",
    "\n",
    "B = A[var_names].copy()\n",
    "B = B.iloc[obs_ind]\n",
    "\n",
    "print (\"=== Tibble B == A ===\")\n",
    "display(B)\n",
    "\n",
    "print (\"=== Tibble C != A ===\")\n",
    "C = A.copy()\n",
    "C.columns = var_names\n",
    "display(C)\n",
    "\n",
    "assert tibbles_are_equivalent(A, B)\n",
    "assert not tibbles_are_equivalent(A, C)\n",
    "assert not tibbles_are_equivalent(B, C)\n",
    "\n",
    "print (\"\\n(Passed.)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-599e914efcabe2ad",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "# Basic tidying transformations: Melting and casting\n",
    "\n",
    "Given a data set and a target set of variables, there are at least two common issues that require tidying.\n",
    "## Melting\n",
    "First, values often appear as columns. Table 4a is an example. To tidy up, you want to turn columns into rows:\n",
    "\n",
    "![Gather example](http://r4ds.had.co.nz/images/tidy-9.png)\n",
    "\n",
    "Because this operation takes columns into rows, making a \"fat\" table more tall and skinny, it is sometimes called _melting_.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-148ef4b7030d8566",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "To melt the table, you need to do the following.\n",
    "\n",
    "1. Extract the _column values_ into a new variable. In this case, columns `\"1999\"` and `\"2000\"` of `table4` need to become the values of the variable, `\"year\"`.\n",
    "2. Convert the values associated with the column values into a new variable as well. In this case, the values formerly in columns `\"1999\"` and `\"2000\"` become the values of the `\"cases\"` variable.\n",
    "\n",
    "In the context of a melt, let's also refer to `\"year\"` as the new _key_ variable and `\"cases\"` as the new _value_ variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-ba520d87d798d903",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "**Exercise 5** (4 points). Implement the melt operation as a function,\n",
    "\n",
    "```python\n",
    "    def melt(df, col_vals, key, value):\n",
    "        ...\n",
    "```\n",
    "\n",
    "It should take the following arguments:\n",
    "- `df`: the input data frame, e.g., `table4` in the example above;\n",
    "- `col_vals`: a list of the column names that will serve as values;\n",
    "- `key`: name of the new variable, e.g., `year` in the example above;\n",
    "- `value`: name of the column to hold the values.\n",
    "\n",
    "> You may need to refer to the Pandas documentation to figure out how to create and manipulate tables. The bits related to [indexing](http://pandas.pydata.org/pandas-docs/stable/indexing.html) and [merging](http://pandas.pydata.org/pandas-docs/stable/merging.html) may be especially helpful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true,
    "nbgrader": {
     "grade": false,
     "grade_id": "melt",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def melt(df, col_vals, key, value):\n",
    "    assert type(df) is pd.DataFrame\n",
    "#\n",
    "# YOUR CODE HERE\n",
    "    index = [df.columns[0]]\n",
    "    A = pd.melt(df, id_vars=index, value_vars=col_vals, var_name=key, value_name=value).sort_values(by=index)\n",
    "    return A\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "melt_test",
     "locked": true,
     "points": 4,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== table4a ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>1999</th>\n",
       "      <th>2000</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>745</td>\n",
       "      <td>2666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>37737</td>\n",
       "      <td>80488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>212258</td>\n",
       "      <td>213766</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country    1999    2000\n",
       "0  Afghanistan     745    2666\n",
       "1       Brazil   37737   80488\n",
       "2        China  212258  213766"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== melt(table4a) ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>cases</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>745</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>2666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>37737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>80488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>212258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>213766</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year   cases\n",
       "0  Afghanistan  1999     745\n",
       "3  Afghanistan  2000    2666\n",
       "1       Brazil  1999   37737\n",
       "4       Brazil  2000   80488\n",
       "2        China  1999  212258\n",
       "5        China  2000  213766"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== table4b ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>1999</th>\n",
       "      <th>2000</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>19987071</td>\n",
       "      <td>20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>172006362</td>\n",
       "      <td>174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>1272915272</td>\n",
       "      <td>1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country        1999        2000\n",
       "0  Afghanistan    19987071    20595360\n",
       "1       Brazil   172006362   174504898\n",
       "2        China  1272915272  1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== melt(table4b) ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year  population\n",
       "0  Afghanistan  1999    19987071\n",
       "3  Afghanistan  2000    20595360\n",
       "1       Brazil  1999   172006362\n",
       "4       Brazil  2000   174504898\n",
       "2        China  1999  1272915272\n",
       "5        China  2000  1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== inner-join(melt(table4a), melt (table4b)) ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>cases</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>745</td>\n",
       "      <td>19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>2666</td>\n",
       "      <td>20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>37737</td>\n",
       "      <td>172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>80488</td>\n",
       "      <td>174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>212258</td>\n",
       "      <td>1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>213766</td>\n",
       "      <td>1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year   cases  population\n",
       "0  Afghanistan  1999     745    19987071\n",
       "1  Afghanistan  2000    2666    20595360\n",
       "2       Brazil  1999   37737   172006362\n",
       "3       Brazil  2000   80488   174504898\n",
       "4        China  1999  212258  1272915272\n",
       "5        China  2000  213766  1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== table1 (target solution) ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>cases</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>745</td>\n",
       "      <td>19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>2666</td>\n",
       "      <td>20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>37737</td>\n",
       "      <td>172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>80488</td>\n",
       "      <td>174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>212258</td>\n",
       "      <td>1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>213766</td>\n",
       "      <td>1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year   cases  population\n",
       "0  Afghanistan  1999     745    19987071\n",
       "1  Afghanistan  2000    2666    20595360\n",
       "2       Brazil  1999   37737   172006362\n",
       "3       Brazil  2000   80488   174504898\n",
       "4        China  1999  212258  1272915272\n",
       "5        China  2000  213766  1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "(Passed.)\n"
     ]
    }
   ],
   "source": [
    "# Test: `melt_test`\n",
    "\n",
    "table4a = pd.read_csv('table4a.csv')\n",
    "print(\"\\n=== table4a ===\")\n",
    "display(table4a)\n",
    "\n",
    "m_4a = melt(table4a, col_vals=['1999', '2000'], key='year', value='cases')\n",
    "print(\"=== melt(table4a) ===\")\n",
    "display(m_4a)\n",
    "\n",
    "table4b = pd.read_csv('table4b.csv')\n",
    "print(\"\\n=== table4b ===\")\n",
    "display(table4b)\n",
    "\n",
    "m_4b = melt(table4b, col_vals=['1999', '2000'], key='year', value='population')\n",
    "print(\"=== melt(table4b) ===\")\n",
    "display(m_4b)\n",
    "\n",
    "m_4 = pd.merge(m_4a, m_4b, on=['country', 'year'])\n",
    "print (\"\\n=== inner-join(melt(table4a), melt (table4b)) ===\")\n",
    "display(m_4)\n",
    "\n",
    "m_4['year'] = m_4['year'].apply (int)\n",
    "\n",
    "table1 = pd.read_csv('table1.csv')\n",
    "print (\"=== table1 (target solution) ===\")\n",
    "display(table1)\n",
    "assert tibbles_are_equivalent(table1, m_4)\n",
    "print (\"\\n(Passed.)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Casting\n",
    "The second most common issue is that an observation might be split across multiple rows. Table 2 is an example. To tidy up, you want to merge rows:\n",
    "\n",
    "![Spread example](http://r4ds.had.co.nz/images/tidy-8.png)\n",
    "\n",
    "Because this operation is the moral opposite of melting, and \"rebuilds\" observations from parts, it is sometimes called _casting_.\n",
    "\n",
    "> Melting and casting are Wickham's terms from [his original paper on tidying data](http://www.jstatsoft.org/v59/i10/paper). In his more recent writing, [on which this tutorial is based](http://r4ds.had.co.nz/tidy-data.html), he refers to the same operation as _gathering_. Again, this term comes from Wickham's original paper, whereas his more recent summaries use the term _spreading_.\n",
    "\n",
    "The signature of a cast is similar to that of melt. However, you only need to know the `key`, which is column of the input table containing new variable names, and the `value`, which is the column containing corresponding values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6** (4 points). Implement a function to cast a data frame into a tibble, given a key column containing new variable names and a value column containing the corresponding cells.\n",
    "\n",
    "We've given you a partial solution that\n",
    "\n",
    "- verifies that the given `key` and `value` columns are actual columns of the input data frame;\n",
    "- computes the list of columns, `fixed_vars`, that should remain unchanged; and\n",
    "- initializes and empty tibble.\n",
    "\n",
    "Observe that we are asking your `cast()` to accept an optional parameter, `join_how`, that may take the values `'outer'` or `'inner'` (with `'outer'` as the default). Why do you need such a parameter?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true,
    "nbgrader": {
     "grade": false,
     "grade_id": "cast",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def cast(df, key, value, join_how='outer'):\n",
    "    \"\"\"Casts the input data frame into a tibble,\n",
    "    given the key column and value column.\n",
    "    \"\"\"\n",
    "    assert type(df) is pd.DataFrame\n",
    "    assert key in df.columns and value in df.columns\n",
    "    assert join_how in ['outer', 'inner']\n",
    "    \n",
    "    fixed_vars = df.columns.difference([key, value])\n",
    "    tibble = pd.DataFrame(columns=fixed_vars) # empty frame\n",
    "    \n",
    "#\n",
    "# YOUR CODE HERE\n",
    "    index1 = df.columns[0]\n",
    "    index2 = df.columns[1]\n",
    "    tibble = table2.pivot_table(values=value, index=[index1, index2], columns=key).reset_index()\n",
    "    return tibble\n",
    "\n",
    "# cast(table2, 'type', 'count', join_how='outer')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "cast_test",
     "locked": true,
     "points": 4,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== table2 ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>type</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>cases</td>\n",
       "      <td>745</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>population</td>\n",
       "      <td>19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>cases</td>\n",
       "      <td>2666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>population</td>\n",
       "      <td>20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>cases</td>\n",
       "      <td>37737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>population</td>\n",
       "      <td>172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>cases</td>\n",
       "      <td>80488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>population</td>\n",
       "      <td>174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>cases</td>\n",
       "      <td>212258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>population</td>\n",
       "      <td>1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>cases</td>\n",
       "      <td>213766</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>population</td>\n",
       "      <td>1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        country  year        type       count\n",
       "0   Afghanistan  1999       cases         745\n",
       "1   Afghanistan  1999  population    19987071\n",
       "2   Afghanistan  2000       cases        2666\n",
       "3   Afghanistan  2000  population    20595360\n",
       "4        Brazil  1999       cases       37737\n",
       "5        Brazil  1999  population   172006362\n",
       "6        Brazil  2000       cases       80488\n",
       "7        Brazil  2000  population   174504898\n",
       "8         China  1999       cases      212258\n",
       "9         China  1999  population  1272915272\n",
       "10        China  2000       cases      213766\n",
       "11        China  2000  population  1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== tibble2 = cast (table2, \"type\", \"count\") ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>type</th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>cases</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>745</td>\n",
       "      <td>19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>2666</td>\n",
       "      <td>20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>37737</td>\n",
       "      <td>172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>80488</td>\n",
       "      <td>174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>212258</td>\n",
       "      <td>1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>213766</td>\n",
       "      <td>1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "type      country  year   cases  population\n",
       "0     Afghanistan  1999     745    19987071\n",
       "1     Afghanistan  2000    2666    20595360\n",
       "2          Brazil  1999   37737   172006362\n",
       "3          Brazil  2000   80488   174504898\n",
       "4           China  1999  212258  1272915272\n",
       "5           China  2000  213766  1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "(Passed.)\n"
     ]
    }
   ],
   "source": [
    "# Test: `cast_test`\n",
    "\n",
    "table2 = pd.read_csv('table2.csv')\n",
    "print('=== table2 ===')\n",
    "display(table2)\n",
    "\n",
    "print('\\n=== tibble2 = cast (table2, \"type\", \"count\") ===')\n",
    "tibble2 = cast(table2, 'type', 'count')\n",
    "display(tibble2)\n",
    "\n",
    "assert tibbles_are_equivalent(table1, tibble2)\n",
    "print('\\n(Passed.)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-bf3178d2338c5010",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "## Separating variables\n",
    "\n",
    "Consider the following table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-da10ca3481ababeb",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>745/19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>2666/20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>37737/172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>80488/174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>212258/1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>213766/1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year               rate\n",
       "0  Afghanistan  1999       745/19987071\n",
       "1  Afghanistan  2000      2666/20595360\n",
       "2       Brazil  1999    37737/172006362\n",
       "3       Brazil  2000    80488/174504898\n",
       "4        China  1999  212258/1272915272\n",
       "5        China  2000  213766/1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "table3 = pd.read_csv('table3.csv')\n",
    "display(table3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-8d2cead7200f279e",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "In this table, the `rate` variable combines what had previously been the `cases` and `population` data. This example is an instance in which we might want to _separate_ a column into two variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-a2b68ca72f4469cc",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "**Exercise 6** (3 points). Write a function that takes a data frame (`df`) and separates an existing column (`key`) into new variables (given by the list of new variable names, `into`).\n",
    "\n",
    "How will the separation happen? The caller should provide a function, `splitter(x)`, that given a value returns a _list_ containing the components. Observe that the partial solution below defines a default splitter, which uses the regular expression, `(\\d+\\.?\\d+)`, to find all integer or floating-point values in a string input `x`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "nbgrader": {
     "grade": false,
     "grade_id": "separate",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def default_splitter(text):\n",
    "    \"\"\"Searches the given spring for all integer and floating-point\n",
    "    values, returning them as a list _of strings_.\n",
    "    E.g., the call\n",
    "      default_splitter('Give me $10.52 in exchange for 91 kitten stickers.')\n",
    "    will return ['10.52', '91'].\n",
    "    \"\"\"\n",
    "    fields = re.findall('(\\d+\\.?\\d+)', text)\n",
    "    return fields\n",
    "\n",
    "def separate(df, key, into, splitter=default_splitter):\n",
    "    \"\"\"Given a data frame, separates one of its columns, the key,\n",
    "    into new variables.\n",
    "    \"\"\"\n",
    "    assert type(df) is pd.DataFrame\n",
    "    assert key in df.columns\n",
    "    \n",
    "    # Hint: http://stackoverflow.com/questions/16236684/apply-pandas-function-to-column-to-create-multiple-new-columns\n",
    "\n",
    "#\n",
    "# YOUR CODE HERE\n",
    "#\n",
    "    tibble = df.copy()\n",
    "    num_list = tibble[key].apply(default_splitter)\n",
    "    tibble[into[0]] = [num_list[i][0] for i in range(len(num_list))]\n",
    "    tibble[into[1]] = [num_list[i][1] for i in range(len(num_list))]\n",
    "    tibble = tibble.drop(key, axis=1)\n",
    "    return tibble\n",
    "    \n",
    "#separate(table3, key='rate', into=['cases', 'population'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "separate_test",
     "locked": true,
     "points": 3,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Recall: table3 ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>745/19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>2666/20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>37737/172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>80488/174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>212258/1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>213766/1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year               rate\n",
       "0  Afghanistan  1999       745/19987071\n",
       "1  Afghanistan  2000      2666/20595360\n",
       "2       Brazil  1999    37737/172006362\n",
       "3       Brazil  2000    80488/174504898\n",
       "4        China  1999  212258/1272915272\n",
       "5        China  2000  213766/1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== tibble3 = separate (table3, ...) ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>cases</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>745</td>\n",
       "      <td>19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>2666</td>\n",
       "      <td>20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>37737</td>\n",
       "      <td>172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>80488</td>\n",
       "      <td>174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>212258</td>\n",
       "      <td>1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>213766</td>\n",
       "      <td>1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year   cases  population\n",
       "0  Afghanistan  1999     745    19987071\n",
       "1  Afghanistan  2000    2666    20595360\n",
       "2       Brazil  1999   37737   172006362\n",
       "3       Brazil  2000   80488   174504898\n",
       "4        China  1999  212258  1272915272\n",
       "5        China  2000  213766  1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "(Passed.)\n"
     ]
    }
   ],
   "source": [
    "# Test: `separate_test`\n",
    "\n",
    "print(\"=== Recall: table3 ===\")\n",
    "display(table3)\n",
    "\n",
    "tibble3 = separate(table3, key='rate', into=['cases', 'population'])\n",
    "print(\"\\n=== tibble3 = separate (table3, ...) ===\")\n",
    "display(tibble3)\n",
    "\n",
    "assert 'cases' in tibble3.columns\n",
    "assert 'population' in tibble3.columns\n",
    "assert 'rate' not in tibble3.columns\n",
    "\n",
    "tibble3['cases'] = tibble3['cases'].apply(int)\n",
    "tibble3['population'] = tibble3['population'].apply(int)\n",
    "\n",
    "assert tibbles_are_equivalent(tibble3, table1)\n",
    "print(\"\\n(Passed.)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-db2c5eb2a87177d1",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "**Exercise 7** (2 points). Implement the inverse of separate, which is `unite`. This function should take a data frame (`df`), the set of columns to combine (`cols`), the name of the new column (`new_var`), and a function that takes the subset of the `cols` variables from a single observation. It should return a new value for that observation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true,
    "nbgrader": {
     "grade": false,
     "grade_id": "unite",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def str_join_elements(x, sep=\"\"):\n",
    "    assert type(sep) is str\n",
    "    return sep.join([str(xi) for xi in x])\n",
    "\n",
    "def unite(df, cols, new_var, combine=str_join_elements):\n",
    "    # Hint: http://stackoverflow.com/questions/13331698/how-to-apply-a-function-to-two-columns-of-pandas-dataframe\n",
    "#\n",
    "# YOUR CODE HERE\n",
    "#\n",
    "    tibble = df.copy()\n",
    "    tibble['rate'] = tibble[cols].apply(combine, axis=1)\n",
    "    tibble = tibble.drop(cols, axis=1)    \n",
    "    return tibble\n",
    "\n",
    "# unite(tibble3, ['cases', 'population'], 'rate', combine=lambda x: str_join_elements(x, \"/\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "unite_test",
     "locked": true,
     "points": 2,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1999</td>\n",
       "      <td>745/19987071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2000</td>\n",
       "      <td>2666/20595360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>1999</td>\n",
       "      <td>37737/172006362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Brazil</td>\n",
       "      <td>2000</td>\n",
       "      <td>80488/174504898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>China</td>\n",
       "      <td>1999</td>\n",
       "      <td>212258/1272915272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>China</td>\n",
       "      <td>2000</td>\n",
       "      <td>213766/1280428583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year               rate\n",
       "0  Afghanistan  1999       745/19987071\n",
       "1  Afghanistan  2000      2666/20595360\n",
       "2       Brazil  1999    37737/172006362\n",
       "3       Brazil  2000    80488/174504898\n",
       "4        China  1999  212258/1272915272\n",
       "5        China  2000  213766/1280428583"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "(Passed.)\n"
     ]
    }
   ],
   "source": [
    "# Test: `unite_test`\n",
    "\n",
    "table3_again = unite(tibble3, ['cases', 'population'], 'rate',\n",
    "                     combine=lambda x: str_join_elements(x, \"/\"))\n",
    "display(table3_again)\n",
    "assert tibbles_are_equivalent(table3, table3_again)\n",
    "\n",
    "print(\"\\n(Passed.)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-ea8e4973d3e8b091",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "\n",
    "# Putting it all together #\n",
    "\n",
    "Let's use primitives to tidy up the original WHO TB data set. First, here is the raw data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-37ccd3c020f69268",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== WHO TB data set: 7240 rows x 60 columns ===\n",
      "Column names: Index(['country', 'iso2', 'iso3', 'year', 'new_sp_m014', 'new_sp_m1524',\n",
      "       'new_sp_m2534', 'new_sp_m3544', 'new_sp_m4554', 'new_sp_m5564',\n",
      "       'new_sp_m65', 'new_sp_f014', 'new_sp_f1524', 'new_sp_f2534',\n",
      "       'new_sp_f3544', 'new_sp_f4554', 'new_sp_f5564', 'new_sp_f65',\n",
      "       'new_sn_m014', 'new_sn_m1524', 'new_sn_m2534', 'new_sn_m3544',\n",
      "       'new_sn_m4554', 'new_sn_m5564', 'new_sn_m65', 'new_sn_f014',\n",
      "       'new_sn_f1524', 'new_sn_f2534', 'new_sn_f3544', 'new_sn_f4554',\n",
      "       'new_sn_f5564', 'new_sn_f65', 'new_ep_m014', 'new_ep_m1524',\n",
      "       'new_ep_m2534', 'new_ep_m3544', 'new_ep_m4554', 'new_ep_m5564',\n",
      "       'new_ep_m65', 'new_ep_f014', 'new_ep_f1524', 'new_ep_f2534',\n",
      "       'new_ep_f3544', 'new_ep_f4554', 'new_ep_f5564', 'new_ep_f65',\n",
      "       'newrel_m014', 'newrel_m1524', 'newrel_m2534', 'newrel_m3544',\n",
      "       'newrel_m4554', 'newrel_m5564', 'newrel_m65', 'newrel_f014',\n",
      "       'newrel_f1524', 'newrel_f2534', 'newrel_f3544', 'newrel_f4554',\n",
      "       'newrel_f5564', 'newrel_f65'],\n",
      "      dtype='object')\n",
      "\n",
      "=== A few randomly selected rows ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>iso2</th>\n",
       "      <th>iso3</th>\n",
       "      <th>year</th>\n",
       "      <th>new_sp_m014</th>\n",
       "      <th>new_sp_m1524</th>\n",
       "      <th>new_sp_m2534</th>\n",
       "      <th>new_sp_m3544</th>\n",
       "      <th>new_sp_m4554</th>\n",
       "      <th>new_sp_m5564</th>\n",
       "      <th>...</th>\n",
       "      <th>newrel_m4554</th>\n",
       "      <th>newrel_m5564</th>\n",
       "      <th>newrel_m65</th>\n",
       "      <th>newrel_f014</th>\n",
       "      <th>newrel_f1524</th>\n",
       "      <th>newrel_f2534</th>\n",
       "      <th>newrel_f3544</th>\n",
       "      <th>newrel_f4554</th>\n",
       "      <th>newrel_f5564</th>\n",
       "      <th>newrel_f65</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1692</th>\n",
       "      <td>Cote d'Ivoire</td>\n",
       "      <td>CI</td>\n",
       "      <td>CIV</td>\n",
       "      <td>2002</td>\n",
       "      <td>102.0</td>\n",
       "      <td>1271.0</td>\n",
       "      <td>2194.0</td>\n",
       "      <td>1490.0</td>\n",
       "      <td>833.0</td>\n",
       "      <td>385.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6021</th>\n",
       "      <td>Sri Lanka</td>\n",
       "      <td>LK</td>\n",
       "      <td>LKA</td>\n",
       "      <td>1997</td>\n",
       "      <td>11.0</td>\n",
       "      <td>215.0</td>\n",
       "      <td>390.0</td>\n",
       "      <td>596.0</td>\n",
       "      <td>623.0</td>\n",
       "      <td>396.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6471</th>\n",
       "      <td>Trinidad and Tobago</td>\n",
       "      <td>TT</td>\n",
       "      <td>TTO</td>\n",
       "      <td>1993</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6790</th>\n",
       "      <td>United Kingdom of Great Britain and Northern I...</td>\n",
       "      <td>GB</td>\n",
       "      <td>GBR</td>\n",
       "      <td>2006</td>\n",
       "      <td>9.0</td>\n",
       "      <td>173.0</td>\n",
       "      <td>244.0</td>\n",
       "      <td>213.0</td>\n",
       "      <td>148.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6809</th>\n",
       "      <td>United Republic of Tanzania</td>\n",
       "      <td>TZ</td>\n",
       "      <td>TZA</td>\n",
       "      <td>1991</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 60 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                country iso2 iso3  year  \\\n",
       "1692                                      Cote d'Ivoire   CI  CIV  2002   \n",
       "6021                                          Sri Lanka   LK  LKA  1997   \n",
       "6471                                Trinidad and Tobago   TT  TTO  1993   \n",
       "6790  United Kingdom of Great Britain and Northern I...   GB  GBR  2006   \n",
       "6809                        United Republic of Tanzania   TZ  TZA  1991   \n",
       "\n",
       "      new_sp_m014  new_sp_m1524  new_sp_m2534  new_sp_m3544  new_sp_m4554  \\\n",
       "1692        102.0        1271.0        2194.0        1490.0         833.0   \n",
       "6021         11.0         215.0         390.0         596.0         623.0   \n",
       "6471          NaN           NaN           NaN           NaN           NaN   \n",
       "6790          9.0         173.0         244.0         213.0         148.0   \n",
       "6809          NaN           NaN           NaN           NaN           NaN   \n",
       "\n",
       "      new_sp_m5564     ...      newrel_m4554  newrel_m5564  newrel_m65  \\\n",
       "1692         385.0     ...               NaN           NaN         NaN   \n",
       "6021         396.0     ...               NaN           NaN         NaN   \n",
       "6471           NaN     ...               NaN           NaN         NaN   \n",
       "6790          88.0     ...               NaN           NaN         NaN   \n",
       "6809           NaN     ...               NaN           NaN         NaN   \n",
       "\n",
       "      newrel_f014  newrel_f1524  newrel_f2534  newrel_f3544  newrel_f4554  \\\n",
       "1692          NaN           NaN           NaN           NaN           NaN   \n",
       "6021          NaN           NaN           NaN           NaN           NaN   \n",
       "6471          NaN           NaN           NaN           NaN           NaN   \n",
       "6790          NaN           NaN           NaN           NaN           NaN   \n",
       "6809          NaN           NaN           NaN           NaN           NaN   \n",
       "\n",
       "      newrel_f5564  newrel_f65  \n",
       "1692           NaN         NaN  \n",
       "6021           NaN         NaN  \n",
       "6471           NaN         NaN  \n",
       "6790           NaN         NaN  \n",
       "6809           NaN         NaN  \n",
       "\n",
       "[5 rows x 60 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "who_raw = pd.read_csv('who.csv')\n",
    "\n",
    "print(\"=== WHO TB data set: {} rows x {} columns ===\".format(who_raw.shape[0],\n",
    "                                                              who_raw.shape[1]))\n",
    "print(\"Column names:\", who_raw.columns)\n",
    "\n",
    "print(\"\\n=== A few randomly selected rows ===\")\n",
    "import random\n",
    "row_sample = sorted(random.sample(range(len(who_raw)), 5))\n",
    "display(who_raw.iloc[row_sample])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-9c932436c54e2936",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "The data set has 7,240 rows and 60 columns. Here is how to decode the columns.\n",
    "- Columns `'country'`, `'iso2'`, and `'iso3'` are different ways to designate the country and redundant, meaning you only really need to keep one of them.\n",
    "- Column `'year'` is the year of the report and is a natural variable.\n",
    "- Among columns `'new_sp_m014'` through `'newrel_f65'`, the `'new...'` prefix indicates that the column's values count new cases of TB. In this particular data set, all the data are for new cases.\n",
    "- The short codes, `rel`, `ep`, `sn`, and `sp` describe the type of TB case. They stand for relapse, extrapulmonary, pulmonary not detectable by a pulmonary smear test (\"smear negative\"), and pulmonary detectable by such a test (\"smear positive\"), respectively.\n",
    "- The codes `'m'` and `'f'` indicate the gender (male and female, respectively).\n",
    "- The trailing numeric code indicates the age group: `014` is 0-14 years of age, `1524` for 15-24 years, `2534` for 25-34 years, etc., and `65` stands for 65 years or older.\n",
    "\n",
    "In other words, it looks like you are likely to want to treat all the columns as values of multiple variables!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-1a1b02b4e0738566",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "**Exercise 8** (3 points). As a first step, start with `who_raw` and create a new data frame, `who2`, with the following properties:\n",
    "\n",
    "- All the `'new...'` columns of `who_raw` become values of a _single_ variable, `case_type`. Store the counts associated with each `case_type` value as a new variable called `'count'`.\n",
    "- Remove the `iso2` and `iso3` columns, since they are redundant with `country` (which you should keep!).\n",
    "- Keep the `year` column as a variable.\n",
    "- Remove all not-a-number (`NaN`) counts. _Hint_: You can test for a `NaN` using Python's [`math.isnan()`](https://docs.python.org/3/library/math.html).\n",
    "- Convert the counts to integers. (Because of the presence of NaNs, the counts will be otherwise be treated as floating-point values, which is undesirable since you do not expect to see non-integer counts.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "who2",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>case_type</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>202747</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>new_ep_m014</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209987</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>new_ep_m1524</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>217227</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>new_ep_m2534</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224467</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>new_ep_m3544</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231707</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>new_ep_m4554</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            country  year     case_type  count\n",
       "202747  Afghanistan  2007   new_ep_m014      0\n",
       "209987  Afghanistan  2007  new_ep_m1524      0\n",
       "217227  Afghanistan  2007  new_ep_m2534      0\n",
       "224467  Afghanistan  2007  new_ep_m3544      0\n",
       "231707  Afghanistan  2007  new_ep_m4554      0"
      ]
     },
     "execution_count": 173,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from math import isnan\n",
    "\n",
    "#\n",
    "# YOUR CODE HERE\n",
    "\n",
    "who_test = who_raw.copy()\n",
    "who_test = who_test.drop(['iso2', 'iso3'], axis=1)\n",
    "\n",
    "cols = ['new_sp_m014', 'new_sp_m1524',\n",
    "       'new_sp_m2534', 'new_sp_m3544', 'new_sp_m4554', 'new_sp_m5564',\n",
    "       'new_sp_m65', 'new_sp_f014', 'new_sp_f1524', 'new_sp_f2534',\n",
    "       'new_sp_f3544', 'new_sp_f4554', 'new_sp_f5564', 'new_sp_f65',\n",
    "       'new_sn_m014', 'new_sn_m1524', 'new_sn_m2534', 'new_sn_m3544',\n",
    "       'new_sn_m4554', 'new_sn_m5564', 'new_sn_m65', 'new_sn_f014',\n",
    "       'new_sn_f1524', 'new_sn_f2534', 'new_sn_f3544', 'new_sn_f4554',\n",
    "       'new_sn_f5564', 'new_sn_f65', 'new_ep_m014', 'new_ep_m1524',\n",
    "       'new_ep_m2534', 'new_ep_m3544', 'new_ep_m4554', 'new_ep_m5564',\n",
    "       'new_ep_m65', 'new_ep_f014', 'new_ep_f1524', 'new_ep_f2534',\n",
    "       'new_ep_f3544', 'new_ep_f4554', 'new_ep_f5564', 'new_ep_f65',\n",
    "       'newrel_m014', 'newrel_m1524', 'newrel_m2534', 'newrel_m3544',\n",
    "       'newrel_m4554', 'newrel_m5564', 'newrel_m65', 'newrel_f014',\n",
    "       'newrel_f1524', 'newrel_f2534', 'newrel_f3544', 'newrel_f4554',\n",
    "       'newrel_f5564', 'newrel_f65']\n",
    "\n",
    "who_test3 = pd.melt(who_test, id_vars=['country', 'year'],  value_vars=cols, var_name='case_type', value_name='count') #.reset_index()\n",
    "who2 = who_test3.dropna().sort_values(['country', 'case_type'])\n",
    "who2['count'] = who2['count'].map(int)\n",
    "who2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "who2_test",
     "locked": true,
     "points": 3,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== First few rows of your solution ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>case_type</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>202747</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>new_ep_m014</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209987</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>new_ep_m1524</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>217227</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>new_ep_m2534</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224467</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>new_ep_m3544</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231707</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>new_ep_m4554</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            country  year     case_type  count\n",
       "202747  Afghanistan  2007   new_ep_m014      0\n",
       "209987  Afghanistan  2007  new_ep_m1524      0\n",
       "217227  Afghanistan  2007  new_ep_m2534      0\n",
       "224467  Afghanistan  2007  new_ep_m3544      0\n",
       "231707  Afghanistan  2007  new_ep_m4554      0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== First few rows of the instructor's solution ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>case_type</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Albania</td>\n",
       "      <td>2006</td>\n",
       "      <td>new_ep_f014</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Albania</td>\n",
       "      <td>2007</td>\n",
       "      <td>new_ep_f014</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Albania</td>\n",
       "      <td>2008</td>\n",
       "      <td>new_ep_f014</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Albania</td>\n",
       "      <td>2009</td>\n",
       "      <td>new_ep_f014</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Albania</td>\n",
       "      <td>2010</td>\n",
       "      <td>new_ep_f014</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   country  year    case_type  count\n",
       "0  Albania  2006  new_ep_f014      7\n",
       "1  Albania  2007  new_ep_f014      1\n",
       "2  Albania  2008  new_ep_f014      3\n",
       "3  Albania  2009  new_ep_f014      2\n",
       "4  Albania  2010  new_ep_f014      2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "(Passed.)\n"
     ]
    }
   ],
   "source": [
    "# Test: `who2_test`\n",
    "\n",
    "print(\"=== First few rows of your solution ===\")\n",
    "display(who2.head())\n",
    "\n",
    "print (\"=== First few rows of the instructor's solution ===\")\n",
    "who2_soln = pd.read_csv('who2_soln.csv')\n",
    "display(who2_soln.head())\n",
    "\n",
    "# Check it\n",
    "assert tibbles_are_equivalent(who2, who2_soln)\n",
    "print (\"\\n(Passed.)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-820d5d17acf2028d",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "**Exercise 9** (5 points). Starting from your `who2` data frame, create a new tibble, `who3`, for which each `'key'` value is split into three new variables:\n",
    "- `'type'`, to hold the TB type, having possible values of `rel`, `ep`, `sn`, and `sp`;\n",
    "- `'gender'`, to hold the gender as a string having possible values of `female` and `male`; and\n",
    "- `'age_group'`, to hold the age group as a string having possible values of `0-14`, `25-34`, `35-44`, `45-54`, `55-64`, and `65+`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "who3",
     "locked": false,
     "schema_version": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "#\n",
    "# YOUR CODE HERE\n",
    "re_names = re.compile ('''^\n",
    "                           (?P<new>[new]+)\n",
    "                           \\_?\n",
    "                           (?P<type>[a-z]+)?\n",
    "                           \\_\n",
    "                           (?P<sex>[m?f?])\n",
    "                           (?P<age>[\\d]+)\n",
    "                           $\n",
    "                        ''',\n",
    "                        re.VERBOSE)\n",
    "\n",
    "who3 = who2.copy()\n",
    "who3['case_type_split'] = who3['case_type'].apply(re_names.findall) \n",
    "who3['type'] = [x[0][1] for x in who3['case_type_split']]\n",
    "who3['gender'] = [x[0][2] for x in who3['case_type_split']]\n",
    "who3['gender'] = ['male' if x=='m' else 'female' for x in who3['gender']]\n",
    "who3['age_group'] = [x[0][3] for x in who3['case_type_split']]\n",
    "who3['age_group'] = [x[0]+'-'+x[1:] if len(x)==3 else x[:2]+'-'+x[2:] for x in who3['age_group']]\n",
    "who3['age_group'] = [x.replace('65-', '65+') for x in who3['age_group']]\n",
    "who3 = who3.drop(['case_type', 'case_type_split'], axis=1).reset_index(drop=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "who3_test",
     "locked": true,
     "points": 5,
     "schema_version": 1,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== First few rows of your solution ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>count</th>\n",
       "      <th>type</th>\n",
       "      <th>gender</th>\n",
       "      <th>age_group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>0</td>\n",
       "      <td>ep</td>\n",
       "      <td>male</td>\n",
       "      <td>0-14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>0</td>\n",
       "      <td>ep</td>\n",
       "      <td>male</td>\n",
       "      <td>15-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>0</td>\n",
       "      <td>ep</td>\n",
       "      <td>male</td>\n",
       "      <td>25-34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>0</td>\n",
       "      <td>ep</td>\n",
       "      <td>male</td>\n",
       "      <td>35-44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>2007</td>\n",
       "      <td>0</td>\n",
       "      <td>ep</td>\n",
       "      <td>male</td>\n",
       "      <td>45-54</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       country  year  count type gender age_group\n",
       "0  Afghanistan  2007      0   ep   male      0-14\n",
       "1  Afghanistan  2007      0   ep   male     15-24\n",
       "2  Afghanistan  2007      0   ep   male     25-34\n",
       "3  Afghanistan  2007      0   ep   male     35-44\n",
       "4  Afghanistan  2007      0   ep   male     45-54"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== First few rows of the instructor's solution ===\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>country</th>\n",
       "      <th>year</th>\n",
       "      <th>age_group</th>\n",
       "      <th>gender</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7</td>\n",
       "      <td>Albania</td>\n",
       "      <td>2006</td>\n",
       "      <td>0-14</td>\n",
       "      <td>female</td>\n",
       "      <td>ep</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Albania</td>\n",
       "      <td>2007</td>\n",
       "      <td>0-14</td>\n",
       "      <td>female</td>\n",
       "      <td>ep</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Albania</td>\n",
       "      <td>2008</td>\n",
       "      <td>0-14</td>\n",
       "      <td>female</td>\n",
       "      <td>ep</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>Albania</td>\n",
       "      <td>2009</td>\n",
       "      <td>0-14</td>\n",
       "      <td>female</td>\n",
       "      <td>ep</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Albania</td>\n",
       "      <td>2010</td>\n",
       "      <td>0-14</td>\n",
       "      <td>female</td>\n",
       "      <td>ep</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   count  country  year age_group  gender type\n",
       "0      7  Albania  2006      0-14  female   ep\n",
       "1      1  Albania  2007      0-14  female   ep\n",
       "2      3  Albania  2008      0-14  female   ep\n",
       "3      2  Albania  2009      0-14  female   ep\n",
       "4      2  Albania  2010      0-14  female   ep"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "(Passed.)\n"
     ]
    }
   ],
   "source": [
    "# Test: `who3_test`\n",
    "\n",
    "print(\"=== First few rows of your solution ===\")\n",
    "display(who3.head())\n",
    "\n",
    "who3_soln = pd.read_csv('who3_soln.csv')\n",
    "print(\"\\n=== First few rows of the instructor's solution ===\")\n",
    "display(who3_soln.head())\n",
    "\n",
    "assert tibbles_are_equivalent(who3, who3_soln)\n",
    "print(\"\\n(Passed.)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-51ee73431fcc1d30",
     "locked": true,
     "schema_version": 1,
     "solution": false
    }
   },
   "source": [
    "**Fin!** That's the end of this assignment. Don't forget to restart and run this notebook from the beginning to verify that it works top-to-bottom before submitting."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": [],
  "celltoolbar": "Create Assignment",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}