DataViz for bird brains - teacher.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"#Overview\"></a>\n",
    "# Overview\n",
    "* <a href=\"#Data-viz-for-bird-brains\">Data viz for bird brains</a>\n",
    "  * <a href=\"#Introduction\">Introduction</a>\n",
    "  * <a href=\"#Loading-the-data\">Loading the data</a>\n",
    "    * <a href=\"#Exercise-1\">Exercise 1</a>\n",
    "    * <a href=\"#Exercise-2\">Exercise 2</a>\n",
    "    * <a href=\"#Exercise-3\">Exercise 3</a>\n",
    "  * <a href=\"#Data-maniupulation\">Data maniupulation</a>\n",
    "    * <a href=\"#Exercise-4\">Exercise 4</a>\n",
    "    * <a href=\"#Exercise-5\">Exercise 5</a>\n",
    "    * <a href=\"#Exercise-6\">Exercise 6</a>\n",
    "    * <a href=\"#Exercise-7\">Exercise 7</a>\n",
    "  * <a href=\"#Data-visualization-using-matplotlib\">Data visualization using matplotlib</a>\n",
    "    * <a href=\"#Exercise-8\">Exercise 8</a>\n",
    "    * <a href=\"#Exercise-9\">Exercise 9</a>\n",
    "  * <a href=\"#Data-visualization-using-seaborn\">Data visualization using seaborn</a>\n",
    "    * <a href=\"#Exercise-10\">Exercise 10</a>\n",
    "    * <a href=\"#Exercise-11\">Exercise 11</a>\n",
    "    * <a href=\"#Exercise-12\">Exercise 12</a>\n",
    "    * <a href=\"#Exercise-13\">Exercise 13</a>\n",
    "    * <a href=\"#Exercise-14\">Exercise 14</a>\n",
    "    * <a href=\"#Exercise-15\">Exercise 15</a>\n",
    "    * <a href=\"#Exercise-16\">Exercise 16</a>\n",
    "  * <a href=\"#Data-visualization-with-bokeh\">Data visualization with bokeh</a>\n",
    "    * <a href=\"#Exercise-17\">Exercise 17</a>\n",
    "    * <a href=\"#Exercise-18\">Exercise 18</a>\n",
    "    * <a href=\"#Exercise-19\">Exercise 19</a>\n",
    "    * <a href=\"#Exercise-20\">Exercise 20</a>\n",
    "    * <a href=\"#Exercise-21\">Exercise 21</a>\n",
    "    * <a href=\"#Exercise-22\">Exercise 22</a>\n",
    "  * <a href=\"#Machine-Learning-with-Scikit-Learn\">Machine Learning with Scikit-Learn</a>\n",
    "    * <a href=\"#Exercise-23\">Exercise 23</a>\n",
    "    * <a href=\"#Exercise-24\">Exercise 24</a>\n",
    "    * <a href=\"#Exercise-25\">Exercise 25</a>\n",
    "    * <a href=\"#Exercise-26\">Exercise 26</a>\n",
    "    * <a href=\"#Exercise-27\">Exercise 27</a>\n",
    "  * <a href=\"#Wrap-Up\">Wrap-Up</a>\n",
    "    * <a href=\"#Discussion\">Discussion</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Data-viz-for-bird-brains\"></a>\n",
    "# Data viz for bird brains\n",
    "<a href=\"#Overview\">Return to overview</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://i.kym-cdn.com/photos/images/original/000/844/924/126.jpg)\n",
    "\n",
    "<a name=\"#Introduction\"></a>\n",
    "## Introduction\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "\n",
    "Today, we'll be focusing on some of the neat ways to display you data using a few different graphing packages that go beyond our usual suspect, `matplotlib`. \n",
    "\n",
    "Data viz is about a lot more than just displaying your data - by visualizing your data, you may **discover relationships or patterns** that stats alone might not have clued you into. Thus, data viz shouldn't be thought of solely as a last step in data analysis, but rather an iterative and explorative part of your data processing.\n",
    "\n",
    "The dataset we'll be playing with today is from a study published in 2016 by Olcowicz et al. titled \n",
    "[Birds have primate-like numbers of neurons in the forebrain](http://www.pnas.org/content/early/2016/06/07/1517131113). The dataset is straight-forward as it is mostly brain cell counts of various brain areas from various bird species. \n",
    "\n",
    "The major finding of this paper (i.e. the title) provides neurobiological support for a major shift in our idea of what it means to be a \"bird brain.\" Mounting evidence from this and other studies (and YouTube videos, see links) demonstrate the emotional and cognitive intelligence of our distant but warm-blooded relatives. For example, [crows](https://www.youtube.com/watch?v=cbSu2PXOTOc) make and use tools, [magpies](https://www.youtube.com/watch?v=60Zg9sGnQf8) gather to mourn another's death, and [keas](https://www.youtube.com/watch?v=6bbWF6Gx8UM) produce a laughter-like vocalization when engaging in play that seems to be contagious. But how can a brain as small as a songbird's engage in complex problem solving and rich social regulation? Based on the results of this study, part of what explains advanced avian cognition is that bird brains pack a high density of brain cells.\n",
    "\n",
    "Here are some comparisons between bird and mammal brains. The two numbers under each brain are the brain mass in grams and the total number of brain neurons in millions.\n",
    "\n",
    "\n",
    "<img src=\"braincmp.jpg\">\n",
    "\n",
    "This is all super neat, and Alex and I would be happy to nerd out out birds all day. But our goal today isn't to convince you that birds are fascinating and empathetic and far from feathered robots. **Our goal today is to learn how to craft some graphs that are clear, polished, and information-dense.** So let's hop to it, shall we?\n",
    "\n",
    "As always, we begin by loading the libraries we'll need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "import os\n",
    "import glob as glob\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from bokeh.io import output_notebook\n",
    "from bokeh.models import ColumnDataSource, HoverTool, LinearColorMapper\n",
    "from bokeh.palettes import inferno, plasma\n",
    "from bokeh.plotting import figure, show\n",
    "from bokeh.transform import transform\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neighbors import KNeighborsClassifier  \n",
    "from sklearn.metrics import classification_report \n",
    "\n",
    "pd.options.display.max_rows = 10\n",
    "pd.options.display.float_format = '{:.2f}'.format  # limit the number of digits displayed after the decimal to 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Loading-the-data\"></a>\n",
    "## Loading the data\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "It's not uncommon for data to be spread out over multiple files. In our case, the data lives in 6 different .csv files, each named after the brain region that they were collected from. Sure, we could write a line of code for each file we need to import, ending up with 6 lines of code and 6 dataframes. But what if our data were split into dozens or even hundreds of files? (think imaging files). Instead of writing a line of code for each data file, let's utilize our programmatic powers to do this more efficiently and flexibly in such a way that it can be applied to any number of data files.\n",
    "<a name=\"#Exercise-1\"></a>\n",
    "### Exercise 1\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Before we can use a `read` function, we first need to generate a list of data files we want to import. Finding this method from the `glob` package is not exactly intuitive, so we've gone ahead and written the code for you. \n",
    "\n",
    "Go ahead and run it, then display it. Based on the output, what do you think this line of code is doing? What type of object is `data_files`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_files = glob.glob(\"*bb.csv\")\n",
    "\n",
    "%load \"answers/answer_001.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`glob` is a python module that searches for filenames containing a specified string or pattern within your current working directory. Here, we told `glob` to search within the current working directory for any files that end in `bb` + our data file format, `.csv`. \n",
    "\n",
    "Note: The `*` is a special \"wildcard\" character indicating that any string of characters can precede the `bb.csv`.\n",
    "\n",
    "Now that we have our list of data files, we need to import them. For the sake of time, we've provided some code to do this below. Have a look-see. What is each line doing?\n",
    "\n",
    "**Bonus question 1**: Why do we define the object `data` _first_ & _outside_ the for loop?\n",
    "\n",
    "**Bonus question 2**: How could you re-write this for loop to a single line of code using list comprehension?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = []\n",
    "for file in data_files:\n",
    "    df = pd.read_csv(file, index_col=None, header=0)\n",
    "    data.append(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of writing one line of code per data file, we've made use of a for loop. As it loops through each .csv file in our `data_files` list, it:\n",
    "\n",
    "   * reads the data into a pandas dataframe called `df` (think of this a temporary holding variable)\n",
    "    \n",
    "   * appends (adds) that dataframe to a list called `data`\n",
    "\n",
    "**Bonus answer 1**: We need to define data outside the for loop to avoid getting a python error that says\n",
    "`NameError: name 'data' is not defined'`. Basically, you can't run a method (`.append` in our case) on an object that didn't exist _before_ calling the method.\n",
    "\n",
    "Note that initializing/preallocating an empty data structure to fill as you process/analyze is very common practice, as it is usually less computationally expensive than iteratively restructuring, e.g. adding a row to an array with each iteration of a loop.\n",
    "\n",
    "**Bonus answer 2**: The above four lines can be reduced to one using a list comprehension:\n",
    "\n",
    "    data = [pd.read_csv(file, index_col=None, header=0) for file in data_files]\n",
    "\n",
    "    \n",
    "\n",
    "<a name=\"#Exercise-2\"></a>\n",
    "### Exercise 2\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "\n",
    "If you haven't already, take a look at this `data` object. What type of object is it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_002.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does this object contain? Do you see any problems with how this is formatted if, say, we wanted to plot values from different brain regions on the same graph?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_003.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, the good news is that we've got the data from our 6 .csv files imported, and we could've just as easily handled importing 200 (or any other arbitrary number of) files with our glob + for loop sleight of hand. \n",
    "\n",
    "These data sets contain the following columns:\n",
    "* **id**, an alphanumeric tag indicating which individual animal the data is from\n",
    "* **common_name**, the common name for the species\n",
    "* **mass**, the mass of the brain region specified in grams (g)\n",
    "* **cells**, the total number of cells in the brain region specified\n",
    "* **neurons**, the total number of neurons in the brain region specified\n",
    "* **non-neurons**, the total number of non-neurons in the brain region specified\n",
    "* **brain_region**, area of the brain from which the data originated\n",
    "\n",
    "Unfortunately, instead of ending up with all of our data in one dataframe object, we're left with a **list of dataframes** in an object called `data`. This could lead to some clunky code if we want to transform or plot data across data sets. It would be a lot easier to manipulate our data if it were all in one dataframe...\n",
    "\n",
    "<a name=\"#Exercise-3\"></a>\n",
    "### Exercise 3\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "\n",
    "How can we combine or (hint!) **concatenate** a list of dataframes into a single dataframe? Find a pandas method that will take our `data` list and do just this, and name your new dataframe `bb`.\n",
    "\n",
    "Reminder: `pd.<TAB>` is your friend for finding a suitable method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_004.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ta-daaa! Every datapoint together at last. Let's do some data manipulation now.\n",
    "\n",
    "<a name=\"#Data-maniupulation\"></a>\n",
    "## Data maniupulation\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "This dataset contains two types of numerical data: mass (g) and cell counts. This is a good place to start, but limits the kinds of comparisons we can do if we'd like to control for, say brain size. For example, if you predict that the number of neurons correlate in a brain correlates with the level of intelligence for that animal, you might predict that an ostrich would be far more intelligent than a crow. But consider the difference in size of an ostrich brain vs. that of a crow...\n",
    "\n",
    "<a name=\"#Exercise-4\"></a>\n",
    "### Exercise 4\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "One way to \"normalize\" these cell counts is to transform cell counts into cell densities. Before we do that, let's convert the `mass` column from grams to miligrams. Save these converted values as `mass_mg` to `bb`.\n",
    "\n",
    "Hint: if you're like me and get easily turned around by unit conversion, think about whether the final value should be bigger or smaller than the starting value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_005.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good job! \n",
    "\n",
    "<a name=\"#Exercise-5\"></a>\n",
    "### Exercise 5\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Now, using our newly-crafted `mass_mg` column, calculate the density of neurons and assign it to a new column called \"neuronal_density\" in our `bb` dataframe. What is the unit of this column?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_006.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Exercise-6\"></a>\n",
    "### Exercise 6\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Repeat this operation of calculating cell counts per mg for both the `cells` and `non-neurons` columns, saving the resulting densities to columns named `cellular_density` and `non-neuronal_density`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_007.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right-o! Densities abound. Let's do one last manipulation before moving onto graphing.\n",
    "\n",
    "<a name=\"#Exercise-7\"></a>\n",
    "### Exercise 7\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Note that there are 3 individuals (encoded as 3 unique `id`s) for each species in this dataset, each with data for 6 different brain regions. Calculate the **averages** for our numerical data by **both species and brain region** so that each species has one row of data per brain region. Call this new dataframe `bb_means`.\n",
    "\n",
    "Hint: You'll need to create a **grouping variable**, and pass that grouping function an additonal parameter: `as_index=False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_008.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A little help from `.groupby` and birb's your uncle! \n",
    "\n",
    "Jumping ahead a teeny bit, we include the `as_index=False` parameter is so that `seaborn` treats the `common_name` and `brain_region` columns as categorical **variables** instead of **indices**.\n",
    "\n",
    "<a name=\"#Data-visualization-using-matplotlib\"></a>\n",
    "## Data visualization using matplotlib\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "You've seen it at work before - good ol' matplotlib. Let's do a quick refresher.\n",
    "\n",
    "<a name=\"#Exercise-8\"></a>\n",
    "### Exercise 8\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Do you remember how to make a scatterplot using `matplotlib`? Create a scatterplot using `matplotlib` (imported as `plt`) with mass on the x axis and neurons on the y axis. Source this data from `bb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_009.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woah! There are some serious outliers here. Sure would be nice to know what those are *without* consulting the dataframe itself... but more on that later!\n",
    "\n",
    "What if we wanted to add another dimension to this plot, such as neuronal density? How might we add to our scatterplot to include density?\n",
    "\n",
    "...\n",
    "\n",
    "Any ideas?\n",
    "\n",
    "...\n",
    "\n",
    "One possible solution is to encode the density as the color of the markers (similar to a heatmap.) Another solution is to change the size of the markers to encode density. Can we scale our marker size to encode neuronal density by passing an additional argument to `plt.scatter`?\n",
    "\n",
    "<a name=\"#Exercise-9\"></a>\n",
    "### Exercise 9\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Use the '?' to inspect the documentation for plt.scatter. Do you see any parameters that set the **marker size**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_010.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to a quick glance at the documentation, the 's' parameter appears to deal with marker size. What happens when we pass 's' our neuronal density column? Try it here.\n",
    "\n",
    "Hint: The resulting graph may not look... informative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_011.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow, a giant blue box, enlightening, eh? Looks like we're going to have to do some more fanangling and transforming to get `matplotlib` to encode our density data as marker size. Everyone, please turn to page 1 of the User Guide for `matplotlib` and begin reading up on-\n",
    "\n",
    "JUST KIDDING! Ain't nobody got time for that. Not when there are perfectly good graphing packages to be tapped!\n",
    "\n",
    "<a name=\"#Data-visualization-using-seaborn\"></a>\n",
    "## Data visualization using seaborn\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "\n",
    "Lucky for us, there's a better way to add additional dimensions to your plots using a python graphing package we have brushed upon before by the name of `seaborn`.`seaborn`'s parameters look a lot like `matplotlib`, but include parameters that make **semantic groupings** (e.g. marker color as an indicator of brain area) a breeze.\n",
    "\n",
    "Let's start with a bare bones `seaborn` scatterplot. \n",
    "\n",
    "<a name=\"#Exercise-10\"></a>\n",
    "### Exercise 10\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Using `seaborn` (imported as `sns`), create a scatterplot with mass on the x axis and neuron count on the y axis sourcing data from our `bb_means` dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_012.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Exercise-11\"></a>\n",
    "### Exercise 11\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Now take a second to generate the same plot neurons vs mass from `bb_means`using matplotlib. Do you see any differences between this graph and that generated with `matplotlib`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_013.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a two differences that pop out at me:\n",
    "* marker outlines\n",
    "* axis labels\n",
    "\n",
    "Already, `seaborn` is coming out ahead in terms of readability, wouldn't you agree? \n",
    "\n",
    "Under the hood, `seaborn` uses `matplotlib`. It just uses more sensible \"defaults\" for many of the plotting functions (e.g., you can achive marker outlines in matplotlib, but you have to pass in additional arguments, which makes for more work.)\n",
    "\n",
    "<a name=\"#Exercise-12\"></a>\n",
    "### Exercise 12\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "\n",
    "Let's recreate that same neurons vs mass scatter plot using `seaborn`, assign your plot to an object (handle) named `p`, then add two lines of code involving methods on `p` to **scale** the x and y axes to be log-scaled. \n",
    "\n",
    "Hint: The methods for x and y scaling are somewhere in the list pulled up by entering `p.set_<TAB>`. Check the method's documentation to determine what **value** you should pass to the function for a log scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_014.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Very nice! Now there is less overlap between points, so we can see more of the data. We'll make these same log-scaling modifications to the next several plots, so keep it handy on your copy-paste clipboard.\n",
    "\n",
    "<a name=\"#Exercise-13\"></a>\n",
    "### Exercise 13\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Let's see if we can find a way to encode neuronal density in the size of each marker. Inspect the documentation for `sns.scatterplot` to find additional parameters that set the size of markers. Then pass our `neuronal_densities` column to this additional parameter to create a scatterplot with densities indicated by marker size.\n",
    "\n",
    "Continue to assign your plot to `p` and copy&paste the log-scaling methods from the previous exercise.\n",
    "\n",
    "Hint: You might find more than one parameter that have to do with marker size. Try running your code with each one separately if you're not sure which one is the right one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.scatterplot?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_015.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a bit hard to see the differences between small dots and large dots here... Surely there must be a way to adjust the ranges of marker sizes.\n",
    "\n",
    "<a name=\"#Exercise-14\"></a>\n",
    "### Exercise 14\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Let's pass `sns.scatterplot` another parameter, `sizes`, as a tuple, which is a data type formated as two numbers separated by a comma and surrounded by parenthesis, e.g. (12,200). Experiment with different values for `sizes` until you find one that you like. Again, add your log scaling modification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_016.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neat! And look at that auto-generated legend! Our buddy `seaborn` does it again. Let's add one last dimension to this plot before moving on to some interactive graphs.\n",
    "\n",
    "<a name=\"#Exercise-15\"></a>\n",
    "### Exercise 15\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Based on the plot above, it appears that the larger dots are clustering around the smaller masses. But remember that this dataset contains cell counts (and our added densities) for **multiple brain regions**. \n",
    "\n",
    "Take another gander at `sns.scatterplot`'s documentation on parameters to find one that deals with the **color** of the markers. Then, pass it our `brain_region` column to recreate the plot above with brain region encoded as color.\n",
    "\n",
    "Hint: What's another word for color?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_017.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Piece of cake! Now we see that the high density datapoints at lower masses all belong to densities of the cerebellum. This is consistent with what's known about neuronal density in mammals, where ~70% of your neurons are not, as one might expect, in your cerebrum, but rather jam-packed into the granule cell layer of your cerebellum.\n",
    "\n",
    "<a name=\"#Exercise-16\"></a>\n",
    "### Exercise 16\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "\n",
    "One last final tweak - the legend has gotten rather large now that it must describe our two **semantic groupings**: brain region by marker color and neural density by marker size. \n",
    "\n",
    "Run the code below to move our legend outside the plotting area."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p = sns.scatterplot(x=\"mass\", y=\"neurons\", hue=\"brain_region\", size=\"neuronal_density\", sizes=(40, 400), data=bb_means)\n",
    "p.set_xscale('log')\n",
    "p.set_yscale('log')\n",
    "plt.legend(bbox_to_anchor=(1.05, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A rainbow of data, isn't it beautiful?\n",
    "\n",
    "Notice that we're using `plt.legend` here to modify the legend, not `sns.legend`. This is because `seaborn` was built on top of `matplotlib`, so it utilizes some of `matplotlib`'s methods.\n",
    "\n",
    "To briefly break down how this line of code works, you can think of your plotting area as an x-y coordinate system that ranges from 0 to 1 along each axis, where y=1 be the top edge and x=1 would be the right edge of your plot. We set the `bbox_to_anchor` parameter to start drawing the legend at x=1.05, slightly beyond the x=1 edge of our plot, and y=1, in line with the top edge of our plot. If you're confused, try changing the vaules passed to `bbox_to_anchor` and see how it changes the placement of your legend.\n",
    "\n",
    "<a name=\"#Data-visualization-with-bokeh\"></a>\n",
    "## Data visualization with bokeh\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Take another look at our neurons vs mass plot (based on the `bb` dataframe and without log-scaling). What are the values at higher masses that don't follow the same trend as the other data points? `matplotlib` isn't too informative when it comes to determining what each of the points on the scatter plot represent. For that, we will use an interactive data visualization package for python called `bokeh`. Using `bokeh` we will create a visually appealing, interactive plot which will allow us to learn about individual data points.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(x=\"mass\", y=\"neurons\", data=bb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Exercise-17\"></a>\n",
    "### Exercise 17\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "\n",
    "Lets's begin by making variables `x` that corresponds to mass, `y` that corresponds to neurons, `desc` that corresponds to common_name, `ndensity` that corresponds to neuronal_density, and `brain_region` that corresponds to brain_region."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_018.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Exercise-18\"></a>\n",
    "### Exercise 18\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "As we saw in the earlier exercises, seaborn and matplotlib both accept dataframes and/or numpy arrays to make plots. Bokeh requires a little more work to generate a plot (but, that extra work can be worth it). Here, Bokeh requires that you \"package\" your data into a special object known as a `ColumnDataSource` that's designed to work with Bokeh plots. `ColumnDataSource` requires the data to be stored as a **`dict`ionary**. One way of creating a dictionary is by</font> calling \"dict\" with key-value pairs inside parentheses like this:\n",
    "\n",
    "    dict(key1=value1, key2=value2, key3=value3, ...)\n",
    "    \n",
    "First, set our variables x, y, desc, and ndensity each equal to themselves in a dictionary called `plot_dict`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_019.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Exercise-19\"></a>\n",
    "### Exercise 19\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Now, call the function ColumnDataSource and set it's parameter `data` equal to the `plot_dict` we just made. Store this as `data_for_graph`. We now have the data prepared for our graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_020.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where Bokeh shines is when using tools and mappers to help visualize your data. Here we use `HoverTool` which enables us to hover over data points and extract information from our graph. We assign labels to our `x`, `y`, `desc`, `ndensity` and `brain_region` variables. `LinearColorMapper` is used to map a color gradient onto our graph. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hover = HoverTool(tooltips=[\n",
    "    ('Common Name', '@desc'),\n",
    "    ('Brain Mass', \"@x\"),\n",
    "    ('No. of Neurons', \"@y\"),\n",
    "    ('Neuronal density', \"@ndensity\"),\n",
    "    ('Brain region', \"@brain_region\"),\n",
    "])\n",
    "mapper = LinearColorMapper(plasma(100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Exercise-20\"></a>\n",
    "### Exercise 20\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "It is now time to begin crafting our plot. Bokeh requires us to call `figure` containing various parameters for the plot. The parameters we will use are `plot_width` and `plot_height` both set to 500. We will also set the `tools` parameter to our `[hover]` variable contained in a list (don't forget the square brackets), and set the `title` parameter to something useful.\n",
    "\n",
    "This sets up a frame for our figure, on which we can plot our data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_021.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Exercise-21\"></a>\n",
    "### Exercise 21\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "We are now going to instruct Bokeh to make the data points on our graph. We will use `circle` to make our data points circles (you can also use `triangle` or `square` if your heart desires). Our values for the parameters x and y will be our `x` and `y` variables (as strings). The `size` of our data points is 10, and the `source` of our data is our `data_for_graph` object. The last parameter we will use is `fill_color` which we will set to `transform('y', mapper)` to make a pretty color gradient on our graph.\n",
    "\n",
    "Hint: start out with plot.circle("
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_022.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Exercise-22\"></a>\n",
    "### Exercise 22\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Finally, call the function `output_notebook` with no parameters to show the plot inline, and call the function `show` on `plot`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_023.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Emu!!!\n",
    "![](https://tribwxmi.files.wordpress.com/2016/10/emu.jpg?quality=85&strip=all&w=400&h=225&crop=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Machine-Learning-with-Scikit-Learn\"></a>\n",
    "## Machine Learning with Scikit-Learn\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Scikit-learn is a machine learning package which allows us to use algorithms that can learn from data and make predictions about our data.\n",
    "\n",
    "From the package sklearn.neighbors we are going to use KNeighborsClassifier. KNeighborsClassifier uses the k-nearest neighbors (KNN) algorithm to classify our data. KNN allows us to look at the data points nearby an unknown value (it's \"neighbors\"), and uses the neighbors to classify the unknown data point. Using labeled data to classify a new data point is reffered to as supervised learning.\n",
    "\n",
    "<img src=\"knn.png\">\n",
    "\n",
    "Basic machine learning steps:\n",
    "\n",
    "1. instantiate the model \n",
    "2. fit the model to the data \n",
    "3. predict using your test data \n",
    "4. evaluate your model\n",
    "\n",
    "In the following exercises we will use the KNN algorithm to determine if unlabeled data points are part of the Passeriformes order (songbirds belong to this clade) or Psittaciformes (parrots).\n",
    "\n",
    "\n",
    "\n",
    "<a name=\"#Exercise-23\"></a>\n",
    "### Exercise 23\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Import 'scikitdataset.csv' using our usual pandas method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_024.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use indexing and slicing on the df dataframe to set X equal to all rows of our two feature columns, and y equal to the 'order' column of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_025.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: `scikit-learn` convention uses an uppercase X to denote data and a lowercase y denote labels.\n",
    "\n",
    "`train_test_split` will randomly split our data into two categories: data to train our model and data to test our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Exercise-24\"></a>\n",
    "### Exercise 24\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Show the head of both X and y and the tail of y (all in one line using `display` if you can) to confirm the we properly indexed and sliced our dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_026.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Exercise-25\"></a>\n",
    "### Exercise 25\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "\n",
    "Step 1: instantiate the model\n",
    "\n",
    "Create a variable called classifier and set it equal to the `KNeighborsClassifier` model. This is called instantiating the model. Within the classifier, set the parameter for number of neighbors to 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_027.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Exercise-26\"></a>\n",
    "### Exercise 26\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Step 2: fit the model to the training data\n",
    "\n",
    "Next use `fit` on our `classifier` variable to fit our model to `X_train` and `y_train`. This will output some information about our KNN classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_028.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Exercise-27\"></a>\n",
    "### Exercise 27\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "Step 3: predict using your test data\n",
    "\n",
    "Next make a variable called y_pred. y_pred will be a `predict`ion on `classifier` using the `X_test` data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load \"answers/answer_029.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 4: evaluate your model\n",
    "\n",
    "Next we'll test the accuracy of our model using `classification_report`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Precision is the number of correctly identified labels divided by the total number of attemps. Recall is number of correctly identified labels divided by the total possible correct responses. F1-score is the harmonic mean of the precision and recall. For all these values, 1 is the best and 0 is the worst.\n",
    "\n",
    "According to our classification report shows our classifier does a pretty good job but is not always accurate.\n",
    "Run the following cell and enter responses for a body mass of 120 and a telencephalon mass of 9. The correct answer is Psittaciformes. What does our model predict? What about 4.6 for brain mass and 0.2 for mass of telencephalon? The correct answer is Passeriformes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = KNeighborsClassifier(n_neighbors = 5) \n",
    "\n",
    "model.fit(X, y)\n",
    "\n",
    "input1 = input('Enter body mass ')\n",
    "input2 = input('Enter mass of telencephalon ')\n",
    "classifier = model.predict([[input1,input2]])\n",
    "print(\"the classifier determines the bird is part of the \" + classifier + \" order\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our simple model, we could predict the order a newly discovered unknown bird species belongs to! (but there are definitely more accurate ways to do that...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"#Wrap-Up\"></a>\n",
    "## Wrap-Up\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "\n",
    "Hopefully after these exercise you have learned that matplotlib can be quite limiting, how to color- or size- code your markers in seaborn, how to create interactive plots in bokeh, and how to use a machine learning library to make predictive classifiers using scikit-learn.\n",
    "\n",
    "<a name=\"#Discussion\"></a>\n",
    "### Discussion\n",
    "<a href=\"#Overview\">Return to overview</a>\n",
    "When might you use each of these tools? What are the pros and cons of each? \n",
    "\n",
    "~*~*~*~*~*~*~*~*~*~*~*~\n",
    "\n",
    "Hope you enjoyed our data viz party as much as this pirrot.\n",
    "\n",
    "![](https://cultofthepartyparrot.com/parrots/hd/pirateparrot.gif)\n",
    "\n",
    "\n",
    "Kthxbye,\n",
    "\n",
    "Alex & Sami"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}