Lab_02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 2: Importing and plotting data\n",
    "**Data Science for Biologists** &#8226; University of Washington &#8226; BIOL 419/519 &#8226; Winter 2019\n",
    "\n",
    "Course design and lecture material by [Bingni Brunton](https://github.com/bwbrunton) and [Kameron Harris](https://github.com/kharris/). Lab design and materials by [Eleanor Lutz](https://github.com/eleanorlutz/), with helpful comments and suggestions from Bing and Kam.\n",
    "\n",
    "### Table of Contents\n",
    "1. Review of Numpy arrays\n",
    "2. Importing data from a file into a Numpy array\n",
    "3. Examining and plotting data in a Numpy array\n",
    "4. Bonus exercise\n",
    "\n",
    "### Helpful Resources\n",
    "- [Python Data Science Handbook](http://shop.oreilly.com/product/0636920034919.do) by Jake VanderPlas\n",
    "- [Python Basics Cheat Sheet](https://datacamp-community-prod.s3.amazonaws.com/e30fbcd9-f595-4a9f-803d-05ca5bf84612) by Python for Data Science\n",
    "- [Jupyter Notebook Cheat Sheet](https://datacamp-community-prod.s3.amazonaws.com/48093c40-5303-45f4-bbf9-0c96c0133c40) by Python for Data Science\n",
    "- [Matplotlib Cheat Sheet](https://datacamp-community-prod.s3.amazonaws.com/28b8210c-60cc-4f13-b0b4-5b4f2ad4790b) by Python for Data Science\n",
    "- [Numpy Cheat Sheet](https://datacamp-community-prod.s3.amazonaws.com/e9f83f72-a81b-42c7-af44-4e35b48b20b7) by Python for Data Science\n",
    "\n",
    "### Data\n",
    "- The data in this lab is from the [Palmer Penguin Project](https://github.com/allisonhorst/palmerpenguins) by Dr. Kristen Gorman. The data was edited for teaching purposes. \n",
    "\n",
    "## Lab 2 Part 1: Review of Numpy arrays\n",
    "\n",
    "In lecture this week we used Numpy arrays to generate random numbers, look at data, and make patterns. In this first lab section we'll review how to create, access, and edit parts of a Numpy array. \n",
    "\n",
    "To use the Numpy library we need to first import it using the command `import numpy as np`. We'll also import Matplotlib in this same code block, since we'll use this library later in the lab. It's good practice to import all of your libraries at the very beginning of your code file, so that anyone can quickly see what external libraries are necessary to run your code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Magic command to turn on in-line plotting \n",
    "# (show plots within the Jupyter Notebook)\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a Numpy array from existing data\n",
    "To review some important concepts about Numpy arrays, let's make a small 3x3 array called `alphabet_data`, filled with different letters of the alphabet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "row_A = [\"A\", \"B\", \"C\"]\n",
    "row_D = [\"D\", \"E\", \"F\"]\n",
    "row_G = [\"G\", \"H\", \"I\"]\n",
    "\n",
    "alphabet_data = np.array([row_A, row_D, row_G])\n",
    "print(alphabet_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `print` command to look at the entire `alphabet_data` Numpy array. But often we'll work with very large arrays full of data, and we'll want to pick small subsets of the data to look at. Therefore, it's useful to know how to ask Python to give you just a section of any Numpy array. \n",
    "\n",
    "### Selecting subsets of Numpy arrays\n",
    "In lab 1, we talked about how index values describe where to find a specific item within a Python list or array. For example, the variable `example_list` is a list with one row, containing three items. To print the first item in the list we would print `example_list[0]`, or *the value in the variable example_list at index 0*. Remember that the first item in a Python list corresponds to the *index* 0. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_list = [\"avocado\", \"tomato\", \"onion\"]\n",
    "\n",
    "print(\"example_list is:\", example_list)\n",
    "print(\"example_list[0] is:\", example_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting a single value in a Numpy array\n",
    "`alphabet_data` is a little more complicated since it has rows *and* columns, but the general principle of indexing is still the same. Each value in a Numpy array has a unique index value for its row location, and a separate unique index value for its column location. We can ask Numpy to give us just the value we want by using the syntax `alphabet_data[row index, column index]`. \n",
    "\n",
    "![Array indexing figure](figures/array_indexing.jpg)\n",
    "\n",
    "**Exercise 1:** Use indexing to print the second item in the first row of `alphabet_data`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting a range of values in a Numpy array\n",
    "\n",
    "In addition to selecting just one value, we can use the syntax `lower index range : upper index range` to select a range of values. Remember that ranges in Python are *exclusive* - the last index in the range is not included. Below is an example of range indexing syntax used on `example_list`: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_list = [\"avocado\", \"tomato\", \"onion\"]\n",
    "\n",
    "print(\"example_list is:\", example_list)\n",
    "print(\"example_list[0:2] is:\", example_list[0:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use exactly the same notation in a Numpy array. However, since we have both row *and* column indices, we can declare one range for the rows and one range for the columns. For example, the following code prints all rows from index 0 to index 3, and all columns from index 0 to index 2. Note that index 3 doesn't actually exist - but since the upper index range is not included in a Python range, we need to use an index of 3 to print everything up to index 2. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(alphabet_data[0:3, 0:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2:** Print the first two rows of the first two columns in `alphabet_data`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3:** Print the last two rows of the last two columns in `alphabet_data`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we know how to select subsets of arrays, we can use this knowledge to *change* the items in these selections. For example, in a list we can assign a value found at a specific index to be something else. In this example we use indexing to reference the first item in `example_list`, and then change it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_list = [\"avocado\", \"tomato\", \"onion\"]\n",
    "print(\"before assignment, the example_list is:\", example_list)\n",
    "\n",
    "example_list[0] = \"banana\"\n",
    "print(\"after assignment, the example_list is:\", example_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we can change items in a Numpy array using indexing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"before assignment, alphabet_data is:\")\n",
    "print(alphabet_data)\n",
    "\n",
    "alphabet_data[0, 0] = \"Z\"\n",
    "print(\"after assignment, alphabet_data is:\")\n",
    "print(alphabet_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 4:** Replace the item in the third row and second column of `alphabet_data` with `\"V\"`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 5:** Replace the entire second row of `alphabet_data` with a new row: `[\"X\", \"Y\", \"X\"]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lab 2 Part 2: Importing data from a file into a Numpy array\n",
    "\n",
    "Let's apply these principles of Numpy arrays to some real biological data. In the `Lab_02` data folder there are three data files: \n",
    "- `./data/Lab_02/Adelie_Penguin.csv`\n",
    "- `./data/Lab_02/Chinstrap_Penguin.csv`\n",
    "- `./data/Lab_02/Gentoo_Penguin.csv`\n",
    "\n",
    "These files contain data collected by Dr. Kristen Gorman and the Palmer Station, Antarctica LTER - a member of the Long Term Ecological Research Network.\n",
    "\n",
    "![Illustration showing the three penguin species](figures/lter_penguins.jpg)\n",
    "\n",
    "*Credit:* Artwork by @allison_horst\n",
    "\n",
    "The data is formatted as a large table, with one file for each species of penguin. The files contain 50 rows, each representing one individual, and four columns, which represent culmen length and depth, flipper length, and body mass. For example, `Adelie_Penguin.csv` corresponds to the column and row labels shown below:  \n",
    "\n",
    "| Penguin ID      | Culmen Length (mm) | Culmen Depth (mm) | Flipper Length (mm) | Body Mass (g) |\n",
    "| --- | ----------- | ----------- | ----------- | ----------- |\n",
    "| Individual 1  |    39.1 |\t18.7\t | 181\t |3750 |\n",
    "| Individual 2  |  39.5\t|17.4\t|186|\t3800|\n",
    "| ...      | ...      | ...      | ...      | ...      |       \n",
    "| Individual 50 |  39.6\t|17.7|\t186|3500|\n",
    "\n",
    "![Illustration showing the measurement locations for the culmen](figures/culmen_depth.jpg)\n",
    "\n",
    "*Credit:* Artwork by @allison_horst\n",
    "\n",
    "We'll use the Numpy command `loadtxt` to read in our first file, `Adelie_Penguin.csv`. We will save this data in a Numpy array called `adelie_data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load our file data from \"filename\" into a variable called adelie_data\n",
    "filename = \"./data/Lab_02/Adelie_Penguin.csv\"\n",
    "adelie_data = np.loadtxt(fname=filename, delimiter=\",\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data description above tells us that `adelie_data` should contain 50 rows and 4 columns, so let's use the Numpy `shape` command to double check that's the case. Numpy `shape` will print two numbers in the format `(number of rows, number of columns)`. \n",
    "\n",
    "**Exercise 6:** Right now, the code below prints a warning if we don't have the expected 50 rows. Edit the code so that the warning is also printed if the number of columns is not 4. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the shape of the loaded dataset\n",
    "data_shape = adelie_data.shape\n",
    "print(\"Adelie data shape is:\", data_shape)\n",
    "\n",
    "# Print a warning if the data shape is not what we expect\n",
    "if data_shape[0] != 50:\n",
    "    print(\"Unexpected data shape!\")\n",
    "else:\n",
    "    print(\"Correct data shape of 50 rows, 4 columns!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like our `adelie_data` Numpy array is the shape we expect. Now let's look at a subset of data to see what kind of data we're working with. \n",
    "\n",
    "**Exercise 7:** Use Python array indexing to print the first three rows, first four columns of `adelie_data`. Check to make sure that the printed data matches what is given to you in the data description above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lab 2 Part 3: Examining and plotting data in a Numpy array\n",
    "\n",
    "### Calculate interesting characteristics of a Numpy array\n",
    "Now that we have loaded our Adelie penguin data into a Numpy array, there are several interesting commands we can use to find out more about our data. First let's look at the culmen length column (the first column in the dataset). Using array indexing, we will put this entire first column into a new variable called `culmen_lengths`. When indexing between a range of values, leaving the upper range bound blank causes Python to include everything until the end of the array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put the culmen lengths for this dataset in a variable called culmen_lengths\n",
    "culmen_lengths = adelie_data[0:, 0]\n",
    "print(\"The culmen lengths in this dataset is:\")\n",
    "print(culmen_lengths)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy contains many useful functions for finding out different characteristics of a dataset. The code below shows some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print some interesting characteristics of the data\n",
    "print(\"Mean:\", np.mean(culmen_lengths))  \n",
    "print(\"Standard deviation:\", np.std(culmen_lengths))   \n",
    "print(\"Median:\", np.median(culmen_lengths)) \n",
    "print(\"Minimum:\", np.min(culmen_lengths))    \n",
    "print(\"Maximum:\", np.max(culmen_lengths))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use our `culmen_lengths` variable and the useful characteristics we found above to make a histogram of our data. In the below code we've created a histogram, and added a line that shows where the mean of the dataset is.\n",
    "\n",
    "**Exercise 8:** Edit the code block below to plot the maximum and minimum data values as two additional vertical lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a histogram with an opacity of 50% (alpha=0.5)\n",
    "plt.hist(culmen_lengths, alpha=0.5)\n",
    "\n",
    "# Add a vertical line to the plot showing the mean. \n",
    "plt.axvline(np.mean(culmen_lengths), label=\"mean\") \n",
    "# Your code here!\n",
    "\n",
    "# Don't forget to label the axes! \n",
    "plt.xlabel(\"Culmen length (mm)\")\n",
    "plt.ylabel(\"Frequency (number of penguins)\")\n",
    "\n",
    "# Add a legend to the plot\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot in our jupyter notebook\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Review of for loops using indexing\n",
    "\n",
    "Last week in lab we went over an example of a `for` loop that uses indices to loop through a list. Let's pretend that in this Adelie penguin dataset, we have marked in our lab notebook that the first, 12th, 26th, and 44th penguins we sampled seemed suspiciously small. Let's use a `for` loop to print out the culmen length of each of these penguins. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First let's make a list of all of the indexes where we can find suspicious penguins.\n",
    "interesting_indices = [0, 11, 25, 43]\n",
    "\n",
    "# Now we'll look at every single index in the list of suspicious indices.\n",
    "for index in interesting_indices:\n",
    "    \n",
    "    # Because we are looking at indices, we need to use indexing to find the \n",
    "    # value in culmen_lengths that we're interested in. \n",
    "    culmen = culmen_lengths[index]\n",
    "    \n",
    "    print(\"The culmen length at index\", index, \":\", culmen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 9:** Instead of using a `for` loop to look at just the indices in `interesting_indices`, use a `for` loop to look at *all* indices in the `culmen_lengths` dataset. Remember that you can use the command `len(culmen_lengths)` to find out how many values are in the data. Print the culmen length and index if the culmen length is larger than the mean culmen length. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far we've only looked at the culmen lengths in this dataset. Let's use a `for` loop to also look at the culmen depths, flipper lengths, and body mass. Remember that the columns in this dataset stand for:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "culmen_lengths = adelie_data[0:, 0]\n",
    "culmen_depths = adelie_data[0:, 1]\n",
    "flipper_lengths = adelie_data[0:, 2]\n",
    "body_mass = adelie_data[0:, 3]\n",
    "\n",
    "morphologies = [culmen_lengths, culmen_depths, flipper_lengths, body_mass]\n",
    "\n",
    "for morphology in morphologies: \n",
    "    # Create a histogram \n",
    "    plt.hist(morphology)\n",
    "    # Show the plot in our jupyter notebook\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the code in the above box is doing the same action for every column in the array. So instead of re-assigning every column in the array to a new variable called `culmen_lengths`, `body_mass`, etc, let's use array indexing to loop through the data instead. Notice that the only thing changing when looking at different columns is the *column index*. \n",
    "\n",
    "**Exercise 10:** Change the following code so that it creates a histogram for all columns in the Adelie penguin data, like in the previous block. However, instead of making a new variable for each column called `culmen_lengths`, `body_mass`, etc, use indexing instead. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_indices = [0, 1, 2, 3]\n",
    "\n",
    "for index in column_indices:\n",
    "    data_subset = # Your code here\n",
    "    # Create a histogram \n",
    "    plt.hist(data_subset)\n",
    "    # Show the plot in our jupyter notebook\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Putting it all together: Using a for loop to load, analyze, and plot multiple data files\n",
    "\n",
    "We've now found some interesting things about Adelie penguins. But our original dataset included three different species - Adelie penguins, Chinstrap penguins, and Gentoo penguins. We probably want to run these exact same analyses for each species, and this is a great opportunity to use a `for` loop to make our lives easier. Because all three of our datasets are exactly the same shape and format, we can reuse all of our code that we've already written. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, make a list of each filename that we're interested in analyzing\n",
    "filenames = [\"./data/Lab_02/Adelie_Penguin.csv\", \n",
    "             \"./data/Lab_02/Chinstrap_Penguin.csv\", \n",
    "             \"./data/Lab_02/Gentoo_Penguin.csv\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a list of filenames to analyze, we can turn this into a `for` loop that loads each file and then runs analyses on the file. The code block below has started the process - for each filename, we load in the file data as a variable called `penguin_data`. Note that we're not actually doing anything with the data yet, so we don't see many interesting things being printed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for filename in filenames:\n",
    "    # Load our file data from \"filename\" into a variable called penguin_data\n",
    "    penguin_data = np.loadtxt(fname=filename, delimiter=\",\")\n",
    "    print(\"NOW ANALYZING DATASET: \", filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data loading doesn't seem to have caused any errors, so we'll continue to copy and paste the code we've already written to work with the data. Note that everything we've copied and pasted is code we've already written - but now we're asking Python to run this same code on *all* the data files, instead of just Adelie penguins. For the purposes of this exercise, we'll analyze just the culmen lengths of the dataset, so that we end up with a manageable number of output plots. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for filename in filenames:\n",
    "    # Load our file data from \"filename\" into a variable called penguin_data\n",
    "    penguin_data = np.loadtxt(fname=filename, delimiter=\",\")\n",
    "    print(\"----\")\n",
    "    print(\"NOW ANALYZING DATASET: \", filename)\n",
    "    \n",
    "    # Print the shape of the loaded dataset\n",
    "    data_shape = penguin_data.shape\n",
    "    print(\"Penguin data shape is:\", data_shape)\n",
    "\n",
    "    # Print a warning if the data shape is not what we expect\n",
    "    if (data_shape[0] != 50) or (data_shape[1] != 4):\n",
    "        print(\"Unexpected data shape!\")\n",
    "    else:\n",
    "        print(\"Correct data shape of 50 rows, 4 columns!\")\n",
    "        \n",
    "    # put the culmen lengths for this dataset in a variable called culmen_lengths\n",
    "    culmen_lengths = penguin_data[0:, 0]\n",
    "    print(\"The culmen lengths in this dataset is:\")\n",
    "    print(culmen_lengths)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 11:** Similarly, add in the code you've already written to print the interesting characteristics of the data (mean, median, max, etc.) and create a histogram for each data file that includes the mean and median. Run your final for loop. Which penguin species has the longest mean culmen length? Smallest minimum culmen length? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lab 2 Bonus exercise\n",
    "\n",
    "**Bonus Exercise 1:** Now take the above code and edit it so that we analyze all of the 4 penguin morphology variables, for all of the species. Label the plot axis and title with the appropriate information (penguin species for title, and the morphological variable on the x axis). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}