lists.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lists\n",
    "\n",
    "By [Allison Parrish](https://www.decontextualize.com/)\n",
    "\n",
    "A list is a type of value in Python that represents a sequence of values. The list is a very common and versatile data structure in Python and is used frequently to represent (among other things) tabular data. Here's how you write one out in Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 10, 15, 20, 25, 30]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[5, 10, 15, 20, 25, 30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is: a left square bracket, followed by a series of comma-separated expressions, followed by a right square bracket. Items in a list don't have to be values; they can be more complex expressions as well. Python will evaluate those expressions and put them in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 10, 15, 20, 25, 30]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[5, 2*5, 3*5, 4*5, 5*5, 6*5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lists can have an arbitrary number of values. Here's a list with only one value in it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's a list with no values in it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what happens when we ask Python what type of value a list is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type([1, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a value of type `list`.\n",
    "\n",
    "Like any other kind of Python value, you can assign a list to a variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_numbers = [5, 10, 15, 20, 25, 30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting values out of lists\n",
    "\n",
    "Once we have a list, we might want to get values *out* of the list. You can write a Python expression that evaluates to a particular value in a list using square brackets to the right of your list, with a number representing which value you want, numbered from the beginning (the left-hand side) of the list. Here's an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[5, 10, 15, 20][2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we were to say this expression out loud, it might read, \"I have a list of four things: 5, 10, 15, 20. Give me back the second item in the list.\" Python evaluates that expression to `15`, the second item in the list.\n",
    "\n",
    "Here's what it looks like to use this indexing notation on a list stored in a variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The second item? Am I seeing things. 15 is clearly the third item in the list.\n",
    "\n",
    "You're right---good catch. But for reasons too complicated to go into here, Python (along with many other programming languages!) starts list indexes at 0, instead of 1. So what looks like the third element of the list to human eyes is actually the second element to Python. The first element of the list is accessed using index 0, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[5, 10, 15, 20][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The way I like to conceptualize this is to think of list indexes not as specifying the number of the item you want, but instead specifying how \"far away\" from the beginning of the list to look for that value.\n",
    "\n",
    "If you attempt to use a value for the index of a list that is beyond the end of the list (i.e., the value you use is higher than the last index in the list), Python gives you an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[17], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m my_numbers[\u001b[38;5;241m47\u001b[39m]\n",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "my_numbers[47]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that while the type of a list is `list`, the type of an expression using index brackets to get an item out of the list is the type of whatever was in the list to begin with. To illustrate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(my_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "int"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(my_numbers[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexes can be expressions too\n",
    "\n",
    "The thing that goes inside of the index brackets doesn't have to be a number that you've just typed in there. Any Python expression that evaluates to an integer can go in there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[2 * 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = 3\n",
    "[5, 10, 15, 20][x]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other operations on lists\n",
    "\n",
    "Because lists are so central to Python programming, Python includes a number of built-in functions that allow us to write expressions that evaluate to interesting facts about lists. For example, try putting a list between the parentheses of the `len()` function. It will evaluate to the number of items in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(my_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len([20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len([])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `in` operator checks to see if the value on the left-hand side is in the list on the right-hand side."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "3 in my_numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "15 in my_numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `max()` function will evaluate to the highest value in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "readings = [9, 8, 42, 3, -17, 2]\n",
    "max(readings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and the `min()` function will evaluate to the lowest value in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-17"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min(readings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `sum()` function evaluates to the sum of all values in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum([2, 4, 6, 8, 80])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the `sorted()` function evaluates to a copy of the list, sorted from smallest value to largest value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-17, 2, 3, 8, 9, 42]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(readings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Negative indexes\n",
    "\n",
    "If you use `-1` as the value inside of the brackets, something interesting happens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fib = [1, 1, 2, 3, 5]\n",
    "fib[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expression evaluates to the *last* item in the list. This is essentially the same thing as the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fib[len(fib) - 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... except easier to write. In fact, you can use any negative integer in the index brackets, and Python will count that many items from the end of the list, and evaluate the expression to that item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fib[-3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the value in the brackets would \"go past\" the beginning of the list, Python will raise an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[34], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m fib[\u001b[38;5;241m-\u001b[39m\u001b[38;5;241m14\u001b[39m]\n",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "fib[-14]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating lists with `range()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expression `list(range(n))` returns a list from 0 up to (but not including) `n`. This is helpful when you just want numbers in a sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(range(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify where the list should start and end by supplying two parameters to the call to `range`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-10, -9, -8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(range(-10, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List slices\n",
    "\n",
    "The index bracket syntax explained above allows you to write an expression that evaluates to a particular item in a list, based on its position in the list. Python also has a powerful way for you to write expressions that return a *section* of a list, starting from a particular index and ending with another index. In Python parlance we'll call this section a *slice*.\n",
    "\n",
    "Writing an expression to get a slice of a list looks a lot like writing an expression to get a single value. The difference is that instead of putting one number between square brackets, we put *two* numbers, separated by a colon. The first number tells Python where to begin the slice, and the second number tells Python where to end it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 6, 10]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[4, 5, 6, 10, 12, 15][1:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the value after the colon specifies at which index the slice should end, but the slice does *not* include the value at that index. (You can tell how long the slice will be by subtracting the value before the colon from the value after it.)\n",
    "\n",
    "Also note that—as always!—any expression that evaluates to an integer can be used for either value in the brackets. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[10, 12]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = 3\n",
    "[4, 5, 6, 10, 12, 15][x:x+2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, note that the type of a slice is `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(my_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(my_numbers[1:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Omitting slice values\n",
    "\n",
    "Because it's so common to use the slice syntax to get a list that is either a slice starting at the beginning of the list or a slice ending at the end of the list, Python has a special shortcut. Instead of writing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 5, 6]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[4, 5, 6, 10, 12, 15][0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can leave out the `0` and write this instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 5, 6]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[4, 5, 6, 10, 12, 15][:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, if you wanted a slice that starts at index 4 and goes to the end of the list, you might write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[12, 15]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[4, 5, 6, 10, 12, 15][4:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting the last two items in `my_numbers`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 10]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Negative index values in slices\n",
    "\n",
    "Now for some tricky stuff: You can use negative index values in slice brackets as well! For example, to get a slice of a list from the fourth-to-last element of the list up to (but not including) the second-to-last element of the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[6, 10]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[4, 5, 6, 10, 12, 15][-4:-2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the last three elements of the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 5, 6]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[4, 5, 6, 10, 12, 15][:-3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All items from `my_numbers` from the third item from the end of the list upto the end of the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[20, 25, 30]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[-3:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lists within lists\n",
    "\n",
    "So far we've seen lists that contain integers and floating-point numbers. But we're not limited to just those types! Importantly, lists can themselves contain... other lists. Lists within lists is one of the ways Python represents a matrix of values, like a spreadsheet. Here's what it looks like when you have lists inside of a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1, 2, 3, 4], [5, 10, 15, 20], [100, 200, 300, 400]]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[[1, 2, 3, 4], [5, 10, 15, 20], [100, 200, 300, 400]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whew, that's a lot of brackets! This is a list that has three items, each of which is itself a list containing four items. One way to visualize this list is to think of it as a table or spreadsheet that looks like this:\n",
    "\n",
    "col 0|col 1|col 2|col 3\n",
    "-----|-----|-----|-----\n",
    "1    | 2   | 3   | 4\n",
    "5    | 10  | 15  | 20\n",
    "100  | 200 | 300 | 400\n",
    "\n",
    "Using the `len()` function on this list returns the number of items in the outer list, or the number of \"rows\" in the \"table\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len([[1, 2, 3, 4], [5, 10, 15, 20], [100, 200, 300, 400]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It'll be clearer if we assign our list-of-lists to a variable first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [[1, 2, 3, 4], [5, 10, 15, 20], [100, 200, 300, 400]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result of getting an item at an index in a list of lists is the list at that index (i.e., the row):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 10, 15, 20]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a single item from the resulting list all in the same expression, use a second pair of square brackets, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[1][3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whoa, weird! But it works. If you have a variable `x` that is a list of lists, you can get the value in column `col` in row `row` with the following expression:\n",
    "\n",
    "    x[row][col]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List comprehensions: Applying transformations to lists\n",
    "\n",
    "A very common task in both data analysis and computer programming is applying some operation to every item in a list (e.g., scaling the numbers in a list by a fixed factor), or to create a copy of a list with only those items that match a particular criterion (e.g., eliminating values that fall below a certain threshold). Python has a succinct syntax, called a *list comprehension*, which allows you to easily write expressions that transform and filter lists.\n",
    "\n",
    "A list comprehension has a few parts:\n",
    "\n",
    "- a *source list*, or the list whose values will be transformed or filtered;\n",
    "- a *predicate expression*, to be evaluated for every item in the list; \n",
    "- (optionally) a *membership expression* that determines whether or not an item in the source list will be included in the result of evaluating the list comprehension, based on whether the expression evaluates to `True` or `False`; and\n",
    "- a *temporary variable name* by which each value from the source list will be known in the predicate expression and membership expression.\n",
    "\n",
    "These parts are arranged like so:\n",
    "\n",
    "> `[` *predicate expression* `for` *temporary variable name* `in` *source list* `if` *membership expression* `]`\n",
    "\n",
    "The words `for`, `in`, and `if` are a part of the syntax of the expression. They don't mean anything in particular (and in fact, they do completely different things in other parts of the Python language). You just have to spell them right and put them in the right place in order for the list comprehension to work.\n",
    "\n",
    "Here's an example, returning the squares of integers zero up to ten:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x * x for x in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example above, `x*x` is the predicate expression; `x` is the temporary variable name; and `[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]` is the source list. There's no membership expression in this example, so we omit it (and the word `if`).\n",
    "\n",
    "There's nothing special about the variable `x`; it's just a name that we chose. We could easily choose any other temporary variable name, as long as we use it in the predicate expression as well. Below, I use the name of one of my cats as the temporary variable name, and the expression evaluates the same way it did with `x`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[shumai * shumai for shumai in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the type of the value that a list comprehension evaluates to is itself type `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type([x * x for x in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expression we supply for the source list component of a list comprehension doesn't have to be a list that you've written out by hand. It can also be a variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numbers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
    "[x * x for x in numbers]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... or it can be the result of some other expression that evaluates to a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x * x for x in range(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've used the expression `x * x` as the predicate expression in the examples above, but you can use any expression you want. For example, to scale the values of a list by 0.5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x * 0.5 for x in range(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, the expression in the list comprehension can just be the temporary variable itself, in which case the list comprehension will simply evaluate to a copy of the original list: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x for x in range(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't technically even need to use the temporary variable in the predicate expression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[42, 42, 42, 42, 42, 42, 42, 42, 42, 42]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[42 for x in range(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Bonus exercise: Write a list comprehension for the list `range(5)` that evaluates to a list where every value has been multiplied by two (i.e., the expression would evaluate to `[0, 2, 4, 6, 8]`). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The membership expression\n",
    "\n",
    "As indicated above, you can include an expression at the end of the list comprehension to determine whether or not the item in the source list will be evaluated and included in the resulting list. One way, for example, of including only those values from the source list that are greater than or equal to five:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[25, 36, 49, 64, 81]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x*x for x in range(10) if x >= 5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making lists from other kinds of data\n",
    "\n",
    "We've learned how lists work, and we've learned some basic techniques for getting data from lists and turning lists into other lists. But so far it's been pretty abstract—we've been working with lists of numbers we've been typing in by hand, instead of with, you know, actual data fetched from some source. The reason for this is that getting data from real-world sources is... well, it's *hard*, and learning how to do that constitutes much of the content of this course.\n",
    "\n",
    "Data files from real world sources usually come in a series of bytes—basically, a long sequence of numbers that correlate to how data is stored on disk or transmitted over the network. Our job as data mungers is to figure out how to \"parse\" this data (\"parse\" is used here loosely, in its colloquial meaning), transforming it from its \"raw\" form into actual Python data structures, like integers and lists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Strings and lists\n",
    "\n",
    "Usually that \"raw\" form comes to us as a string. Consequently, Python has a grand variety of tools to transform strings into lists and vice versa.\n",
    "\n",
    "First off, you can turn a string into a list of its component characters by passing it to `list()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "message = \"grumpy pout\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['g', 'r', 'u', 'm', 'p', 'y', ' ', 'p', 'o', 'u', 't']"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The letters in a string in alphabetical order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' ', 'g', 'm', 'o', 'p', 'p', 'r', 't', 'u', 'u', 'y']"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(list(message))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting strings\n",
    "\n",
    "Strings have a `.split()` method that evaluates to a list consisting of the parts of that string, as parsed using a smaller string. With no parameter, the `.split()` method defaults to splitting on whitespace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this', 'is', 'a', 'test']"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"this is a test\".split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notably, while the `type` of a string is `str`, the type of the result of `split()` is `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(\"this is a test\".split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the string in question has some delimiter in it other than whitespace that we want to use to separate the fields in the resulting list, we can put a string with that delimiter inside the parentheses of the `split()` method. Maybe you can tell where I'm going with this at this point!\n",
    "\n",
    "### From string to list of numbers: an example\n",
    "\n",
    "For example, I happen to have here a string that represents the total points scored by LeBron James in each of his NBA games in the 2013-2014 regular season.\n",
    "\n",
    "> 17,25,26,25,35,18,25,33,39,30,13,21,22,35,28,27,26,23,21,21,24,17,25,30,24,18,38,19,33,26,26,15,30,32,32,36,25,21,34,30,29,27,18,34,30,24,31,13,37,36,42,33,31,20,61,22,19,17,23,19,21,24,43,15,25,32,38,17,13,32,17,34,38,29,37,36,27\n",
    "\n",
    "You can either cut-and-paste this string from the notes, or see a file on github with these values [here](https://gist.githubusercontent.com/aparrish/56ea528159c97b085a34/raw/8406bdd866101cf64347349558d9a806c82aceb7/scores.txt).\n",
    "\n",
    "Now if I just cut-and-pasted this string into a variable and tried to call list functions on it, I wouldn't get very helpful responses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'9'"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_str = \"17,25,26,25,35,18,25,33,39,30,13,21,22,35,28,27,26,23,21,21,24,17,25,30,24,18,38,19,33,26,26,15,30,32,32,36,25,21,34,30,29,27,18,34,30,24,31,13,37,36,42,33,31,20,61,22,19,17,23,19,21,24,43,15,25,32,38,17,13,32,17,34,38,29,37,36,27\"\n",
    "max(raw_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is wrong—we know that LeBron James scored more than nine points in his highest scoring game. The `max()` function clearly does strange things when we give it a string instead of a list. The reason for this is that all Python knows about a string is that it's a *series of characters*. It's easy for a human to look at this string and think, \"Hey, that's a list of numbers!\" But Python doesn't know that. We have to explicitly \"translate\" that string into the kind of data we want Python to treat it as.\n",
    "\n",
    "> Bonus advanced exercise: Take a guess as to why, specifically, Python evaluates `max(raw_str)` to `9`. Hint: what's the result of `type(max(raw_str))`?\n",
    "\n",
    "What we want to do, then, is find some way to convert this string that *represents* integer values into an actual Python list of integer values. We'll start by splitting this string into a list, using the `split()` method, passing `\",\"` as a parameter so it splits on commas instead of on whitespace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['17',\n",
       " '25',\n",
       " '26',\n",
       " '25',\n",
       " '35',\n",
       " '18',\n",
       " '25',\n",
       " '33',\n",
       " '39',\n",
       " '30',\n",
       " '13',\n",
       " '21',\n",
       " '22',\n",
       " '35',\n",
       " '28',\n",
       " '27',\n",
       " '26',\n",
       " '23',\n",
       " '21',\n",
       " '21',\n",
       " '24',\n",
       " '17',\n",
       " '25',\n",
       " '30',\n",
       " '24',\n",
       " '18',\n",
       " '38',\n",
       " '19',\n",
       " '33',\n",
       " '26',\n",
       " '26',\n",
       " '15',\n",
       " '30',\n",
       " '32',\n",
       " '32',\n",
       " '36',\n",
       " '25',\n",
       " '21',\n",
       " '34',\n",
       " '30',\n",
       " '29',\n",
       " '27',\n",
       " '18',\n",
       " '34',\n",
       " '30',\n",
       " '24',\n",
       " '31',\n",
       " '13',\n",
       " '37',\n",
       " '36',\n",
       " '42',\n",
       " '33',\n",
       " '31',\n",
       " '20',\n",
       " '61',\n",
       " '22',\n",
       " '19',\n",
       " '17',\n",
       " '23',\n",
       " '19',\n",
       " '21',\n",
       " '24',\n",
       " '43',\n",
       " '15',\n",
       " '25',\n",
       " '32',\n",
       " '38',\n",
       " '17',\n",
       " '13',\n",
       " '32',\n",
       " '17',\n",
       " '34',\n",
       " '38',\n",
       " '29',\n",
       " '37',\n",
       " '36',\n",
       " '27']"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str_list = raw_str.split(\",\")\n",
    "str_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good so far. What does `max()` have to say about it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'61'"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max(str_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This.. works. (But only by accident—see below.) But what if we wanted to find the total number of points scored by LBJ? We should be able to do something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "unsupported operand type(s) for +: 'int' and 'str'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[77], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;28msum\u001b[39m(str_list)\n",
      "\u001b[0;31mTypeError\u001b[0m: unsupported operand type(s) for +: 'int' and 'str'"
     ]
    }
   ],
   "source": [
    "sum(str_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... but we get an error. Why this error? The reason lies in what kind of data is in our list. We can check the data type of an element of the list with the `type()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(str_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A-ha! The type is `str`. So the error message we got before (`unsupported operand type(s) for +: 'int' and 'str'`) is Python's way of telling us, \"You gave me a list of strings and then asked me to add them all together. I'm not sure what I can do for you.\"\n",
    "\n",
    "So there's one step left in our process of \"converting\" our \"raw\" string, consisting of comma-separated numbers, into a list of numbers. What we have is a list of strings; what we want is a list of numbers. Fortunately, we know how to write an expression to transform one list into another list, applying an expression to each member of the list along the way—it's called a list comprehension. Equally fortunately, we know how to write an expression that converts a string representing an integer into an actual integer (`int()`). Here's how to write that expression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[17,\n",
       " 25,\n",
       " 26,\n",
       " 25,\n",
       " 35,\n",
       " 18,\n",
       " 25,\n",
       " 33,\n",
       " 39,\n",
       " 30,\n",
       " 13,\n",
       " 21,\n",
       " 22,\n",
       " 35,\n",
       " 28,\n",
       " 27,\n",
       " 26,\n",
       " 23,\n",
       " 21,\n",
       " 21,\n",
       " 24,\n",
       " 17,\n",
       " 25,\n",
       " 30,\n",
       " 24,\n",
       " 18,\n",
       " 38,\n",
       " 19,\n",
       " 33,\n",
       " 26,\n",
       " 26,\n",
       " 15,\n",
       " 30,\n",
       " 32,\n",
       " 32,\n",
       " 36,\n",
       " 25,\n",
       " 21,\n",
       " 34,\n",
       " 30,\n",
       " 29,\n",
       " 27,\n",
       " 18,\n",
       " 34,\n",
       " 30,\n",
       " 24,\n",
       " 31,\n",
       " 13,\n",
       " 37,\n",
       " 36,\n",
       " 42,\n",
       " 33,\n",
       " 31,\n",
       " 20,\n",
       " 61,\n",
       " 22,\n",
       " 19,\n",
       " 17,\n",
       " 23,\n",
       " 19,\n",
       " 21,\n",
       " 24,\n",
       " 43,\n",
       " 15,\n",
       " 25,\n",
       " 32,\n",
       " 38,\n",
       " 17,\n",
       " 13,\n",
       " 32,\n",
       " 17,\n",
       " 34,\n",
       " 38,\n",
       " 29,\n",
       " 37,\n",
       " 36,\n",
       " 27]"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[int(x) for x in str_list]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's double-check that the values in this list are, in fact, integers, by spot-checking the first item in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "int"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type([int(x) for x in str_list][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hey, voila! Now we'll assign that list to a variable, for the sake of convenience, and then check to see if `sum()` works how we expect it to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2089"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_list = [int(x) for x in str_list]\n",
    "sum(int_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! 2089 points in one season! Good work, King James."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making changes to lists\n",
    "\n",
    "Often we'll want to make changes to a list after we've created it---for\n",
    "example, we might want to append elements to the list, remove elements from\n",
    "the list, or change the order of elements in the list. Python has a number\n",
    "of methods for facilitating these operations.\n",
    "\n",
    "The first method we'll talk about is `.append()`, which adds an item on to\n",
    "the end of an existing list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['flour', 'milk', 'eggs', 'sugar']"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ingredients = [\"flour\", \"milk\", \"eggs\"]\n",
    "ingredients.append(\"sugar\")\n",
    "ingredients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that invoking the `.append()` method doesn't itself evaluate to\n",
    "anything! (Technically, it evaluates to a special value of type `None`.)\n",
    "Unlike many of the methods and syntactic constructions we've looked at so far,\n",
    "the `.append()` method changes the underlying value---it doesn't return a\n",
    "new value that is a copy with changes applied.\n",
    "\n",
    "There are two methods to facilitate removing values from a list: `.pop()` and\n",
    "`.remove()`. The `.remove()` method removes from the list the first value that\n",
    "matches the value in the parentheses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['milk', 'eggs', 'sugar']"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "ingredients.remove(\"flour\")\n",
    "ingredients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note that `.remove()`, like `.append()` doesn't evaluate to anything---it\n",
    "changes the list itself.)\n",
    "\n",
    "The `.pop()` method works slightly differently: give it an expression that\n",
    "evaluates to an integer, and it evaluates to the expression at the index\n",
    "named by the integer. But it also has a side effect: it *removes* that item\n",
    "from the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['flour', 'eggs', 'sugar']"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "ingredients.pop(1)\n",
    "ingredients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> EXERCISE: What happens when you try to `.pop()` a value from a list at an index that doesn't exist in the list? What happens you try to `.remove()` an item from a list if that item isn't in that list to begin with?\n",
    "\n",
    "> ANOTHER EXERCISE: Write an expression that `.pop()`s the second-to-last item from a list. SPOILER: <span style=\"background: black;\">(Did you guess that you could use negative indexing with `.pop()`?</span>\n",
    "\n",
    "The `.sort()` and `.reverse()` methods do exactly the same thing as their\n",
    "function counterparts `sorted()` and `reversed()`, with the only difference\n",
    "being that the methods don't evaluate to anything, instead opting to change\n",
    "the list in-place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['eggs', 'flour', 'milk', 'sugar']"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "ingredients.sort()\n",
    "ingredients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['sugar', 'eggs', 'milk', 'flour']"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "ingredients.reverse()\n",
    "ingredients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lists and randomness\n",
    "\n",
    "Python's `random` library provides several helpful functions for performing\n",
    "chance operations on lists. The first is `shuffle`, which takes a list and\n",
    "randomly shuffles its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['milk', 'eggs', 'sugar', 'flour']"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "random.shuffle(ingredients)\n",
    "ingredients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second is `choice`, which returns a single random element from list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sugar'"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "random.choice(ingredients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the `sample` function returns a list of values, selected at random,\n",
    "from a list. The `sample` function takes two parameters: the first is a list,\n",
    "and the second is how many items should be in the resulting list of randomly\n",
    "selected values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['sugar', 'milk']"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "random.sample(ingredients, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterating over lists with `for`\n",
    "\n",
    "The list comprehension syntax discussed earlier is very powerful: it allows you to succinctly transform one list into another list by thinking in terms of filtering and modification. But sometimes your primary goal isn't to make a new list, but simply to perform a set of operations on an existing list.\n",
    "\n",
    "Let's say that you want to print every string in a list. Here's a short text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"it was the best of times, it was the worst of times\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make a list of all the words in the text by splitting on whitespace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = text.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, we can see what's in the list simply by evaluating the variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['it',\n",
       " 'was',\n",
       " 'the',\n",
       " 'best',\n",
       " 'of',\n",
       " 'times,',\n",
       " 'it',\n",
       " 'was',\n",
       " 'the',\n",
       " 'worst',\n",
       " 'of',\n",
       " 'times']"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But let's say that we want to print out each word on a separate line, without any of Python's weird punctuation. In other words, I want the output to look like:\n",
    "\n",
    "    it\n",
    "    was\n",
    "    the\n",
    "    best\n",
    "    of\n",
    "    times,\n",
    "    it\n",
    "    was\n",
    "    the\n",
    "    worst\n",
    "    of\n",
    "    times\n",
    "    \n",
    "But how can this be accomplished? We know that the `print()` function can display an individual string in this manner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello\n"
     ]
    }
   ],
   "source": [
    "print(\"hello\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what we need, clearly, is a way to call the `print()` function with every item of the list. We could do this by writing a series of `print()` statements, one for every item in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it\n",
      "was\n",
      "the\n",
      "best\n",
      "of\n",
      "times,\n",
      "it\n",
      "was\n",
      "the\n",
      "worst\n",
      "of\n",
      "times\n"
     ]
    }
   ],
   "source": [
    "print(words[0])\n",
    "print(words[1])\n",
    "print(words[2])\n",
    "print(words[3])\n",
    "print(words[4])\n",
    "print(words[5])\n",
    "print(words[6])\n",
    "print(words[7])\n",
    "print(words[8])\n",
    "print(words[9])\n",
    "print(words[10])\n",
    "print(words[11])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice, but there are some problems with this approach:\n",
    "\n",
    "1. It's kind of verbose—we're doing exactly the same thing multiple times, only with slightly different expressions. Surely there's an easier way to tell the computer to do this?\n",
    "2. It doesn't scale. What if we wrote a program that we want to produce hundreds or thousands of lines. Would we really need to write a `print` statement for each of those expressions?\n",
    "3. It requires us to know how many items are going to end up in the list to begin with."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Things are looking grim! But there's hope. Performing the same operation on all items of a list is an extremely common task in computer programming. So common,\n",
    "that Python has some built-in syntax to make the task easy: the `for` loop.\n",
    "\n",
    "Here's how a `for` loop looks:\n",
    "\n",
    "    for tempvar in sourcelist:\n",
    "        statements\n",
    "\n",
    "The words `for` and `in` just have to be there---that's how Python knows it's\n",
    "a `for` loop. Here's what each of those parts mean.\n",
    "\n",
    "* *tempvar*: A name for a variable. Inside of the for loop, this variable will contain the current item of the list.\n",
    "* *sourcelist*: This can be any Python expression that evaluates to a list---a variable that contains a list, or a list slice, or even a list literal that you just type right in!\n",
    "* *statements*: One or more Python statements. Everything tabbed over underneath the `for` will be executed once for each item in the list. The statements tabbed over underneath the `for` line are called the *body* of the loop.\n",
    "\n",
    "Here's what the `for` loop for printing out every item in a list might look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it\n",
      "was\n",
      "the\n",
      "best\n",
      "of\n",
      "times,\n",
      "it\n",
      "was\n",
      "the\n",
      "worst\n",
      "of\n",
      "times\n"
     ]
    }
   ],
   "source": [
    "for item in words:\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variable name `item` is arbitrary. You can pick whatever variable name you like, as long as you're consistent about using the same variable name in the body of the loop. If you wrote out this loop in a long-hand fashion, it might look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it\n",
      "was\n",
      "the\n",
      "best\n"
     ]
    }
   ],
   "source": [
    "item = words[0]\n",
    "print(item)\n",
    "item = words[1]\n",
    "print(item)\n",
    "item = words[2]\n",
    "print(item)\n",
    "item = words[3]\n",
    "print(item)\n",
    "# etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, the body of the loop can have more than one statement, and you can assign values to variables inside the loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IT\n",
      "WAS\n",
      "THE\n",
      "BEST\n",
      "OF\n",
      "TIMES,\n",
      "IT\n",
      "WAS\n",
      "THE\n",
      "WORST\n",
      "OF\n",
      "TIMES\n"
     ]
    }
   ],
   "source": [
    "for item in words:\n",
    "    yelling = item.upper()\n",
    "    print(yelling)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also include other kinds of nested statements inside the `for` loop, like `if/else`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IT\n",
      "   was\n",
      "   the\n",
      "best\n",
      "OF\n",
      "times,\n",
      "IT\n",
      "   was\n",
      "   the\n",
      "worst\n",
      "OF\n",
      "times\n"
     ]
    }
   ],
   "source": [
    "for item in words:\n",
    "    if len(item) == 2:\n",
    "        print(item.upper())\n",
    "    elif len(item) == 3:\n",
    "        print(\"   \" + item)\n",
    "    else:\n",
    "        print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This structure is called a \"loop\" because when Python reaches the end of the statements in the body, it \"loops\" back to the beginning of the body, and executes the same statements again (this time with the next item in the list). \n",
    "\n",
    "Python programmers tend to use `for` loops most often when the problem would otherwise be too tricky or complicated to solve using a list comprehension. It's easy to paraphrase any list comprehension in `for` loop syntax. For example, this list comprehension, which evaluates to a list of the squares of even integers from 1 to 25:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 16, 36, 64, 100, 144, 196, 256, 324, 400, 484, 576]"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x * x for x in range(1, 26) if x % 2 == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can rewrite this list comprehesion as a `for` loop by starting out with an empty list, then appending an item to the list inside the loop. The source list remains the same:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 16, 36, 64, 100, 144, 196, 256, 324, 400, 484, 576]"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = []\n",
    "for x in range(1, 26):\n",
    "    if x % 2 == 0:\n",
    "        result.append(x * x)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Join: Making strings from lists\n",
    "\n",
    "Once we've created a list of words, it's a common task to want to take that\n",
    "list and \"glue\" it back together, so it's a single string again, instead of\n",
    "a list. So, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hydrogen, and helium, and lithium, and beryllium, and boron'"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element_list = [\"hydrogen\", \"helium\", \"lithium\", \"beryllium\", \"boron\"]\n",
    "glue = \", and \"\n",
    "glue.join(element_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.join()` method needs a \"glue\" string to the left of it---this is the\n",
    "string that will be placed in between the list elements. In the parentheses\n",
    "to the right, you need to put an expression that evaluates to a list. Very\n",
    "frequently with `.join()`, programmers don't bother to assign the \"glue\"\n",
    "string to a variable first, so you end up with code that looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'this is a test'"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words = [\"this\", \"is\", \"a\", \"test\"]\n",
    "\" \".join(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we're working with `.split()` and `.join()`, our workflow usually looks\n",
    "something like this:\n",
    "\n",
    "1. Split a string to get a list of units (usually words).\n",
    "2. Use some of the list operations discussed above to modify or slice the list.\n",
    "3. Join that list back together into a string.\n",
    "4. Do something with that string (e.g., print it out).\n",
    "\n",
    "With this in mind, here's a program that splits a string into words, randomizes the order of the words, then prints out the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'stormy it dark a and was night'"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"it was a dark and stormy night\"\n",
    "words = text.split()\n",
    "random.shuffle(words)\n",
    "' '.join(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> EXERCISE: Write a Python command-line program that prints out the lines of a text file in random order."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "We've put down the foundation today for you to become fluent in Python's very powerful and super-convenient syntax for lists. We've also done a bit of data parsing and analysis! Pretty good for day one."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}