notes/programming/03_python_plus.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "title: \"More Python: Poetry, Lists, Loops\"\n",
    "date: 2024-01-30\n",
    "categories: \n",
    "    - python\n",
    "filters:\n",
    "    - codeblocklabel\n",
    "highlight-style: github\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This lesson has two goals:\n",
    "\n",
    "1. Begin walking you through processing text in Python, with an eye towards our next computational linguistics topic: ngram models.\n",
    "2. To introduce you to additional datastructures that we didn't get to cover in the original python workthrough.\n",
    "\n",
    "## Finding and reading in files\n",
    "\n",
    "In order to read in a file, first we need to tell python *where* a file is. We'll do this with `pathlib.Path()`. For a recap of what paths *are*, see [Understanding Directories](../concepts/02_undersdanding-paths.qmd)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's just call `Path()` and see what it gives us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('.')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "here = Path()\n",
    "here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some detail of what gets printed out for `here` will depend on your operating system, but one thing that should be the same across the board is that `'.'` is a placeholder meaning \"the dirtory you're currently in.\n",
    "\n",
    "To see the full path to your current directory, we'll use the `.resolve()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/workspaces/Lin511-2024.github.io/notes/programming')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "here.resolve()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What this full path looks like *really* depends on where you are running your code.\n",
    "\n",
    "### Getting directory contents\n",
    "\n",
    "To get the full contents of a directory, we'll use the the `.glob()` method. To get readable output, we need to wrap `list()` around everything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('03_python_plus.ipynb'),\n",
       " PosixPath('assets'),\n",
       " PosixPath('_02_Paths.ipynb'),\n",
       " PosixPath('01_Python_basics.ipynb'),\n",
       " PosixPath('00_setup.qmd'),\n",
       " PosixPath('poems'),\n",
       " PosixPath('02_regex.ipynb')]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(\n",
    "    here.glob(pattern = \"*\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## *Lists (a new data structure)\n",
    "\n",
    "The code above created a \"list\", which is, like the name suggests, a list of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "here_list = list(\n",
    "    here.glob(pattern = \"*\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create our own lists by hand, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_list = [\"a\", \"b\", \"c\", \"d\", \"e\"] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Doing things with lists\n",
    "\n",
    "#### Getting their length\n",
    "\n",
    "To get the length of anything, including lists, `len()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(here_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(my_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Getting a value from a list\n",
    "\n",
    "To get a value out of a list in python, we need to [\"index\"](../concepts/00_glossary.qmd#index-verb) it. To get just the first value from a list, we'll use the index `0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('03_python_plus.ipynb')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "here_list[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_list[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a range of values, we'll use a [\"slice\"](../concepts/00_glossary.qmd#slice-verb) operator `:`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'c']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_list[0:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['b', 'c', 'd']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_list[1:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to think about indices\n",
    "\n",
    "The index for `'a'` is 0, and the index for `'c'` is 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_list.index(\"a\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_list.index(\"c\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But to get out every value between `'a'` and `'c'`, we need to use `0:3`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'c']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_list[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The way to conceptualize these numeric indices is that they come *between* the values in a list.\n",
    "\n",
    "![Index locations](assets/0_index.png){width=80%}\n",
    "\n",
    "When we give just a single number as an index, we get back the value immediately to the right of the index.\n",
    "\n",
    "![Single value index](assets/1_index.png){width=80%}\n",
    "\n",
    "When we give an index slice, we get back every value between the start of the slice and the end.\n",
    "\n",
    "![Slice indexing](assets/2_index.png){width=80%}\n",
    "\n",
    "\n",
    "##### Negative indexing\n",
    "\n",
    "Sometimes, you know you'll want the last, or second to last value from a list, so there are also negative indicies defined.\n",
    "\n",
    "![Negative indexing](assets/3_index.png){width=80%}\n",
    "\n",
    "These work in the same way. A single index returns the value immediately to the left, a slice returns the values between the first and the second.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'e'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_list[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['c', 'd']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_list[-3:-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['d', 'e']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_list[-2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading in a text file\n",
    "\n",
    "I've put two poems in the `poems` directory right next to the current notebook. We can get a list of them by \n",
    "\n",
    "- adding `poems/` to the `here` path\n",
    "- Globbing the new path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('poems/the_tiger.txt'), PosixPath('poems/this_is_just_to_say.txt')]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poems_dir = here.joinpath(\"poems\")\n",
    "poems_files = list(poems_dir.glob(\"*\"))\n",
    "poems_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read in `poems/this_is_just_to_say.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('poems/this_is_just_to_say.txt')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plums_path = poems_files[1]\n",
    "plums_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right now `plums_path` just contains information about *where* the file is. In order to read its contents into python, we need to\n",
    "\n",
    "1. Open it.\n",
    "2. Read it.\n",
    "3. Close it.\n",
    "\n",
    "We'll manage all three steps with the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This Is Just To Say\n",
      "By William Carlos Williams\n",
      "\n",
      "I have eaten\n",
      "the plums\n",
      "that were in\n",
      "the icebox\n",
      "\n",
      "and which\n",
      "you were probably\n",
      "saving\n",
      "for breakfast\n",
      "\n",
      "Forgive me\n",
      "they were delicious\n",
      "so sweet\n",
      "and so cold\n"
     ]
    }
   ],
   "source": [
    "plums_poem = plums_path.read_text()\n",
    "print(plums_poem)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Doing things to strings\n",
    "Right now, `plums_poem` is just one long string. When we print it out, we get some nice formatting, but if we look at the unformatted output, you'll see it's one long string with newline characters: `\\n`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This Is Just To Say\\nBy William Carlos Williams\\n\\nI have eaten\\nthe plums\\nthat were in\\nthe icebox\\n\\nand which\\nyou were probably\\nsaving\\nfor breakfast\\n\\nForgive me\\nthey were delicious\\nso sweet\\nand so cold'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plums_poem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start doing computational linguistics things to the poem, we'll need to start splitting it up into pieces, like separate lines, or separate words.\n",
    "\n",
    "### Upper and Lower case\n",
    "\n",
    "One thing that can make life a little easier is to [\"case fold\"](../concepts/00_glossary.qmd#case-folding) text data, which we can do with the `.lower()` and `.upper()` methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this is just to say\n",
      "by william carlos williams\n"
     ]
    }
   ],
   "source": [
    "plums_lower = plums_poem.lower()\n",
    "print(plums_lower[0:46])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting into lines\n",
    "We can also split the poem into separate lines with the `.split()` method, which will return a list with the poem split by whatever value we give it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['This Is Just To Say',\n",
       " 'By William Carlos Williams',\n",
       " '',\n",
       " 'I have eaten',\n",
       " 'the plums',\n",
       " 'that were in',\n",
       " 'the icebox',\n",
       " '',\n",
       " 'and which',\n",
       " 'you were probably',\n",
       " 'saving',\n",
       " 'for breakfast',\n",
       " '',\n",
       " 'Forgive me',\n",
       " 'they were delicious',\n",
       " 'so sweet',\n",
       " 'and so cold']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# split according to newlines\n",
    "plums_lines = plums_poem.split(\"\\n\")\n",
    "plums_lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can get each individual line by indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This Is Just To Say'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plums_lines[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looping through the poem\n",
    "\n",
    "Above, I ran `.split()` on the non-case folded poem. What if I wanted to lowercase each line of the poem now, *after* the fact. We can use a `for` loop to do this. Let's build up to it.\n",
    "\n",
    "### for-loop skeleton"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "xXx This Is Just To Say xXx\n",
      "xXx By William Carlos Williams xXx\n",
      "xXx  xXx\n",
      "xXx I have eaten xXx\n",
      "xXx the plums xXx\n",
      "xXx that were in xXx\n",
      "xXx the icebox xXx\n",
      "xXx  xXx\n",
      "xXx and which xXx\n",
      "xXx you were probably xXx\n",
      "xXx saving xXx\n",
      "xXx for breakfast xXx\n",
      "xXx  xXx\n",
      "xXx Forgive me xXx\n",
      "xXx they were delicious xXx\n",
      "xXx so sweet xXx\n",
      "xXx and so cold xXx\n"
     ]
    }
   ],
   "source": [
    "for line in plums_lines:\n",
    "    myspace_line = \"xXx \" + line + \" xXx\"\n",
    "    print(myspace_line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's unpack the first line of the for loop:\n",
    "\n",
    "`for line in plums_line:`\n",
    "\n",
    "This tells python to take each value in `plums_line`, assign it to a new variable called `line`, then *do* something. After it has *done* something, it goes and grabs the next value from `plums_line`, assigns it to a variable called `line`, and starts over.\n",
    "\n",
    ":::{.callout-important}\n",
    "## Some common misconceptions\n",
    "\n",
    "1. What we tell python to name the variable it creates does *not* affect the way the for loop works. We called it `line` to make the code readable. If we really wanted to pull out every noun from the poem, and said `for noun in plums_lines:`, it is *not* going to pull out every noun."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This Is Just To Say\n",
      "By William Carlos Williams\n"
     ]
    }
   ],
   "source": [
    "for noun in plums_lines[0:2]:\n",
    "    print(noun)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. *What* we do inside of the loop doen't have to involve the `line` variable at all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n",
      "plum\n"
     ]
    }
   ],
   "source": [
    "for line in plums_lines:\n",
    "    print(\"plum\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. What the for-loop did does not get saved anywhere. The only \"state\" that gets preserved at all is the final one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xXx and so cold xXx'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myspace_line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This last point is super important. To save, or remember, what happened inside of a for-loop, you need to do two things.\n",
    "\n",
    "1. Create a variable *outside* of the loop. \n",
    "2. Update the variable *inside* of the loop.\n",
    "\n",
    "For example, if we wanted to get the total number of characters in the peom, we'd need to\n",
    "\n",
    "1. Create a collector variable *outside* of the loop.\n",
    "2. Add update its total value *inside* of the loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The total number of chars is now 0\n",
      "The total number of chars is now 19\n",
      "The total number of chars is now 45\n",
      "The total number of chars is now 45\n",
      "The total number of chars is now 57\n",
      "The total number of chars is now 66\n",
      "The total number of chars is now 78\n",
      "The total number of chars is now 88\n",
      "The total number of chars is now 88\n",
      "The total number of chars is now 97\n",
      "The total number of chars is now 114\n",
      "The total number of chars is now 120\n",
      "The total number of chars is now 133\n",
      "The total number of chars is now 133\n",
      "The total number of chars is now 143\n",
      "The total number of chars is now 162\n",
      "The total number of chars is now 170\n",
      "The total number of chars is now 181\n",
      "The final number of chars is 181\n"
     ]
    }
   ],
   "source": [
    "total_chars = 0\n",
    "\n",
    "print(f\"The total number of chars is now {total_chars}\")\n",
    "\n",
    "for line in plums_lines:\n",
    "    line_len = len(line)\n",
    "    total_chars = total_chars + line_len\n",
    "    print(f\"The total number of chars is now {total_chars}\")\n",
    "\n",
    "print(f\"The final number of chars is {total_chars}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This use of a for-loop makes the \"loop\" part clearer.\n",
    "\n",
    "```{mermaid}\n",
    "stateDiagram\n",
    "\tstate \"for line in lines:\" as for\n",
    "\tstate if_line <<choice>>\n",
    "\tstate fork <<fork>>\n",
    "\tstate fork1 <<fork>>\n",
    "\t[*] --> fork1\n",
    "\tfork1 --> total_chars: set to 0\n",
    "\tfork1 --> for\n",
    "\tfor --> if_line\n",
    "\tif_line --> fork: If another line\n",
    "\tif_line --> [*]: If no more lines\n",
    "\tfork --> total_chars: +len(line)\n",
    "\tfork --> for\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To actually lowercase the poem now, we need to\n",
    "\n",
    "1. Create an empty list *outside* of the loop.\n",
    "2. Add the lowercased line to this list *inside* of the loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this is just to say',\n",
       " 'by william carlos williams',\n",
       " '',\n",
       " 'i have eaten',\n",
       " 'the plums',\n",
       " 'that were in',\n",
       " 'the icebox',\n",
       " '',\n",
       " 'and which',\n",
       " 'you were probably',\n",
       " 'saving',\n",
       " 'for breakfast',\n",
       " '',\n",
       " 'forgive me',\n",
       " 'they were delicious',\n",
       " 'so sweet',\n",
       " 'and so cold']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lower_lines = []\n",
    "\n",
    "for line in plums_lines:\n",
    "    lower_lines.append(\n",
    "        line.lower()\n",
    "    )\n",
    "\n",
    "lower_lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering the poem\n",
    "\n",
    "There are a few lines of the poem that are blank and have a length of 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "print(lower_lines[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(lower_lines[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a new list without these lines by\n",
    "\n",
    "1. Creating an empty list *outside* of the loop.\n",
    "2. Adding lines to that list *inside* the loop, only if they have a length greater than 0.\n",
    "\n",
    "This introduces an `if` control block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dumped an empty line\n",
      "dumped an empty line\n",
      "dumped an empty line\n"
     ]
    }
   ],
   "source": [
    "full_lines = []\n",
    "\n",
    "for line in lower_lines:\n",
    "    if len(line) > 0:\n",
    "        full_lines.append(line)\n",
    "    else:\n",
    "        print(\"dumped an empty line\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this is just to say',\n",
       " 'by william carlos williams',\n",
       " 'i have eaten',\n",
       " 'the plums',\n",
       " 'that were in',\n",
       " 'the icebox',\n",
       " 'and which',\n",
       " 'you were probably',\n",
       " 'saving',\n",
       " 'for breakfast',\n",
       " 'forgive me',\n",
       " 'they were delicious',\n",
       " 'so sweet',\n",
       " 'and so cold']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `if` block:\n",
    "\n",
    "1. Checks to see if the logical statement is `True` or `False.`\n",
    "2. If `True`, it executes the code inside.\n",
    "\n",
    "In this case, I've also included an `else` block. This executes its code if the `if` block above was `False`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nesting loops\n",
    "\n",
    "We can nest for loops and if statements to an arbitrary level, but good programming practice tries to keep it to a minumum. For example, to get every word in the poem, based on where the spaces are, we can\n",
    "\n",
    "1. Create an empty `words` list *outside* of a loop.\n",
    "2. Split each line using spaces.\n",
    "3. Start a loop over *that* list.\n",
    "4. Update the `words` list *inside* this loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['this', 'is', 'just', 'to', 'say', 'by', 'william', 'carlos', 'williams', 'i', 'have', 'eaten', 'the', 'plums', 'that', 'were', 'in', 'the', 'icebox']\n"
     ]
    }
   ],
   "source": [
    "all_words = []\n",
    "\n",
    "for line in full_lines:\n",
    "    words = line.split(\" \")\n",
    "    for word in words:\n",
    "        all_words.append(word)\n",
    "\n",
    "print(all_words[0:19])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A slightly nicer way\n",
    "\n",
    "A slightly nicer way to do some of the operations we worked on here is with \"list comprehensions\". These are special instructions to build a list that wind up saving us a bit of typing.\n",
    "\n",
    "Here's the code we used to make all of the lines lowercase again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "lower_lines = []\n",
    "\n",
    "for line in plums_lines:\n",
    "    lower_lines.append(line.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a list comprehension, we could rewrite this for-loop like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this is just to say',\n",
       " 'by william carlos williams',\n",
       " '',\n",
       " 'i have eaten',\n",
       " 'the plums',\n",
       " 'that were in',\n",
       " 'the icebox',\n",
       " '',\n",
       " 'and which',\n",
       " 'you were probably',\n",
       " 'saving',\n",
       " 'for breakfast',\n",
       " '',\n",
       " 'forgive me',\n",
       " 'they were delicious',\n",
       " 'so sweet',\n",
       " 'and so cold']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lower_lines2 = [\n",
    "    line.lower() \n",
    "    for line in plums_lines\n",
    "]\n",
    "lower_lines2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even include the empty line filtering into the list comprehension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this is just to say',\n",
       " 'by william carlos williams',\n",
       " 'i have eaten',\n",
       " 'the plums',\n",
       " 'that were in',\n",
       " 'the icebox',\n",
       " 'and which',\n",
       " 'you were probably',\n",
       " 'saving',\n",
       " 'for breakfast',\n",
       " 'forgive me',\n",
       " 'they were delicious',\n",
       " 'so sweet',\n",
       " 'and so cold']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_lines2 = [\n",
    "    line.lower() \n",
    "    for line in plums_lines \n",
    "    if len(line)>0\n",
    "]\n",
    "\n",
    "full_lines2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nested for statements also work inside of list comprehensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this',\n",
       " 'is',\n",
       " 'just',\n",
       " 'to',\n",
       " 'say',\n",
       " 'by',\n",
       " 'william',\n",
       " 'carlos',\n",
       " 'williams',\n",
       " 'i',\n",
       " 'have',\n",
       " 'eaten',\n",
       " 'the',\n",
       " 'plums',\n",
       " 'that',\n",
       " 'were',\n",
       " 'in',\n",
       " 'the',\n",
       " 'icebox']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_words2 = [\n",
    "    word\n",
    "    for line in full_lines2\n",
    "    for word in line.split(\" \")\n",
    "]\n",
    "\n",
    "all_words[0:19]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}