classes/Class 9.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "53fce03f",
   "metadata": {},
   "source": [
    "## Module 9 Class activities\n",
    "This notebook is a starting point for the exercises and activities that we'll do in class. We'll estimate topic models and do some sentiment analysis.\n",
    "\n",
    "We'll continue with our example from last class: the [City of Palm Springs General Plan update](https://www.psgeneralplan.com).\n",
    "\n",
    "Before you attempt any of these activities, make sure to watch the video lectures for this module."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c092609b",
   "metadata": {},
   "source": [
    "### Reading and cleaning PDFs\n",
    "\n",
    "Let's read in [this PDF of public comments](https://www.psgeneralplan.com/_files/ugd/89af76_0b8c3cd9a25140f4a9791570af8d6ba0.pdf). It's in the `data/` folder in your GitHub repository.\n",
    "\n",
    "This is the code from last class. We read in the text of the PDF, and exclude the first 9 pages which are survey responses, not comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4133f155",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pdfminer.high_level import extract_text\n",
    "\n",
    "fn = 'data/PS_VP_Survey_Results_FINAL.pdf'\n",
    "txt = extract_text(fn)\n",
    "txt = txt[txt.find(\"It doesn\"):]\n",
    "\n",
    "txt[:200] # see what it looks like"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f995697",
   "metadata": {},
   "source": [
    "Before we clean up the text further, let's split this into a list of comments. Note that each comment seems to be separated by `AM\\n\\n` or `PM\\n\\n`. So if we split on `M\\n\\n`, we should get a list of comments.\n",
    "\n",
    "Once you have a list, now we can clean it up using regex. Here's my suggestion:\n",
    "* write a function that takes a string, and returns a clean string (remove excess whitespace and characters that are not letters or a space)\n",
    "* create a new list by applying this function to each element of your list of comments\n",
    "\n",
    "The latter can be done with something like:\n",
    "\n",
    "`newlist = [clean_string(comment) for comment in oldlist]`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89523f4a",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<strong>Exercise:</strong> Create a list of cleaned comments.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2991b6f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "def clean_string(comment):\n",
    "    # your code here\n",
    "    return cleaned_comment\n",
    "\n",
    "# create a list of comments\n",
    "comments = 999 # # your code here \n",
    "\n",
    "# then a list of cleaned comments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10ea52f5",
   "metadata": {},
   "source": [
    "One final cleaning step: you probably notice that all your comments end in `P` or `A`. \n",
    "\n",
    "Remove these terminal letters from each comment (or just delete the last two characters of each comment). And remove comments that are just ` P` or ` A` (perhaps you can ignore all comments that are less than, say, 5 characters long).\n",
    "\n",
    "*Hint*: Try another list comprehension (or two)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34b920d0",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<strong>Exercise:</strong> Clean up your comments further and drop the short ones.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6a7d2a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b78b5b4f",
   "metadata": {},
   "source": [
    "### Sentiment analysis\n",
    "Now we have our cleaned up list of comments. Let's do some sentiment analysis. If we create a dataframe with our comments (as one column) and the polarity score (as a second column), the analysis later on becomes easier."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24327cee",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<strong>Exercise:</strong> Create a list of polarity scores, one for each comment. Then create a dataframe, with one column for the comment and one for the polarity score.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "621f6ff5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from textblob import TextBlob\n",
    "import pandas as pd\n",
    "\n",
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b5e0b78",
   "metadata": {},
   "source": [
    "Take a look at the comments with some of the highest and lowest polarity scores. Do the scores make sense? What words is it picking up on?\n",
    "\n",
    "Note that a simple sentiment analyzer like TextBlob won't capture nuances, but in aggregate the results can be useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b37c0ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb81f53a",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<strong>Exercise:</strong> Do some plots of the sentiment and other analyses that try and capture sentiment on particular issues.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be5b83c3",
   "metadata": {},
   "source": [
    "For example, after you plot the overall sentiment scores (a histogram?), you might want to plot the scores where people mention specific issues. For example, you could add a column that is `True` if the comment mentions housing, and then plot the scores only for those rows. Experiment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a62a780b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "582ec3ad",
   "metadata": {},
   "source": [
    "## Topic modeling\n",
    "Now let's see if we can identify different topics in the list of comments.\n",
    "\n",
    "First, we'll need to do a bit more cleanup. For each comment, turn it into a list of words, and exclude stopwords. (After you see the results of your topic modeling, you might want to add more stopwords.) You should end up with a list of lists."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c947fed",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<strong>Exercise:</strong> Turn your list of comments into a list of lists of words, excluding stopwords.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2efc3b40",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize, sent_tokenize\n",
    "\n",
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47911d5a",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<strong>Exercise:</strong> Estimate and visualize a topic model from your wordlists. Experiment with the number of topics and the other hyperparameters.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f42902d7",
   "metadata": {},
   "source": [
    "Remember: \n",
    "* `alpha` controls the expected distribution of topics across documents. A higher value of `alpha` means that each document is expected to contain more of a mix of topics, rather than focusing on a few topics.\n",
    "* `eta` (sometimes called beta) controls the expected distribution of words across topics. A higher value of `eta` means that topics are more similar in terms of their mixture of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac5215f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "import pyLDAvis\n",
    "import pyLDAvis.gensim_models \n",
    "\n",
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5ed2299",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<strong>Exercise:</strong> Explain your topic model - write a sentence or two, and explain to your neighbor. What does each topic signify, and how did you choose alpha and eta?</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05413b5f",
   "metadata": {},
   "source": [
    "Your comments here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7850b937",
   "metadata": {},
   "source": [
    "## Extensions to topic modeling and sentiment analysis\n",
    "One potential use of these models is to look at differences across space or across time. \n",
    "\n",
    "Or we could look at how word frequency changes across space or cities.\n",
    "\n",
    "If you have time, try this with the San Francisco Board of Supervisors meetings. [This webpage](http://sanfrancisco.granicus.com/ViewPublisher.php?view_id=10) gives the archived transcripts (\"caption notes\").\n",
    "\n",
    "You could scrape all of the URLs (that's a good exercise!), but for now, just manually create a list with a 10-20 or so of the caption notes.\n",
    "\n",
    "Write a function that for a given URL:\n",
    "* Gets the text (use `requests`)\n",
    "* Cleans the text\n",
    "* Tokenizes (splits into words)\n",
    "\n",
    "Then, think about how you might estimate a topic model.\n",
    "\n",
    "Would sentiment analysis be useful here?\n",
    "\n",
    "This is an open-ended prompt, so spend some time thinking through the steps conceptually, even if you don't get far in implementing it. For example, how will you organize the text of each documents and the counts? In a list? A dataframe? Will you loop through each URL?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ae74ea8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2818a26",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<h3>You should now be able to:</h3>\n",
    "<ul>\n",
    "  <li>Do further cleaning of text documents</li>\n",
    "  <li>Estimate and interpret sentiments of texts</li>\n",
    "  <li>Estimate and interpret topic models</li>\n",
    "</ul>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}