week5_notebook.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lmrhody/femethodsS23/blob/main/week5_notebook.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Week 5 Notebook\n",
        "## Exploratory Analysis with Text Data\n",
        "Student Name:\n",
        "\n",
        "Date:\n",
        "\n",
        "Assignment Due:\n",
        "\n",
        "Instructor: Lisa Rhody\n",
        "\n",
        "Methods of Text Analysis, Spring 2023"
      ],
      "metadata": {
        "id": "0-4URUeInnlb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Note: \n",
        "This week we are switching from the NLTK Book to using a companion Google Colab notebook for the O'Reilly series of books titled _Blueprints for Text Analytics Using Python_. This week, the reading is focused on the kinds of questions we can ask with text data, and so it's fitting for you to start learning what the preliminary process is for exploring a text corpus in order to begin thinking about waht questions could be asked of it. \n",
        "\n",
        "However, this notebook is not exactly the same as the one from the Blueprints book, because interspersed in between exercises are questions to answer. So, my recommendation is that you go through the PDF of the chapter in the Commons library while you work through this notebook. At the same time, you'll want to respond to the additional questions with the help of the readings from this week. "
      ],
      "metadata": {
        "id": "g4Gkp0i0oBDp"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e-lFT4xHQUU_"
      },
      "source": [
        "\n",
        "[**Blueprints for Text Analysis Using Python**](https://github.com/blueprints-for-text-analytics-python/blueprints-text)  \n",
        "Jens Albrecht, Sidharth Ramachandran, Christian Winkler\n",
        "\n",
        "**If you like the book or the code examples here, please leave a friendly comment on [Amazon.com](https://www.amazon.com/Blueprints-Text-Analytics-Using-Python/dp/149207408X)!**\n",
        "<img src=\"https://github.com/blueprints-for-text-analytics-python/blueprints-text/blob/master/rating.png?raw=1\" width=\"100\"/>\n",
        "\n",
        "\n",
        "# Chapter 1:<div class='tocSkip'/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z-zRc4OAQUVB"
      },
      "source": [
        "# Gaining Early Insights from Textual Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gV0sUx4xQUVB"
      },
      "source": [
        "## Remark<div class='tocSkip'/>\n",
        "\n",
        "The code in this notebook differs slightly from the printed book. For example we frequently use pretty print (`pp.pprint`) instead of `print` and `tqdm`'s `progress_apply` instead of Pandas' `apply`. \n",
        "\n",
        "Moreover, several layout and formatting commands, like `figsize` to control figure size or subplot commands are removed in the book.\n",
        "\n",
        "You may also find some lines marked with three hashes ###. Those are not in the book as well as they don't contribute to the concept.\n",
        "\n",
        "All of this is done to simplify the code in the book and put the focus on the important parts instead of formatting."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_C6aspQoQUVB"
      },
      "source": [
        "## Setup<div class='tocSkip'/>\n",
        "\n",
        "Set directory locations. If working on Google Colab: copy files and install required libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:41:58.195850Z",
          "start_time": "2021-05-26T10:41:58.177026Z"
        },
        "id": "re0jVaEpQUVC"
      },
      "outputs": [],
      "source": [
        "import sys, os\n",
        "ON_COLAB = 'google.colab' in sys.modules\n",
        "\n",
        "if ON_COLAB:\n",
        "    GIT_ROOT = 'https://github.com/blueprints-for-text-analytics-python/blueprints-text/raw/master'\n",
        "    os.system(f'wget {GIT_ROOT}/ch01/setup.py')\n",
        "\n",
        "%run -i setup.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PMMujPAJQUVC"
      },
      "source": [
        "## Load Python Settings<div class=\"tocSkip\"/>\n",
        "\n",
        "Common imports, defaults for formatting in Matplotlib, Pandas etc."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:00.394826Z",
          "start_time": "2021-05-26T10:41:58.197940Z"
        },
        "id": "AWckmoIVQUVC"
      },
      "outputs": [],
      "source": [
        "%run \"$BASE_DIR/settings.py\"\n",
        "\n",
        "%reload_ext autoreload\n",
        "%autoreload 2\n",
        "%config InlineBackend.figure_format = 'png'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXOFXAWTQUVC"
      },
      "source": [
        "## What you will learn and what we will build\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "326tiqAlQUVD"
      },
      "source": [
        "# Exploratory Data Analysis\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "it-WSgifQUVD"
      },
      "source": [
        "# Introducing the Dataset\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:04.546077Z",
          "start_time": "2021-05-26T10:42:00.397654Z"
        },
        "id": "8Be7c751QUVD"
      },
      "outputs": [],
      "source": [
        "pd.options.display.max_colwidth = 150 ###\n",
        "file = \"un-general-debates-blueprint.csv\"\n",
        "file = f\"{BASE_DIR}/data/un-general-debates/un-general-debates-blueprint.csv.gz\" ### real location\n",
        "df = pd.read_csv(file)\n",
        "df.sample(2, random_state=53)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "99RtcthSQUVD"
      },
      "source": [
        "# Blueprint: Getting an Overview of the Data with Pandas\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0y2l8ANDQUVD"
      },
      "source": [
        "## Calculating Summary Statistics for Columns\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:04.585120Z",
          "start_time": "2021-05-26T10:42:04.548589Z"
        },
        "id": "5wywL68eQUVD"
      },
      "outputs": [],
      "source": [
        "df['length'] = df['text'].str.len()\n",
        "\n",
        "df.describe().T"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:04.644901Z",
          "start_time": "2021-05-26T10:42:04.587343Z"
        },
        "id": "RM22wPl6QUVE"
      },
      "outputs": [],
      "source": [
        "df[['country', 'speaker']].describe(include='O').T"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3UpjsM-EQUVE"
      },
      "source": [
        "## Checking for Missing Data\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:04.673446Z",
          "start_time": "2021-05-26T10:42:04.646924Z"
        },
        "id": "3XrlSw4MQUVE"
      },
      "outputs": [],
      "source": [
        "df.isna().sum()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:04.715677Z",
          "start_time": "2021-05-26T10:42:04.679932Z"
        },
        "id": "yBqFF4VRQUVE"
      },
      "outputs": [],
      "source": [
        "df['speaker'].fillna('unkown', inplace=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:04.752450Z",
          "start_time": "2021-05-26T10:42:04.720245Z"
        },
        "id": "LEZxMKkBQUVE"
      },
      "outputs": [],
      "source": [
        "df[df['speaker'].str.contains('Bush')]['speaker'].value_counts()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PM8ZrfrgQUVE"
      },
      "source": [
        "## Plotting Value Distributions\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:04.948204Z",
          "start_time": "2021-05-26T10:42:04.757532Z"
        },
        "id": "NDPy9eSyQUVE"
      },
      "outputs": [],
      "source": [
        "df['length'].plot(kind='box', vert=False, figsize=(8, 1))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:05.148991Z",
          "start_time": "2021-05-26T10:42:04.950770Z"
        },
        "id": "LJKGcoBFQUVF"
      },
      "outputs": [],
      "source": [
        "df['length'].plot(kind='hist', bins=30, figsize=(8,2))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:05.439106Z",
          "start_time": "2021-05-26T10:42:05.151748Z"
        },
        "id": "lq7S48R5QUVF"
      },
      "outputs": [],
      "source": [
        "# Not in book: seaborn plot with gaussian kernel density estimate\n",
        "import seaborn as sns\n",
        "\n",
        "plt.figure(figsize=(8, 2))\n",
        "sns.distplot(df['length'], bins=30, kde=True);"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Question 1: \n",
        "\n",
        "In this section, you are learning how to get a general insight into the dataset by looking at its length, finding missing data types, and evaluating things that are duplicate values because there are multiple references or multiple forms of textual representation. What is the benefit to computational text analysis of blueprint or (as Salganick calls them, \"readymades\")? "
      ],
      "metadata": {
        "id": "9SCo4UVopOYY"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sekOaqT8QUVF"
      },
      "source": [
        "## Comparing Value Distributions across Categories\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:06.102415Z",
          "start_time": "2021-05-26T10:42:05.441397Z"
        },
        "id": "Ns0ir5FDQUVF"
      },
      "outputs": [],
      "source": [
        "where = df['country'].isin(['USA', 'FRA', 'GBR', 'CHN', 'RUS'])\n",
        "g = sns.catplot(data=df[where], x=\"country\", y=\"length\", kind='box')\n",
        "g.fig.set_size_inches(4, 3) ###\n",
        "g.fig.set_dpi(100) ###\n",
        "g = sns.catplot(data=df[where], x=\"country\", y=\"length\", kind='violin')\n",
        "g.fig.set_size_inches(4, 3) ###\n",
        "g.fig.set_dpi(100) ###"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V9JQuhqZQUVF"
      },
      "source": [
        "## Visualizing Developments over Time\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:06.271029Z",
          "start_time": "2021-05-26T10:42:06.104222Z"
        },
        "id": "4hmOC3xCQUVF"
      },
      "outputs": [],
      "source": [
        "df.groupby('year').size().plot(title=\"Number of Countries\", figsize=(6,2))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:06.558946Z",
          "start_time": "2021-05-26T10:42:06.274077Z"
        },
        "id": "ifq4Md7eQUVF"
      },
      "outputs": [],
      "source": [
        "df.groupby('year').agg({'length': 'mean'}) \\\n",
        "  .plot(title=\"Avg. Speech Length\", ylim=(0,30000), figsize=(6,2))"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Question 2: \n",
        "\n",
        "What kinds of questions can you ask when you can visualize distributions or change? What kind of data do you need in order for these visualizations to work? How does access to or limitations of data such as categorical labels or years change the kind of questions you can ask? "
      ],
      "metadata": {
        "id": "uumNHUudp0c2"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mf1nOqNMQUVG"
      },
      "source": [
        "# Blueprint: Building a Simple Text Preprocessing Pipeline\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "weA_wbYLQUVG"
      },
      "source": [
        "## Tokenization with Regular Expressions\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:06.582112Z",
          "start_time": "2021-05-26T10:42:06.562641Z"
        },
        "id": "C3ZlXdiPQUVG"
      },
      "outputs": [],
      "source": [
        "import regex as re\n",
        "\n",
        "def tokenize(text):\n",
        "    return re.findall(r'[\\w-]*\\p{L}[\\w-]*', text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:06.603608Z",
          "start_time": "2021-05-26T10:42:06.584627Z"
        },
        "id": "7JCcrSu3QUVG"
      },
      "outputs": [],
      "source": [
        "text = \"Let's defeat SARS-CoV-2 together in 2020!\"\n",
        "tokens = tokenize(text)\n",
        "print(\"|\".join(tokens))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NZY_njeOQUVG"
      },
      "source": [
        "## Treating Stop Words\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:06.788245Z",
          "start_time": "2021-05-26T10:42:06.605812Z"
        },
        "id": "uUDMi_P4QUVG"
      },
      "outputs": [],
      "source": [
        "import nltk\n",
        "# not in book: make sure stop words are available\n",
        "nltk.download('stopwords')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:06.813022Z",
          "start_time": "2021-05-26T10:42:06.792334Z"
        },
        "id": "O1lsxW5UQUVG"
      },
      "outputs": [],
      "source": [
        "import nltk\n",
        "\n",
        "stopwords = set(nltk.corpus.stopwords.words('english'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:06.835634Z",
          "start_time": "2021-05-26T10:42:06.815532Z"
        },
        "id": "wcYS62TnQUVG"
      },
      "outputs": [],
      "source": [
        "def remove_stop(tokens):\n",
        "    return [t for t in tokens if t.lower() not in stopwords]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:06.857424Z",
          "start_time": "2021-05-26T10:42:06.839934Z"
        },
        "id": "mhSD2xLWQUVH"
      },
      "outputs": [],
      "source": [
        "include_stopwords = {'dear', 'regards', 'must', 'would', 'also'}\n",
        "exclude_stopwords = {'against'}\n",
        "\n",
        "stopwords |= include_stopwords\n",
        "stopwords -= exclude_stopwords"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Question 3: \n",
        "\n",
        "In _Blueprints_, the authors are particularly careful to say that the way to remove stopwords isn't always a one-size-fits-all solution. Taking into consideration, DÍgnazio and Klein's chapter on \"rational, scientific, objective viewpoints,\" when does it make sense to use standard stop word lists, and when should stop word lists be adjusted? \n"
      ],
      "metadata": {
        "id": "VbJl5dnpqJbP"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A7YpjKloQUVH"
      },
      "source": [
        "## Processing a Pipeline with one Line of Code\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:06.877721Z",
          "start_time": "2021-05-26T10:42:06.859933Z"
        },
        "id": "g7BhhiOLQUVH"
      },
      "outputs": [],
      "source": [
        "pipeline = [str.lower, tokenize, remove_stop]\n",
        "\n",
        "def prepare(text, pipeline):\n",
        "    tokens = text\n",
        "    for transform in pipeline:\n",
        "        tokens = transform(tokens)\n",
        "    return tokens"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:24.578033Z",
          "start_time": "2021-05-26T10:42:06.880896Z"
        },
        "id": "OpCFcxDqQUVH"
      },
      "outputs": [],
      "source": [
        "df['tokens'] = df['text'].progress_apply(prepare, pipeline=pipeline)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:24.743332Z",
          "start_time": "2021-05-26T10:42:24.587982Z"
        },
        "id": "AIKR3dyTQUVH"
      },
      "outputs": [],
      "source": [
        "df['num_tokens'] = df['tokens'].progress_map(len)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bVzYcEm8QUVH"
      },
      "source": [
        "# Blueprints for Word Frequency Analysis\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xPNJYq9LQUVH"
      },
      "source": [
        "## Blueprint: Counting Words with a Counter\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:24.785986Z",
          "start_time": "2021-05-26T10:42:24.752296Z"
        },
        "id": "Ap1qsAHvQUVH"
      },
      "outputs": [],
      "source": [
        "from collections import Counter\n",
        "\n",
        "tokens = tokenize(\"She likes my cats and my cats like my sofa.\")\n",
        "\n",
        "counter = Counter(tokens)\n",
        "print(counter)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:24.818884Z",
          "start_time": "2021-05-26T10:42:24.790881Z"
        },
        "id": "_kFuvcQuQUVH"
      },
      "outputs": [],
      "source": [
        "more_tokens = tokenize(\"She likes dogs and cats.\")\n",
        "counter.update(more_tokens)\n",
        "print(counter)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:26.700698Z",
          "start_time": "2021-05-26T10:42:24.822554Z"
        },
        "id": "7dbF-fBwQUVI"
      },
      "outputs": [],
      "source": [
        "counter = Counter()\n",
        "\n",
        "_ = df['tokens'].map(counter.update)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:26.741941Z",
          "start_time": "2021-05-26T10:42:26.702480Z"
        },
        "id": "Mic_QMiyQUVI"
      },
      "outputs": [],
      "source": [
        "pp.pprint(counter.most_common(5))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:26.765301Z",
          "start_time": "2021-05-26T10:42:26.743907Z"
        },
        "id": "AX-ChknAQUVI"
      },
      "outputs": [],
      "source": [
        "from collections import Counter ###\n",
        "def count_words(df, column='tokens', preprocess=None, min_freq=2):\n",
        "\n",
        "    # process tokens and update counter\n",
        "    def update(doc):\n",
        "        tokens = doc if preprocess is None else preprocess(doc)\n",
        "        counter.update(tokens)\n",
        "\n",
        "    # create counter and run through all data\n",
        "    counter = Counter()\n",
        "    df[column].progress_map(update)\n",
        "\n",
        "    # transform counter into data frame\n",
        "    freq_df = pd.DataFrame.from_dict(counter, orient='index', columns=['freq'])\n",
        "    freq_df = freq_df.query('freq >= @min_freq')\n",
        "    freq_df.index.name = 'token'\n",
        "    \n",
        "    return freq_df.sort_values('freq', ascending=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:28.914276Z",
          "start_time": "2021-05-26T10:42:26.767840Z"
        },
        "id": "euV9LKqCQUVI"
      },
      "outputs": [],
      "source": [
        "freq_df = count_words(df)\n",
        "freq_df.head(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:39.172001Z",
          "start_time": "2021-05-26T10:42:28.917074Z"
        },
        "id": "Y7U9NWlgQUVI"
      },
      "outputs": [],
      "source": [
        "# top words with 10+ characters\n",
        "count_words(df, column='text', \n",
        "            preprocess=lambda text: re.findall(r\"\\w{10,}\", text)).head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OJsBdce5QUVI"
      },
      "source": [
        "## Blueprint: Creating a Frequency Diagram\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:39.648788Z",
          "start_time": "2021-05-26T10:42:39.175200Z"
        },
        "id": "tKasEgdaQUVI"
      },
      "outputs": [],
      "source": [
        "ax = freq_df.head(15).plot(kind='barh', width=0.95, figsize=(8,3))\n",
        "ax.invert_yaxis()\n",
        "ax.set(xlabel='Frequency', ylabel='Token', title='Top Words')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vD137keCQUVJ"
      },
      "source": [
        "## Blueprint: Creating Word Clouds\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:40.143556Z",
          "start_time": "2021-05-26T10:42:39.651100Z"
        },
        "id": "LOGXDFuAQUVJ"
      },
      "outputs": [],
      "source": [
        "from wordcloud import WordCloud\n",
        "from matplotlib import pyplot as plt\n",
        "\n",
        "text = df.query(\"year==2015 and country=='USA'\")['text'].values[0]\n",
        "\n",
        "plt.figure(figsize=(4, 2)) ###\n",
        "wc = WordCloud(max_words=100, stopwords=stopwords)\n",
        "wc.generate(text)\n",
        "plt.imshow(wc, interpolation='bilinear')\n",
        "plt.axis(\"off\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:40.173113Z",
          "start_time": "2021-05-26T10:42:40.146097Z"
        },
        "id": "5cAljVJlQUVJ"
      },
      "outputs": [],
      "source": [
        "from wordcloud import WordCloud ###\n",
        "from collections import Counter ###\n",
        "\n",
        "def wordcloud(word_freq, title=None, max_words=200, stopwords=None):\n",
        "\n",
        "    wc = WordCloud(width=800, height=400, \n",
        "                   background_color= \"black\", colormap=\"Paired\", \n",
        "                   max_font_size=150, max_words=max_words)\n",
        "    \n",
        "    # convert data frame into dict\n",
        "    if type(word_freq) == pd.Series:\n",
        "        counter = Counter(word_freq.fillna(0).to_dict())\n",
        "    else:\n",
        "        counter = word_freq\n",
        "\n",
        "    # filter stop words in frequency counter\n",
        "    if stopwords is not None:\n",
        "        counter = {token:freq for (token, freq) in counter.items() \n",
        "                              if token not in stopwords}\n",
        "    wc.generate_from_frequencies(counter)\n",
        " \n",
        "    plt.title(title) \n",
        "\n",
        "    plt.imshow(wc, interpolation='bilinear')\n",
        "    plt.axis(\"off\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:41.423825Z",
          "start_time": "2021-05-26T10:42:40.175979Z"
        },
        "id": "OON2oc3OQUVJ"
      },
      "outputs": [],
      "source": [
        "freq_2015_df = count_words(df[df['year']==2015])\n",
        "plt.figure(figsize=(12,4))\n",
        "plt.subplot(1,2,1)###\n",
        "wordcloud(freq_2015_df['freq'], max_words=100)\n",
        "plt.subplot(1,2,2)###\n",
        "wordcloud(freq_2015_df['freq'], max_words=100, stopwords=freq_df.head(50).index)\n",
        "#plt.tight_layout()###"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Question 4: \n",
        "\n",
        "Counting seems to be a significant issue for feminist scholars. On the one hand, being counted is a political act (as we remember from the readings in Week 4 as well as in Mandell's chapter that we read for this week). How do the kinds of counting that we can do with text analysis create challenges for the feminist scholar according to Laura Mandell? How do you see what she describes at play in these blueprint text processes? "
      ],
      "metadata": {
        "id": "3LsmTqqbrRNK"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qg0_nCaNQUVJ"
      },
      "source": [
        "## Blueprint: Ranking with TF-IDF\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:41.457725Z",
          "start_time": "2021-05-26T10:42:41.427473Z"
        },
        "id": "_63HkvplQUVJ"
      },
      "outputs": [],
      "source": [
        "def compute_idf(df, column='tokens', preprocess=None, min_df=2):\n",
        "\n",
        "    def update(doc):\n",
        "        tokens = doc if preprocess is None else preprocess(doc)\n",
        "        counter.update(set(tokens))\n",
        "\n",
        "    # count tokens\n",
        "    counter = Counter()\n",
        "    df[column].progress_map(update)\n",
        "\n",
        "    # create data frame and compute idf\n",
        "    idf_df = pd.DataFrame.from_dict(counter, orient='index', columns=['df'])\n",
        "    idf_df = idf_df.query('df >= @min_df')\n",
        "    idf_df['idf'] = np.log(len(df)/idf_df['df'])+0.1\n",
        "    idf_df.index.name = 'token'\n",
        "    return idf_df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:44.140253Z",
          "start_time": "2021-05-26T10:42:41.460618Z"
        },
        "id": "h__9OqVOQUVJ"
      },
      "outputs": [],
      "source": [
        "idf_df = compute_idf(df)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:44.178293Z",
          "start_time": "2021-05-26T10:42:44.145248Z"
        },
        "id": "k_uHux2CQUVJ"
      },
      "outputs": [],
      "source": [
        "# Not in book: sample of IDF values\n",
        "# high IDF means rare (interesting) term\n",
        "idf_df.sample(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:44.414313Z",
          "start_time": "2021-05-26T10:42:44.180732Z"
        },
        "id": "3hFp4HVZQUVK"
      },
      "outputs": [],
      "source": [
        "freq_df['tfidf'] = freq_df['freq'] * idf_df['idf']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:44.464686Z",
          "start_time": "2021-05-26T10:42:44.416567Z"
        },
        "id": "N0o6eGTcQUVK"
      },
      "outputs": [],
      "source": [
        "# not in book: for more data: joining is faster\n",
        "freq_df = freq_df.join(idf_df)\n",
        "freq_df['tfidf'] = freq_df['freq'] * freq_df['idf']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:48.388348Z",
          "start_time": "2021-05-26T10:42:44.468397Z"
        },
        "id": "tRdZ7V7UQUVK"
      },
      "outputs": [],
      "source": [
        "freq_1970 = count_words(df[df['year'] == 1970])\n",
        "freq_2015 = count_words(df[df['year'] == 2015])\n",
        "\n",
        "freq_1970['tfidf'] = freq_1970['freq'] * idf_df['idf']\n",
        "freq_2015['tfidf'] = freq_2015['freq'] * idf_df['idf']\n",
        "\n",
        "plt.figure(figsize=(12,6)) ###\n",
        "#wordcloud(freq_df['freq'], title='All years', subplot=(1,3,1))\n",
        "plt.subplot(2,2,1)###\n",
        "wordcloud(freq_1970['freq'], title='1970 - TF', \n",
        "          stopwords=['twenty-fifth', 'twenty-five'])\n",
        "plt.subplot(2,2,2)###\n",
        "wordcloud(freq_2015['freq'], title='2015 - TF', \n",
        "          stopwords=['seventieth'])\n",
        "plt.subplot(2,2,3)###\n",
        "wordcloud(freq_1970['tfidf'], title='1970 - TF-IDF', \n",
        "          stopwords=['twenty-fifth', 'twenty-five', 'twenty', 'fifth'])\n",
        "plt.subplot(2,2,4)###\n",
        "wordcloud(freq_2015['tfidf'], title='2015 - TF-IDF', \n",
        "          stopwords=['seventieth'])"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Question 5: \n",
        "\n",
        "What is the attraction to scholars like Ted Underwood in the ability to make use of classification analyses like TF-IDF? What kinds of questions could be asked of a text corpora using an unsupervised classification study like the one above? "
      ],
      "metadata": {
        "id": "rx4WGSfMrz_r"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E4D72T9DQUVK"
      },
      "source": [
        "# Blueprint: Finding a Keyword in Context (KWIC)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1UvdOybDQUVK"
      },
      "source": [
        "**Note:** textacy's API had major changes from version 0.10.1 (as used in the book) to 0.11.  \n",
        "Here, `textacy.text_utils.KWIC` became `textacy.extract.kwic.keyword_in_context` (see [textacy documentation](https://textacy.readthedocs.io/en/latest/api_reference/extract.html#module-textacy.extract.kwic)).\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-06-04T05:09:08.234120Z",
          "start_time": "2021-06-04T05:09:05.181494Z"
        },
        "id": "e6s4RnXFQUVK"
      },
      "outputs": [],
      "source": [
        "import textacy\n",
        "\n",
        "if textacy.__version__ < '0.11': # as in printed book\n",
        "    from textacy.text_utils import KWIC\n",
        "    \n",
        "else: # for textacy 0.11.x\n",
        "    from textacy.extract.kwic import keyword_in_context\n",
        "\n",
        "    def KWIC(*args, **kwargs):\n",
        "        # call keyword_in_context with all params except 'print_only'\n",
        "        return keyword_in_context(*args, \n",
        "                           **{kw: arg for kw, arg in kwargs.items() \n",
        "                            if kw != 'print_only'})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:58:00.290940Z",
          "start_time": "2021-05-26T10:58:00.270371Z"
        },
        "id": "VW3refvYQUVK"
      },
      "outputs": [],
      "source": [
        "def kwic(doc_series, keyword, window=35, print_samples=5):\n",
        "\n",
        "    def add_kwic(text):\n",
        "        kwic_list.extend(KWIC(text, keyword, ignore_case=True, \n",
        "                              window_width=window, print_only=False))\n",
        "\n",
        "    kwic_list = []\n",
        "    doc_series.progress_map(add_kwic)\n",
        "\n",
        "    if print_samples is None or print_samples==0:\n",
        "        return kwic_list\n",
        "    else:\n",
        "        k = min(print_samples, len(kwic_list))\n",
        "        print(f\"{k} random samples out of {len(kwic_list)} \" + \\\n",
        "              f\"contexts for '{keyword}':\")\n",
        "        for sample in random.sample(list(kwic_list), k):\n",
        "            print(re.sub(r'[\\n\\t]', ' ', sample[0])+'  '+ \\\n",
        "                  sample[1]+'  '+\\\n",
        "                  re.sub(r'[\\n\\t]', ' ', sample[2]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:58:04.386669Z",
          "start_time": "2021-05-26T10:58:04.296378Z"
        },
        "scrolled": true,
        "id": "spMW7aF8QUVK"
      },
      "outputs": [],
      "source": [
        "random.seed(22) ###\n",
        "kwic(df[df['year'] == 2015]['text'], 'sdgs', print_samples=5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "huE0U6aNQUVK"
      },
      "source": [
        "# Blueprint: Analyzing N-Grams\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.206581Z",
          "start_time": "2021-05-26T10:41:58.405Z"
        },
        "id": "6vDi0V53QUVL"
      },
      "outputs": [],
      "source": [
        "def ngrams(tokens, n=2, sep=' '):\n",
        "    return [sep.join(ngram) for ngram in zip(*[tokens[i:] for i in range(n)])]\n",
        "\n",
        "text = \"the visible manifestation of the global climate change\"\n",
        "tokens = tokenize(text)\n",
        "print(\"|\".join(ngrams(tokens, 2)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.208551Z",
          "start_time": "2021-05-26T10:41:58.407Z"
        },
        "id": "Q5Ck4wsbQUVL"
      },
      "outputs": [],
      "source": [
        "def ngrams(tokens, n=2, sep=' ', stopwords=set()):\n",
        "    return [sep.join(ngram) for ngram in zip(*[tokens[i:] for i in range(n)])\n",
        "            if len([t for t in ngram if t in stopwords])==0]\n",
        "\n",
        "print(\"Bigrams:\", \"|\".join(ngrams(tokens, 2, stopwords=stopwords)))\n",
        "print(\"Trigrams:\", \"|\".join(ngrams(tokens, 3, stopwords=stopwords)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.210075Z",
          "start_time": "2021-05-26T10:41:58.410Z"
        },
        "id": "ox0vjxmEQUVL"
      },
      "outputs": [],
      "source": [
        "df['bigrams'] = df['text'].progress_apply(prepare, pipeline=[str.lower, tokenize]) \\\n",
        "                          .progress_apply(ngrams, n=2, stopwords=stopwords)\n",
        "\n",
        "count_words(df, 'bigrams').head(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.211195Z",
          "start_time": "2021-05-26T10:41:58.413Z"
        },
        "id": "p3kYXEilQUVL"
      },
      "outputs": [],
      "source": [
        "idf_df = compute_idf(df) ### re-initialize to be safe\n",
        "# concatenate existing IDF data frame with bigram IDFs\n",
        "idf_df = pd.concat([idf_df, compute_idf(df, 'bigrams', min_df=10)])\n",
        "\n",
        "freq_df = count_words(df[df['year'] == 2015], 'bigrams')\n",
        "freq_df['tfidf'] = freq_df['freq'] * idf_df['idf']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.214118Z",
          "start_time": "2021-05-26T10:41:58.416Z"
        },
        "id": "txtB2O6qQUVL"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(12,6)) ###\n",
        "plt.subplot(1,2,1) ###\n",
        "wordcloud(freq_df['tfidf'], title='all bigrams', max_words=50)\n",
        "\n",
        "plt.subplot(1,2,2) ###\n",
        "# plt.tight_layout() ###\n",
        "where = freq_df.index.str.contains('climate')\n",
        "wordcloud(freq_df[where]['freq'], title='\"climate\" bigrams', max_words=50)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rvd-oVDJQUVL"
      },
      "source": [
        "# Blueprint: Comparing Frequencies across Time-Intervals and Categories\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qxaxum97QUVL"
      },
      "source": [
        "## Creating Frequency Timelines\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.216415Z",
          "start_time": "2021-05-26T10:41:58.420Z"
        },
        "id": "yOqF7Z9yQUVL"
      },
      "outputs": [],
      "source": [
        "def count_keywords(tokens, keywords): \n",
        "    tokens = [t for t in tokens if t in keywords]\n",
        "    counter = Counter(tokens)\n",
        "    return [counter.get(k, 0) for k in keywords]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.218645Z",
          "start_time": "2021-05-26T10:41:58.423Z"
        },
        "id": "OedNixaHQUVL"
      },
      "outputs": [],
      "source": [
        "keywords = ['nuclear', 'terrorism', 'climate', 'freedom']\n",
        "tokens = ['nuclear', 'climate', 'climate', 'freedom', 'climate', 'freedom']\n",
        "\n",
        "print(count_keywords(tokens, keywords))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.221158Z",
          "start_time": "2021-05-26T10:41:58.426Z"
        },
        "id": "w_kUy_CwQUVM"
      },
      "outputs": [],
      "source": [
        "def count_keywords_by(df, by, keywords, column='tokens'):\n",
        "    \n",
        "    df = df.reset_index(drop=True) # if the supplied dataframe has gaps in the index\n",
        "    freq_matrix = df[column].progress_apply(count_keywords, keywords=keywords)\n",
        "    freq_df = pd.DataFrame.from_records(freq_matrix, columns=keywords)\n",
        "    freq_df[by] = df[by] # copy the grouping column(s)\n",
        "    \n",
        "    return freq_df.groupby(by=by).sum().sort_values(by)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.223999Z",
          "start_time": "2021-05-26T10:41:58.428Z"
        },
        "id": "GEjZaHa_QUVM"
      },
      "outputs": [],
      "source": [
        "freq_df = count_keywords_by(df, by='year', keywords=keywords)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.225541Z",
          "start_time": "2021-05-26T10:41:58.430Z"
        },
        "id": "vkOmQFzmQUVM"
      },
      "outputs": [],
      "source": [
        "pd.options.display.max_rows = 4"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.226810Z",
          "start_time": "2021-05-26T10:41:58.432Z"
        },
        "id": "EsgjgTbkQUVM"
      },
      "outputs": [],
      "source": [
        "pd.options.display.max_rows = 60"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.228254Z",
          "start_time": "2021-05-26T10:41:58.434Z"
        },
        "id": "WzG7Jta8QUVM"
      },
      "outputs": [],
      "source": [
        "freq_df.plot(kind='line', figsize=(8, 3))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.230717Z",
          "start_time": "2021-05-26T10:41:58.437Z"
        },
        "id": "TPP8avOOQUVM"
      },
      "outputs": [],
      "source": [
        "random.seed(23) ###\n",
        "# analyzing mentions of 'climate' before 1980\n",
        "kwic(df.query('year < 1980')['text'], 'climate', window=35, print_samples=5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z6bLioTZQUVM"
      },
      "source": [
        "## Creating Frequency Heat Maps\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-26T10:42:49.233155Z",
          "start_time": "2021-05-26T10:41:58.441Z"
        },
        "scrolled": true,
        "id": "BuRVBuLiQUVM"
      },
      "outputs": [],
      "source": [
        "keywords = ['terrorism', 'terrorist', 'nuclear', 'war', 'oil',\n",
        "            'syria', 'syrian', 'refugees', 'migration', 'peacekeeping', \n",
        "            'humanitarian', 'climate', 'change', 'sustainable', 'sdgs']  \n",
        "\n",
        "freq_df = count_keywords_by(df, by='year', keywords=keywords)\n",
        "\n",
        "# compute relative frequencies based on total number of tokens per year\n",
        "freq_df = freq_df.div(df.groupby('year')['num_tokens'].sum(), axis=0)\n",
        "# apply square root as sublinear filter for better contrast\n",
        "freq_df = freq_df.apply(np.sqrt)\n",
        "\n",
        "plt.figure(figsize=(10, 3)) ###\n",
        "sns.set(font_scale=1) ###\n",
        "sns.heatmap(data=freq_df.T, \n",
        "            xticklabels=True, yticklabels=True, cbar=False, cmap=\"Reds\")\n",
        "sns.set(font_scale=1) ###"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8vpRBI6PQUVM"
      },
      "source": [
        "# Closing Remarks\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Question 6: \n",
        "\n",
        "What is the value of exploratory analysis in computational text analysis? How can it be helpful? In what ways is it insufficient? "
      ],
      "metadata": {
        "id": "GsctA7pJsW6R"
      }
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.8"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": false,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {
        "height": "calc(100% - 180px)",
        "left": "10px",
        "top": "150px",
        "width": "281.542px"
      },
      "toc_section_display": true,
      "toc_window_display": true
    },
    "varInspector": {
      "cols": {
        "lenName": 16,
        "lenType": 16,
        "lenVar": 40
      },
      "kernels_config": {
        "python": {
          "delete_cmd_postfix": "",
          "delete_cmd_prefix": "del ",
          "library": "var_list.py",
          "varRefreshCmd": "print(var_dic_list())"
        },
        "r": {
          "delete_cmd_postfix": ") ",
          "delete_cmd_prefix": "rm(",
          "library": "var_list.r",
          "varRefreshCmd": "cat(var_dic_list()) "
        }
      },
      "types_to_exclude": [
        "module",
        "function",
        "builtin_function_or_method",
        "instance",
        "_Feature"
      ],
      "window_display": false
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}