code/Python-Text-Overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "47acd9a9",
   "metadata": {},
   "source": [
    "We did a quick tour of Python and saw some of its features earlier in today's class. Let us now see what off-the-shelf support exists in terms of NLP tools in Python in the rest of this class, and some of next class.\n",
    "\n",
    "Before we get into NLP methods in depth, let us first look at how to do basic corpus analysis in Python. That is the goal of this notebook. \n",
    "\n",
    "Based on: \n",
    "- Chapters 1--3 in NLTK. (Chapters 1--7 may be useful for non NLP focused folks, generally, IMO)\n",
    "\n",
    "Other useful resources:\n",
    "- Python worksheets in Katrin Erk's NLP courses: https://www.katrinerk.com/teaching/resources "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04709241",
   "metadata": {},
   "source": [
    "# Why?\n",
    "The first step before we build any NLP models is to \"understand\" the data a little bit.\n",
    "\n",
    "What is \"understanding\" the data?\n",
    "    - Looking at frequently used words/ngrams in the data\n",
    "    - Looking at collocations (what words go together)\n",
    "    - How can we visualize text corpora?\n",
    "    - lexical diversity, linguistic coverage etc in the dataset\n",
    "    \n",
    "Why do we need this?\n",
    "    - To get a general idea of what the texts are about\n",
    "    - Identify potentially useful features to use in an NLP model that uses this data\n",
    "    - Identify potential noise in the data\n",
    "    - Understand limitations of the data (e.g., potential bias) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5270b079",
   "metadata": {},
   "source": [
    "## Installing necessary libraries \n",
    "Before proceeding further, we have to install the necessary Python libraries (libraries can be thought of as a collection of pre-implemented utilities we can just use right away)\n",
    "\n",
    "- nltk\n",
    "- matplotlib (for plots)\n",
    "\n",
    "installing libraries in python happens through a command called pip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "513395a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install nltk\n",
    "!pip install matplotlib\n",
    "\n",
    "#the ! symbol lets us run programs that are typically run from a command line tool in a notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1fd19c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "#If you are able to import these libraries without errors, that means you installed them correctly\n",
    "import nltk\n",
    "import matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac288873",
   "metadata": {},
   "source": [
    "## Download the required language resources for nltk and spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8297ac2",
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.download(\"book\")\n",
    "#Download any data or other non-code resources we need for using the code in the book."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd2005df",
   "metadata": {},
   "source": [
    "# Elementary Text Analysis \n",
    "\n",
    "I will use NLTK library's existing functions to illustrate some text analysis functions in Python. The following examples are based on Chapter 1. \n",
    "\n",
    "I will start with reading a text file from the web. This is the file: https://www.gutenberg.org/files/2446/2446-0.txt\n",
    "(I don't have any agenda - I just like this play, so chose it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "732f6812",
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib import request\n",
    "#urllib is a built-in python library for handling web interactions such as accessing a web-page, reading its contents etc.\n",
    "myurl = \"https://www.gutenberg.org/files/2446/2446-0.txt\"\n",
    "response = request.urlopen(myurl)\n",
    "mytext = \"\"\n",
    "for line in response:\n",
    "    mytext += line.decode(\"utf-8\").strip() + \"\\n\"\n",
    "    \n",
    "print(mytext)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fffeab0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.text import Text\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "\n",
    "text_nltk = Text(word_tokenize(mytext.lower()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b350f097",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0185ce4d",
   "metadata": {},
   "source": [
    "There are many ways to examine the context of a text apart from simply reading it. A concordance view shows us every occurrence of a given word, together with some context. Here we look up the word \"town\" in this text. This can be done by calling a pre-implemented utility for NLTK Text objects, called \"concordance\", as follows:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c41e6433",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_nltk.concordance(\"town\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9cd9ae6",
   "metadata": {},
   "source": [
    "A concordance permits us to see words in context. What other words appear in a similar range of contexts? We can use \"similar\" function for a given text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e1363e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_nltk.similar(\"people\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3d34463",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_nltk.similar(\"stockmann\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c5503fd",
   "metadata": {},
   "source": [
    "It is one thing to automatically detect that a particular word occurs in a text, and to display some words that appear in the same context. However, we can also determine the location of a word in the text: how many words from the beginning it appears. This positional information can be displayed using a dispersion plot. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2210fc4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_nltk.dispersion_plot([\"peter\", \"citizens\", \"town\", \"enemy\", \"morten\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a9224a2",
   "metadata": {},
   "source": [
    "# Counting Vocabulary\n",
    "\n",
    "Counting words, getting their frequencies etc are all useful functions when we start doing some NLP based analyses with text data. Let us see some simple utilities here. \n",
    "\n",
    "Let's begin by finding out the length of a text from start to finish, in terms of the words and punctuation symbols that appear. We use the term len to get the length of something, which we'll apply here to the our text_nltk variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af45cd5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(text_nltk) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d405fff",
   "metadata": {},
   "source": [
    "So this text has 42789 words and punctuation symbols, or \"tokens.\" A token is the technical name for a sequence of characters — such as hairy, his, or :) — that we want to treat as a group. When we count the number of tokens in a text, say, the phrase to be or not to be, we are counting occurrences of these sequences. Thus, in our example phrase there are two occurrences of to, two of be, and one each of or and not. \n",
    "\n",
    "How many unique tokens does this text have, though? In Python we can obtain that information using the set() function. When you do this, many screens of words will fly past. Now try the following:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e0637bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted(set(text_nltk))\n",
    "len(set(text_nltk))\n",
    "#Why am I seeing only one number instead of two?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59e15f76",
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted(set(text_nltk))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47b7af68",
   "metadata": {},
   "source": [
    "Now, let's calculate a measure of the **lexical richness** of the text. One way of getting this is by looking at the percentage of unique tokens to the total number of tokens in the text. The more the number is, the higher the lexical richness is. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f543dc5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "100*len(set(text_nltk)) / len(text_nltk)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b643df2",
   "metadata": {},
   "source": [
    "We can perhaps say that the vocabulary in this piece of text is \"less varied\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b223df27",
   "metadata": {},
   "source": [
    "Next, let's focus on particular words. We can count how often a word occurs in a text, and compute what percentage of the text is taken up by a specific word:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8ddb646",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(text_nltk.count(\"petra\"))\n",
    "\n",
    "print(100 * text_nltk.count('stockmann') / len(text_nltk))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daa94d3a",
   "metadata": {},
   "source": [
    "# texts and lists\n",
    "\n",
    "In NLP, we commonly see a piece of text being represented as a list of sentences/tokens, and a list of sentences further represented as a list of tokens. It is important to note that there are no perfect sentence splitters or word tokenizers. There are different options around, some are tailored to specific use cases (e.g., NLTK has a tweet tokenizer among others). In this section, we will see a few examples of word and sentence tokenization from nltk. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cf2ed2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#NLTK has pre-implemented functions to split a text into sentences, and sentences into tokens. \n",
    "from nltk.tokenize import sent_tokenize, word_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbb4029e",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_sens = sent_tokenize(mytext) #i am taking the raw mytext variable directly, not text_nltk.\n",
    "len(all_sens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89c37d9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_sens[233] #looking at a random sentence.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a9fe792",
   "metadata": {},
   "outputs": [],
   "source": [
    "words_in_a_sentence = word_tokenize(all_sens[233])\n",
    "print(words_in_a_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3845dfe",
   "metadata": {},
   "source": [
    "## Computing simple statistics from a text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fce662fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import FreqDist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ea557d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "fdist1 = FreqDist(text_nltk)\n",
    "print(fdist1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a607070d",
   "metadata": {},
   "outputs": [],
   "source": [
    "fdist1.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5d5a5ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "fdist1[\"petra\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "068838be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Selecting words longer than 15 characters.\n",
    "myvocab = set(text_nltk)\n",
    "\n",
    "long_words = [w for w in myvocab if len(w) > 15]\n",
    "\n",
    "long_words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fee8be7",
   "metadata": {},
   "source": [
    "Exercise: What are other ways of writing the same line of code for long_words?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcfa6f82",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Words that appear more than 10 times and less than 20 times\n",
    "\n",
    "mylist = [w for w in myvocab if fdist1[w] > 10 and fdist1[w] < 20]\n",
    "print(len(mylist))\n",
    "mylist[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf1ad72d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#The following piece of code does the same too:\n",
    "mylist = [w for w in myvocab if fdist1[w] in range(11,20)]\n",
    "print(len(mylist))\n",
    "mylist[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f87979f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#And, what about this?\n",
    "mylist = [w for w in myvocab if fdist1[w] in range(10,20)]\n",
    "print(len(mylist))\n",
    "mylist[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94b03c0f",
   "metadata": {},
   "source": [
    "Question: Why are these both different? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af474646",
   "metadata": {},
   "source": [
    "## Collocations, ngrams and other counts\n",
    "\n",
    "ngrams are sequences of n words that appear together. Bigrams are pairs of words that appear one after another. Collocations are frequent bigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ce6ddc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import bigrams\n",
    "bigrams_list = list(bigrams(text_nltk))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8c7c32d",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(bigrams_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01e071a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "bigrams_list[22]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcb50ad2",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_nltk.collocations()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8c19070",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_nltk.collocation_list()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "336b7056",
   "metadata": {},
   "source": [
    "Counting words is useful, but we can count other things too. For example, we can look at the distribution of word lengths in a text, by creating a FreqDist out of a long list of numbers, where each number is the length of the corresponding word in the text:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73d4d839",
   "metadata": {},
   "outputs": [],
   "source": [
    "dist = FreqDist([len(w) for w in text_nltk]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "337a91f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "dist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01294b21",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(dist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f5bd09c",
   "metadata": {},
   "outputs": [],
   "source": [
    "dist[21] #seeing how many 21 character tokens exist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc7988a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "dist.max() #What is this doing? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2be5a3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "100*dist.freq(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c14d88de",
   "metadata": {},
   "source": [
    "i.e., words of length 5 occupy about 8% of the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f52cee39",
   "metadata": {},
   "outputs": [],
   "source": [
    "100*dist.freq(19)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e8370b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "dist.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29a9840e",
   "metadata": {},
   "source": [
    "Exercise: Think about how you can extract trigrams and their frequency distribution, based on what you learnt so far, and exploring a little bit of NLTK!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "888fea22",
   "metadata": {},
   "source": [
    "let us take a quick 5 minutes break here, and I will wrap up today's class with a short overview of another useful concept - regular expressions. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}