Session_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3e89c5b3",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/tapi-logo-small.png\" />\n",
    "\n",
    "This notebook free for educational reuse under [Creative Commons CC BY License](https://creativecommons.org/licenses/by/4.0/).\n",
    "\n",
    "Created by [J.D. Porter](https://) for the 2023 Text Analysis Pedagogy Institute, with support from [Constellate](https://constellate.org).\n",
    "\n",
    "For questions/comments/improvements, email porterjd@upenn.sas.edu<br />\n",
    "____"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68f932d1",
   "metadata": {},
   "source": [
    "# `Finding Word Meaning Through Context` `3`\n",
    "\n",
    "This is lesson `3` of 3 in the educational series on `finding word meaning in context`. This notebook is intended `to show how to find the collocates of any given target term—that is, the words that tend to occur near a word that interest you—and then how to find the distinctive collocates in particular`. \n",
    "\n",
    "**Audience:** `Teachers` / `Learners` / `Researchers`\n",
    "\n",
    "**Use case:** `Tutorial` \n",
    "\n",
    "**Difficulty:** `Beginner`\n",
    "\n",
    "**Completion time:** `90 minutes`\n",
    "\n",
    "**Knowledge Required:** \n",
    "```\n",
    "* Python basics (variables, functions, lists, dictionaries)\n",
    "* A basic familiarty with the material from the first two sessions:\n",
    "    * How to extract the words from a .txt file and count them\n",
    "    * How to find the Most Distinctive Words in one text, measured againt a broader corpus\n",
    "\n",
    "```\n",
    "\n",
    "**Knowledge Recommended:**\n",
    "```\n",
    "* n/a\n",
    "```\n",
    "\n",
    "**Learning Objectives:**\n",
    "After this lesson, learners will be able to:\n",
    "```\n",
    "1. Find the key words in context (or KWIC) for any given text and target term\n",
    "2. Identify the most common collocates of a target term\n",
    "3. Find the distinctive collocates of a target term\n",
    "4. Apply these procedures to multiple target terms across multiple texts\n",
    "```\n",
    "**Research Pipeline:**\n",
    "```\n",
    "1. Gather a file in the .txt format and save it somewhere on your machine\n",
    "2. Use whatever steps you're interested in from this notebook\n",
    "3. If you have written out some of your data, explore it in a program like Excel or Google Sheets\n",
    "4. Interpret!\n",
    "```\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "157c0555",
   "metadata": {},
   "source": [
    "# Required Python Libraries\n",
    "\n",
    "* To keep things simple, we will try to work with very few libraries in this notebook. We'll use os, as well as one function from scipy.stats\n",
    "\n",
    "## Install Required Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5480e2a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Import Libraries ###\n",
    "import os\n",
    "\n",
    "from scipy.stats import fisher_exact"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dedd148",
   "metadata": {},
   "source": [
    "# Required Data\n",
    "\n",
    "**Data Format:** \n",
    "* plain text (.txt)\n",
    "\n",
    "**Data Source:**\n",
    "* included files (though you may supplement these whenever you feel comfortable doing so!)\n",
    "\n",
    "**Data Quality/Bias:**\n",
    "\n",
    "Included texts are from freely available sources like Project Gutenberg and Wikipedia. They have not been vetted for textual accuracy relative to, say, a scholarly edition. They are subject to the usual biases of those sites, and (especially in the case of Wikipedia) may not reflect the current state of the material online. The F. Scott Fitzgerald novels also do not reflect his entire corpus, since two novels are subject to copyright law (1934's _Tender is the Night_, as well as the posthumously published _The Last Tycoon_).\n",
    "\n",
    "**Data Description:**\n",
    "\n",
    "This lesson uses textual data in .txt format (utf-8 encoding) from various sources."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f53edaa2",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Introduction..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ffabbb4-123f-49ff-b8c4-ff23acf1533f",
   "metadata": {},
   "source": [
    "So far, we've been working primarily with individual words and their counts and frequencies, without much attention to word _order_. Text miners sometimes call this a \"bag-of-words\" approach: Each text is treated as a collection of words, many of which occur more than once, but features like syntax, dependency, deixis, narratology, and context are ignored. There's nothing wrong with analyzing a text this way—for instance, we learned a lot about the words that distinguished our Beatles articles—but clearly there is plenty more to learn if we treat it less like a bag of words and more like, well, a text! \n",
    "\n",
    "In today's lesson, we take a few relatively simple steps to put word order back into the mix. We have already begun thinking about words in their contexts—for instance, in an article about Paul McCartney vs one about John Lennon—but now we will narrow our focus considerably, often to the level of a sentence or a phrase. As a result, we will likewise narrow our object of comparison. By finding collocates, the words that appear near any given target term, we will be able to identify the words that distinguish _words_. We know \"bass\" is distinctive of Paul and \"peace\" of John—but what distinguishes \"bass\" from \"peace\", or from everything else? We probably won't arrive at a final theory of word meaning 90 minutes from now, but hopefully we'll have some tools to help us think about it in new ways!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceb84723-deb5-488f-8504-71721c8d78ab",
   "metadata": {},
   "source": [
    "# Getting a list of words from a file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03930633-a32f-4b31-8fe2-c0ea6650d55e",
   "metadata": {},
   "source": [
    "Since we've already covered this material, we can use a few familiar functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8966f233-ad26-4ff1-a9c3-034ed740d4ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Takes a word and returns a \"cleaned up\" version of the word\n",
    "def alphaclean(someword):\n",
    "    chars = [i for i in someword if i.isalpha()]\n",
    "    cleanword = ''.join(chars)\n",
    "    cleanword = cleanword.lower()\n",
    "    return cleanword\n",
    "\n",
    "# Takes a filename and returns a list of words (optionally cleaned)\n",
    "def file2words(somefile,clean=True):\n",
    "    with open(somefile) as source:\n",
    "        text = source.read()\n",
    "    words = text.split()\n",
    "    if clean:\n",
    "        words = [alphaclean(i) for i in words]\n",
    "    return words\n",
    "\n",
    "# Takes an items and adds it to a specified count dictionary\n",
    "def addtocountdict(something,somedict,weight=1):\n",
    "    if something not in somedict:\n",
    "        somedict[something] = 0\n",
    "    somedict[something] += weight\n",
    "    return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95bfe8b6-f6a5-4300-b6d1-f791bfca198a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a variable for your filename, using a full path if need be\n",
    "\n",
    "# Turn the file into words\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a0b395e-77d4-4729-89a5-530d8b3033ef",
   "metadata": {},
   "source": [
    "# Getting KWIC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "542f4bd8-37c8-4af2-8bb2-647b6fd66b90",
   "metadata": {},
   "source": [
    "Because lists in Python preserve order, we can use them to extract the context that surrounds any given target word. Let's say we want to find the contexts in which Austen writes the word \"estate\" in _Pride and Prejudice_. The basic method is simple: \n",
    " * Decide how big our \"window\" should be. Different sizes capture different kinds of relationships among words. A window of about 10 words before and after the target term is fairly standard, since it's a nice round number near the scale of a sentence.[$^{1}$](#1) \n",
    " * Find all occurrences of \"estate\" in the text.\n",
    " * Grab the whole window, from 10 words before \"estate\" to 10 words after.\n",
    "\n",
    "If we simply use these results in this form, we'll basically have Key Words in Context, or KWIC. People use the term/acronym KWIC in different ways, sometimes referring to a way of making a concordance, but for our purposes it just means grabbing the context that surrounds target terms!\n",
    "\n",
    "<hr></hr>\n",
    "\n",
    "__Footnote__\n",
    "\n",
    "1. <a id=\"1\"></a> Baroni, Bernardi and Zamparelli say that if the context is very large—e.g., entire documents—then analyses tend to capture \"topical\" relationships (like that between \"war\" and \"Afghanistan\"), whereas small contexts capture \"taxonomical\" relationships (like that between \"dog\" and \"hyena\") (251). They're discussing more complex methods of distance analysis than we're getting into here, but I still think that's an interesting example of the ways context size can affect our results, or even our sense of what \"meaning\" means. Baroni, Marco, Raffaella Bernardi, and Roberto Zamparelli. \"Frege in space: A program for compositional distributional semantics.\" _Linguistic Issues in Language Technology_ 9 (2014): 241-346."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "868759dd-cfc3-4779-883a-463ccd1423b6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Pick a target term\n",
    "\n",
    "# Pick a window\n",
    "\n",
    "# Create an object that can store your results\n",
    "\n",
    "# Go through a list of words and find the target term\n",
    "# Grab the context surrounding the target term\n",
    "\n",
    "# Let's turn these steps into a function!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b2054f3-2646-4cc5-8381-a5921ccbdc27",
   "metadata": {},
   "source": [
    "# From KWIC to collocates"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea2eea1-522b-4a54-af5c-1b12e2ab26e8",
   "metadata": {},
   "source": [
    "For our purposes, a \"collocate\" just means any word that appears within some context window for a target term. The term collocates is sometimes used in more specialized ways. For instance, some researchers will only consider a word to be a collocate if it occurs _significantly_ more often near the target term. But for now, we'll keep it simple!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff2bf183-b5c8-48e9-b077-2d244bbdea24",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pick a target term\n",
    "\n",
    "# Create a count dictionary for your collocates\n",
    "\n",
    "# Count the collocates\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46033dc2-7624-403c-98d2-d56e240890fd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This isn't important to learn today, but for convenience we can use it to sort a count dictionary\n",
    "\n",
    "for i in sorted(counts,key = lambda i:counts[i],reverse=True):\n",
    "    print(i,counts[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6f3fa4a-cc05-422f-a357-068ac3cf678b",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Getting interesting frequencies for target terms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba0ba9ed-2735-4832-a1f6-e67105674460",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Filter stopwords\n",
    "\n",
    "# Consider converting to frequency\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb3e161f-d480-4b85-a410-68e3b77605c9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Here's the sorting function again\n",
    "\n",
    "for i in sorted(int_counts,key = lambda i:int_counts[i],reverse=True):\n",
    "    print(i,\"\\t\",int_counts[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f88cfd6-1a4f-4fec-bc21-dba042ee0e86",
   "metadata": {},
   "source": [
    "# Testing a collocate for distinctiveness"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1870058f-82bf-4ba2-ad20-f818aa6104f8",
   "metadata": {},
   "source": [
    "Here we cash in on the methods we learned in session 2. The trick is to adapt our Fisher's Exact Test so that it can handle a comparison between collocates and the larger text, rather than between one text and some others. Let's work through what we need to build the table that we'll use in the test. We can start with the general principles we covered last time:\n",
    "\n",
    "* a = the number of times the word appeared in the corpus we're interested in\n",
    "* b = the number of times any other word appeared in that corpus\n",
    "* c = the number of times we would have expected the word to appear if it was evenly distributed across all our corpora\n",
    "* d = the number of times any other word would have appeared if the word we're examining had been appearing at its expected rate\n",
    "\n",
    "For collocates, this will translate to:\n",
    "\n",
    "* a = the number of times the word appeared _within the window surrounding our target term_\n",
    "* b = the total number of collocates minus a\n",
    "* c = the number of times we would have expected the word to appear if it appeared _within the window surrounding our target term_ at the same rate as _within the text as a whole_\n",
    "* d = the total number of collocates minus c\n",
    "\n",
    "As with our Beatles example in session 2, the tricky part is figuring out __c__. We need to know the expected rate of occurrence for the word we're testing (let's call it __r__ again). If we want to know the significant collocates of \"estate\" within _Pride and Prejudice_, we can set that rate using the full novel. The important thing to remember is that we need the rate not for \"estate\", but for whatever collocate we want to test. \n",
    "\n",
    "For example, let's say we want to see if \"derbyshire\" is a significant collocate for \"estate\" in _Pride and Prejudice_, even though it only appears 3 times in our KWIC. The word \"derbyshire\" appears 24 times in the novel, which is 121,567 words long. So our __r__ will be:\n",
    "\n",
    "```Python\n",
    "r = 24 / 121567\n",
    "```\n",
    "\n",
    "That comes to about 0.0002. Meanwhile the total number of collocates for \"estate\" is 400 (one easy way to get this is to sum the values of our collocate count dictionary). So that gives us:\n",
    "\n",
    "* a = 3\n",
    "* b = 400 - 3\n",
    "* c = __r__ * 400\n",
    "* d = 400 - c\n",
    "\n",
    "Which comes to:\n",
    "\n",
    "* a = 3\n",
    "* b = 397\n",
    "* c = 1\n",
    "* d = 399\n",
    "\n",
    "If we run our Fisher's Exact Test on this, we get a p-value of .62. So, even though Darcy's estate is in Derbyshire, the word \"derbyshire\" doesn't seem to be a significant collocate of \"estate\"!\n",
    "\n",
    "That's kind of a boring result, so let's pick a different target term, and build a method for finding all of the distinctive collocates!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0a1cb75-850f-4658-974c-922b2b2cfcaa",
   "metadata": {},
   "source": [
    "# Testing _all_ of the collocates for distinctiveness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4c5b944-8537-4dca-9b11-75ed6f67ce85",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Here's the function we used last time for getting the p value via a Fisher's Exact Test\n",
    "\n",
    "def get_fishers(someword,somecountdict,someratedict,alternative='greater'):\n",
    "    r = someratedict[someword]\n",
    "    wc = sum(somecountdict.values())\n",
    "    a = somecountdict[someword]\n",
    "    b = wc - a\n",
    "    c = round(r*wc)\n",
    "    d = wc-c\n",
    "    p = fisher_exact([[a,b],[c,d]],alternative=alternative)[1]\n",
    "    return p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed3bbae0-92a6-419b-b08f-4389c757db3b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# In this cell we'll set up everything we need to find the p value\n",
    "\n",
    "# Find the counts of all words for the text\n",
    "\n",
    "# Make a rate dictionary for the corpus\n",
    "\n",
    "# Pick the target term\n",
    "\n",
    "# Get the KWIC of the target term\n",
    "\n",
    "# Count the collocates\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d68b75d-514a-4d78-b926-ffdd49d4a8c0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6545c5d1-7567-4556-9364-a764983e503f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "76670a61-420d-42ae-b8a2-103758137611",
   "metadata": {},
   "source": [
    "# Checking collocates in a larger corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d29d4ba-b6d1-47f8-9baf-a86ebddfcf84",
   "metadata": {},
   "source": [
    "You may have noticed that we're not getting a _ton_ of results for any given target term. That's because we're dealing with a fairly small amount of text—we only collect text when we find our target term, and even then we might not collect _much_ text. This means that collocate analysis often gets more interesting as we scale up the amount of text we examine. In the context of this session, we don't want to go _too_ far with that, since large corpora require more labor (and processing time) to store, share, analyze, and interpret. But we can get a little bit of a sense by zooming out to Austen's entire corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f205ea40-6c71-4eb7-ad34-9f4e14f199f0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Name the directory containing the novels\n",
    "\n",
    "# Get the texts out of it\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff6e0565-9e91-4ee3-b546-abbe1555a33f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# In this cell we'll set up everything we need to run our Fisher's Exact Test in a bit\n",
    "\n",
    "# Find the counts of all words for the text across all the novels\n",
    "   \n",
    "# Make a rate dictionary for the corpus\n",
    "\n",
    "# Pick the target term\n",
    "\n",
    "# Get the KWIC of the target term across the whole corpus\n",
    "\n",
    "# Count the collocates\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe9c723a-fa6b-4932-8d64-a6c519eb3a53",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ff5de3e-a11a-45a2-a612-223570bc3c72",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "cf0a37ab-835e-4e51-b9b8-c115835e4e9f",
   "metadata": {},
   "source": [
    "# Checking several target terms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c91da73-a89f-47fa-9f71-e9034b0b1a5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "target_terms = ['father','mother','sister','brother','cousin','uncle','aunt','nephew','niece','grandfather','grandmother',\n",
    "               'son','daughter','husband','wife']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6404db2b-64bf-4b54-b88f-e9e362faba89",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Some things won't change!\n",
    "\n",
    "# Find the counts of all words for the text across all the novels\n",
    "   \n",
    "# Make a rate dictionary for the corpus\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "408db37e-0ba7-4312-8cb5-02062aa997e9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# For the KWIC, we'll now need to associate the results with specific target terms\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe6fd34d-e6b1-4ab4-acb6-76cdf7566e49",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Same goes for our collocate counts!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "daca011e-b745-4a5a-a7fe-a1c865f2402c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6da32753-5c77-49f8-bd06-76de79641a9d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a8f7096e-8c35-4910-a784-d993c2d81f37",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Comparing terms across corpora"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36e531b5-a2c4-40b9-b38b-78d4f74b0a34",
   "metadata": {},
   "source": [
    "Comparing terms across different corpora raises some complicated, even profound questions. If, for instance, we try to compare Austen’s language to that of F. Scott Fitzgerald, we quickly run into some insurmountable difficulties. When it comes time to analyze “distinctiveness” for any given target term, on what should we base our expectations?  Let’s consider two possibilities. First, we can set the base rate by considering all of the novels in both corpora. (For simplicity's sake, we'll go back to thinking about one target term at a time.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d571548-e7af-4051-aa09-8f42e0076ab3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "target_term = 'love'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55814ed1-2d4f-4762-b434-8d9f9bb1bdad",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# First, let's grab all of the file names from both directories\n",
    "# Let's also keep track of which corpus contains which files\n",
    "\n",
    "\n",
    "# Then we can proceed as before\n",
    "\n",
    "# Find the counts of all words for the text across all the novels\n",
    "\n",
    "    \n",
    "# Make a rate dictionary for the corpus\n",
    "\n",
    "# For the KWIC, we'll now need to associate the results with specific corpora\n",
    "    \n",
    "# For the collocate counts, we'll likewise separate by corpus\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a93236f4-c238-43b8-afc8-60fa48cca0f8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "alpha = .05\n",
    "cutoff = 5\n",
    "\n",
    "output_table_1 = [['target_corpus','collocate','count','p-value','obs/exp']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6deefb7-3db1-4f74-b131-0adcd73330d4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "028a9411-3eb0-4914-95b4-97fc4711aaf5",
   "metadata": {},
   "source": [
    "These results are interesting, but it’s difficult to say what they’re showing us about the target term, vs what they’re telling us about the difference between Austen and Fitzgerald. After all, one is a British novelist from the turn of the nineteenth century, and the other is an American writing in the Jazz Age (even the name of the era involves a word Austen didn’t have). Do the differences we find reflect different understandings of the target term, or differences between the two authors and their language contexts? Just consider the difference between \"cannot\" and \"can't\"!\n",
    "\n",
    "To resolve this, we could try to base our expectations on each author’s corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "664018de-b6bd-46b1-9ca0-62f3d684406d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# First, we get counts and rates for each corpus\n",
    "\n",
    "# Since all we needed to change was our expectations, or rates, we can then proceed exactly as before\n",
    "# For the KWIC, we'll now need to associate the results with specific corpora\n",
    "    \n",
    "# For the collocate counts, we'll likewise separate by corpus\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "992eb593-e3ad-4d0a-86de-c1ee71297d0e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "alpha = .05\n",
    "cutoff = 5\n",
    "\n",
    "output_table_2 = [['target_corpus','collocate','count','p-value','obs/exp']]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7de7009d-0e16-4fdc-9b91-faf82555e892",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "515a32cf-d9a2-4a91-84b8-da24efd58f5a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# You don't need to learn this stuff, but here we make sets showing the unique (corpus,collocate) pairs from each output table\n",
    "# This enables us to see the difference between the two methods more easily\n",
    "\n",
    "table_1_pairs = set()\n",
    "table_2_pairs = set()\n",
    "\n",
    "for row in output_table_1:\n",
    "    table_1_pairs.add((row[0],row[1]))\n",
    "    \n",
    "for row in output_table_2:\n",
    "    table_2_pairs.add((row[0],row[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4846f40-e5c4-4495-81ab-07f7a7c91557",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Here we see (corpus,collocate) pairs that appear in the first output_table, but not the second\n",
    "table_1_pairs - table_2_pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c70d0e9-405e-4ffe-9105-7420b09de392",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Here we see (corpus,collocate) pairs that appear in the second output_table, but not the first\n",
    "table_2_pairs - table_1_pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb151b01-b129-4402-803b-1e8581751930",
   "metadata": {},
   "source": [
    "Now we can see how each author uses the target term relative to the rest of that author’s corpus. Fitzgerald isn’t saying \"beauty\" bc it’s a new word, but because he uses \"beauty\" near the target term more than he uses it in the rest of his corpus. But what does that tell us? Does \"love\" mean something different for Austen and Fitzgerald? Does \"beauty\"? If the language has changed, does any shared collocate mean the same thing? Does any word?\n",
    "\n",
    "With MDW, we could skip a lot of these questions by comparing the novels in their entirety, rather than zooming in on target terms. But that wouldn’t solve the problem; it would only ignore it. \n",
    "\n",
    "Now we've arrived, more or less, back at the beginning. We can find the words that go with any target term that interests us. We can even understand that target term with respect to these surrounding words. Have we then arrived at the meaning of our target terms? I think we're at least in range of it. But when you consider the decisions we have to make about expectations, comparisons, corpora, word cleaning, counting... and much more — when you consider all of this, you see how many different ways there are to _find_ meaning. In other words (no pun intended), it's tough to say what meaning means. When we look for it, we will always have to consider context."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}